Hello guys, I’ve been wanting to expand about my thoughts on Palantir. I have been trying to contain my excitement for awhile because I honestly didn’t want this company to become a meme or anything (I totally did not help with some of my posts about it) but I do want to share my understanding about the tech and this company. I’ll try to refrain from posting any useless information that’s just going to come off a pumping the stock from now on.
Let me just start this off by requesting that we leave out the ethical and morality discussions and focus on the technology. I do not know enough and thus do not deserve to have opinions regarding the philosophical aspects of the technology. The only thing I can say for certain is that for better or worse, this technology will change lives.
I’ll begin with the basics, I’m going to try to simplify it as much as I can, but it does help if you have some general understanding about software and how a computer works….
Natural Language Processing
Imagine that we have a big email data set, terabytes of emails collected. We want to find out whenever a person named John Doe, who lives in Iowa, was mentioned or involved in an email chain. We task a data analyst, engineer, or a developer to find all this data points.
No problem, I’ll just write up some code that identifies the email by the sender and recipient with John Doe’s email address.
A naive approach. This approach would definitely find emails that involved John Doe but completely miss out any mentions of John Doe that did not have him involved the in the email chain, not to mention, you would need a list of all of John Doe’s emails which may be information that is already hard to come by.
Ok, I can cover my bases by searching for the word John Doe in the subject and body email.
Great, we pulled in more data that could be emails talking about John Doe. Then you realize the fact that people call him Johnny Boy, JD, Joe, Mr. Doe, Doe Man.
Well shit, how can I possibly cover every permutation of this person’s name? Not to mention, there seems to be other John Doe that doesn’t seem to be the same John Doe who lives in Iowa that we are looking for coming back in the results.
Hopefully you can start to see the problem. It would be impossible for a programmer telling the machine what to look for to come up with every possible permutations and edge cases to find the information we are looking for. You think to yourself, we’re going to have to hire thousands of people to read through terabytes of data to identify the emails we want. Then you start to think, how would a human actually identify John Doe from text and determine that THAT is the John Doe we are looking for? The answer is in the context of the email. If the person is talking about Johnny Boy’s burger shack in Iowa, it’s likely the John Doe we’re looking for. But if the person is talking about Mr. Doe’s investment firm in Panama, then that’s probably another Mr. Doe (thought a pretty shady one if you’d as me).
Now, how do we get a computer to understand a context of words, how can we transfer that skill to a computer program? We can use machine learning. Humans can identify examples of John Doe and teach the machine that this is what we are looking for, through a feed back loop we can continually improve the understanding of this “model” and teach the machine to identify the specific block of information we are looking for.
Obviously this doesn’t just work for names. You can be specific as you want and as general as you want. Say phone numbers, (111) 222-3333 can be written as 111-222-3333, 111.222.3333, 111 222 3333, 111 CAB-3333, and the list goes on.
Natural Language Understanding
Ok, we now want to find out where he’s been and what he has been up to. How we an engineer go about solving this problem? Again, there doesn’t seem to be an easy way to accomplish this. We know how to identify entities, we can actually apply the same concepts we used to teach the machine how to identify John Doe.
He's booking a flight to Moscow for 01/02/2023
We already know that “He’s” is referring to John Doe. We know that “Moscow” is a location through Natural Language Processing and we know that “01/02/2023” is a date through named-entity recognition. We can teach the machine that this is an example of travel and label this whole sentence as a “movement”. We hope that the machine can identify and trace is location just by all the cookie crumbs left behind. We can then build a map of all the places he’s been to by tracing all the identified “movement” points, sort them by date grab the locations.
A lot of the criticisms have detailed how a lot of this technology is open sourced which is true. A lot of this tech are still popular academia research topics. This also means that there are progress being made to it every single day. However, the concepts and technology may be open sources but the features and methods you build around it determine the effectiveness of the product. There is a popular saying in academia that goes like this.
Garbage in, garbage out
Using the same technique in training your models may yield results but doesn’t technically mean they would yield great results. Generating a good model has a lot to do with how clean your data is, how well you tuned the learning process, how well your features and weights are built around the algorithm.
When government/businesses consider using these products, they usually create a bake-off and analyze the performance and features included in each product. They use metrics like precision/recall that determines how many data points that the machine identified from the truth set and how many data points that the machine missed. These metrics are important to the clients for many reasons one of which is because it can potentially mean they miss or waste time on data points that lead them to nowhere and is particularly important for applications like defense. If the government picked Palantir, you can be confident that they are the top of their class.
A lot of posts have mentioned that Palantir sends consultants and is therefore a consulting company. Which I don’t disagree that their “Forward Deployed Engineers” are technically consultants. However, these consultants DO serve a purpose. The concepts and applications here are still being discovered in the market today. I’d like to think of these consultants as technical sales consultants. Companies today don’t necessarily understand the value of their data and what can be done with it. A lot of their data are just noise today. They store it and are unable to do anything with this. It is necessary for someone with the skillset to come in and build the foundations required/clean up the data for companies to start leveraging it. However once the value becomes visible, it seems to be a very sticky proposal and becomes evident that it is a leg up among their competitors. We don’t currently have enough talent in the work force today that have the ability to come in and shift the data analysis landscape. It is also very expensive to do inhouse. Having Palantir come in with their consultants is really just these companies dipping their toe in to the ocean for the benefits that is to come.
Hopefully this gives you guys a bit more information about Palantir’s tech and realize that you guys can see how a little bit of creativity in extracting information from data using the tools Palantir provides is game changing.
I will leave you all this this xkcd comic that I found was very representative of the problem at hand. Sometimes, it can be difficult to explain the difference between the the easy and the virtually impossible.