The Economics of AI With GPT-3
What to build, and where to find value in a world where AI does the writing.
Happy Sunday and Welcome to Investing in AI. I’m Rob May, a Partner at PJC. I focus on AI and have invested in companies like Mythic, Botkeeper, Root, and Synthesis. If you have an AI company raising at any stage, I’d love to talk with you.
Also check out our podcast. The second episode just went live and features Heath Terry from Goldman Sachs. Coming up are interviews with Rana El Kaliouby from Affectiva, and James Cham from Bloomberg Beta.
This week I’m writing about the economics of GPT-3, how it changes what you can build, the skill sets you need in your company to use it well, and the best way to think about where value accrues in the NLP AI value chain.
— Interesting Links —
Is China Emerging As The Global Leader in AI? Harvard Business Review.
Driving Model Performance With Synthetic Data. Synthesis AI Blog.
What Chip Startups Can Learn From Google’s TPU Team. NextPlatform.
How to Defeat a Boston Dynamics Robot in Mortal Combat. Vice.
Pentagon Funds Killer Robots. Washington Post.
Cisco Unveils Real-Time Translation for Webex. ZDNet.
The Billion Dollar AI Problem That Keeps Scaling. NextPlatform.
— Research —
Graph Time Convolutional Neural Networks. Link.
The Dota2 Bot Competition. Link.
— Commentary —
GPT-3 made waves when it came out last year from OpenAI. It has been considered the best AI language model currently on the market. Over the past 5-6 years many NLP startups have emerged to perform different tasks. With the release of GPT-3 there have been some who thought it’s game over for these startups. In my opinion though, the right way to think about GPT-3 is closer to a piece of platform infrastructure, more akin to AWS. That isn’t a perfect analogy but I will explain below why I see GPT-3 as the real beginning of opportunities in NLP, not the end.
GPT-3 can’t perform every language task. There are some limitations based on the data set it was trained on. For example, it can’t help with news related issues about new topics because they aren’t in the training data. And it also can’t generate really long pieces of text. An intro paragraph for a blog post is within it’s scope. A full blog post is not. The way to think about GPT-3 is that is can very rapidly generate language for common, short, language tasks.
If the barrier to generating basic language goes down, what goes up in value?
First of all, the skill of prompting GPT-3 goes up in value. The way you interact with the model is to give it various prompts - pieces of language that tell it what type of words and sentences to generate. There is an art to prompting GPT-3 and that type of “prompt engineering” can be valuable. For example, people are using GPT-3 to write code, solve equations, and other creative ideas that don’t seem at first glance to be NLP problems. Better prompts get better results, and prompt engineering is a valuable skill.
Secondly, user experience design for NLP workflows increases in value. One of the things I’ve seen apps doing with GPT-3 is generating several language samples of whatever it is you want - blog post titles, ads, short emails, etc. The workflow for approving those, changing those, declining those, is not something we are used to yet, so building a good UX is valuable at this point in the industry trajectory.
And finally, the value of vertical-specific NLP for language not well represented in GPT-3 will go up. The model wasn’t trained, for example, on a lot of highly technical medical language, so building applications that need that might be best served with combo of GPT-3 plus in-house language models.
One thing to keep in mind is that OpenAI requires you to meet regularly with them to discuss your application, and any changes you are making. To get more access as you scale also requires OpenAI approval, which could be a gating factor to rapid growth for apps built on GPT-3. So there is definitely a tradeoff between using GPT-3 to make language easy, and the possible limitations OpenAI may impose that drag on your business. In my opinion though, it’s worth the risk to build on the platform.
There are some interesting trends to watch in this space. Microsoft has a special license to GPT-3, so I’m curious to see where the OpenAI-MSFT relationship goes. Models are coming down in price to train for a given model size, so, while training GPT-3 from scratch is beyond the price point of most companies, in 2-3 years it might be much closer, and I expect to see GPT-3-like functionality available much more cheaply. Of course, by then we may have a GPT-4 that is the best of breed. Who knows?
The way to play the NLP space as an investor, then, is to focus on use cases that benefit from dramatically reduced costs of common language tasks. GPT-3 can’t yet replace a Forrester analyst or a specialist in any field, but tasks that are labor intensive for writing yet common enough that almost any educated human can perform them are good candidates to be done by GPT-3.
Value will accrue in the areas I mentioned above - savvy NLP UX design and workflows, vertical applications where GPT-3 struggles, and creative prompt engineering for unique use cases.
GPT-3 is a transformative piece of technology, and an important part of the emerging tech stack of AI. But it’s the beginning of a wave of startups and established companies using NLP, not the end. Expect to see use cases we haven’t even imagined yet.
Thanks for reading, and as always, please reach out if you have questions, comments, or other feedback.