The Cracks In LLM Scaling And Where To Invest Next

If LLM growth stalls, what does it mean economically?

Happy Sunday and welcome to Investing in AI. If you haven’t done so recently, check out our AI Innovator’s Podcast. The latest one features Kojo Osei from Matrix, where we discuss “zero cost inference.” Today I want to talk a bit about the slowing growth of LLMs, and what it means for investing.

If you have followed this newsletter for a long time, you know I’ve always been skeptical about the value of LLMs to scale to AGI. I’ve written before that the economics of (most) foundation models aren’t great, and also that OpenAI may be a short. These have not been popular views but, they are starting to come true.

Gary Marcus has been the biggest voice against the LLM craze, and he wrote yesterday about some of the examples where the scaling of LLMs is breaking down. He highlights that even OpenAI is seeing this trend where more data isn’t going to get us the advances it used to in the past.

I’ll also highlight this interesting public bet between Sasha Rush and Jonathan Frankle about whether, by Jan 1 2027, transformer models will still be the state of the art for benchmarked NLP tasks. For those of you who may not know why that’s important, it’s because most of these LLMs are trained using a transformer architecture. Clearly Dr. Rush believes transformers have limits as well, and we need to find something new.

What does all of this mean for investors?

I think it means there are two new intellectual vectors to invest against in AI. The first is new model ideas that extend beyond transformers and LLMs. These are emerging in ideas like state space models, liquid foundation models, and possibly revitalizing some older AI ideas like evolutionary algorithms and hierarchical temporal memories. The next wave of capabilities is likely to come from one or more of these techniques.

The second vector to invest against is things that make all of this more efficient from a power and cost perspective. I’m a big fan of investing in the hardware and infrastructure layers of AI right now. I think some of these chips that are coming to market like Cerebras, Sambanova, Groq, Femtosense, Untethered, Mythic - most will find a spot in the marketplace even as NVIDIA continues to grow for some time. The whole hardware and infrastructure market is going to be a massive layer of value capture because we are effectively taking tasks once done by humans and running them on machines. But as I’ve written before, the software layer is going to get commoditized in many areas because of how easy it will be to write software, so you want to invest one level below that in the tech stack.

In fact, I’m so bullish on this I will have an announcement soon about a new venture I’m helping to start in this area. But for now, the point is that you have to start shifting your mindset away from the view that LLMs are going to run away with AI improvements. All signs say that they aren’t, and that it is time to look at other areas to invest and push AI forward.

Thanks for reading.