Data Traction: How To Evaluate AI Companies With Little or No Revenue - Part 1: Beyond Traditional Metrics

Using metrics from SaaS and older industries isn't the best way.

Happy Sunday and welcome to Investing in AI. I’m Rob May, a Partner at PJC, where I focus on seed stage AI and robotics investments. I’m also a participant in the ScalingAI AngelList syndicate for growth AI companies, which you can join if you want to make some growth AI investments.

Be sure to check out our Investing in AI podcast as well.

— Interesting Links —

The AI Wolf That Preferred Suicide. OneZero.

Training AI To Think With Analogies. Quanta Magazine.

The World’s First Patent That Lists AI as an Inventor. GlobalPost.

China’s Sputnik Moment. Foreign Affairs.

A Reflection on How AI Will Change Conflict. Economist.

The Future of Deep Learning Is Photonic. IEEE.

— Research Papers —

Guided Disentanglement in Generative Networks. Link.

Generative Adversarial Networks in Time Series. Link.

— Commentary —

There was a time when software was a pain in the ass to buy. You had to go through your I.T. department, no matter how small or non-critical the software was to the organization. You needed your own server to install it on. You had to negotiate a contract, and a maintenance package. Sometimes you needed the help of the company that wrote the software to get up and running. It was slow. There was a lot of friction. If you are over 35, you may remember the tail end of that world.

SaaS came along and changed all of that. There are no different versions of the software, it’s just a subscription to one giant code base running in the cloud. All you need to sign up is a credit card and a web browser. It was, and is, very low friction.

Transition points like this, Retail -> Ecommerce, Packaged Software -> SaaS, present challenges for investors. New metrics come along like “eyeballs” instead of revenue, or “MRR” instead of bookings. Initially people try out metrics, some work, some don’t. Even the ones that stick are often misused or poorly understood. But using the new metrics is much better than attempting to make sense of a new company with the old ones.

Now AI comes along, and we are seeing this issue again. Investors look at AI companies through the lens of SaaS expectations, and you know what? By those standards most AI companies SUCK.

I’ll give you an example. There are many companies in the marketplace now trying to auto-create content for you. When you initially sign up, the first few things they create are terrible. It’s not the fault of the AI - the issue is they haven’t been trained yet. AI tools need feedback from you. Many of these companies, as a result, have very high churn rates compared to SaaS companies.

As an investor, you would look at these metrics and, based on your experience with SaaS, you would say “there is no business here.” Because in SaaS, if you have a 12% monthly churn rate early in the company, it doesn’t matter if it gets better. It’s so so so far off you wouldn’t even consider investing in this SaaS platform. But with AI companies, the way to think about it isn’t as 12% monthly churn. “Churn” here doesn’t have the same sources as “churn” in traditional SaaS companies.

For an AI company, some churn might be to traditional SaaS company issues - low product value, bad user interface, better competitors, poor product-market fit. But in AI companies, while those same things may be an issue, we can add other issues. The three I most commonly see are: 1) expectations for how and why the product is used are off - this happens a lot in new markets when people are figuring out exactly how a new paradigm should work, and 2) initial performance is weak - this happens a lot because you haven’t trained the model yet, so it isn’t very customized to your needs (which is the core value prop sometimes), which leads to 3) friction because it takes time to train the model and we no longer have the patience to adopt any software with even the smallest bit of friction.

Your job, as an AI investor, is to figure out the sources of churn. So what do you look for? If some people stick with the product, and those that do consistently perform well, then it’s probably a good company with more AI related churn issues that will go away as the market evolves than broader churn issues that will stay.

The lesson here is that to evaluate AI companies, you need to look beyond the metrics we are used to. Some of them are still relevant, some of them aren’t, but most importantly - new metrics matter. What are these metrics and how, as an AI investor, should you think about them?

Much of this is still being worked out, but in part 2 (published in two weeks), I’ll take a stab at some that I look at, and what I think is important to understand.

Thanks for reading.

@robmay