Novelty vs utility, science vs belief, and where we are with AI-first products
Entropy #3
The AI Crowd is Mad by Tim Daubenschütz is worth a read. I agree that the LLM discussion could benefit from more nuance. I agree that costs need to come down (but of course they will). I find the contemplated future for AI odd. And, in terms of startups and products, I want to use the post to frame where I believe we are with modern AI and what we should expect.
Novelty vs Utility.
Tim argues that we’re collectively experiencing survivorship bias: seeing the algorithmically promoted highlights of ChatGPT on social media that are “truly stunning” is not giving us a reasonable sense of performance.
I agree. Further, much of what we’re seeing is novelty rather than utility. But even if the highlight reel is predominantly novelty, it is indeed stunning:
However, scroll through a twitter search for what ChatGPT is doing with code, or appreciate what Intercom did with ChatGPT in less than 8 weeks, and it’s clear that there’s meaningful value adjacent to the fun.
The degree to which the highlight reel is stunning and the mix of novelty:utility matters, but what matters more is the rate of change for both and how much further the technology can improve.
Belief vs Science.
In November 2019 OpenAI released the full version of GPT-2. I remember playing with Write With Transformer, an accessible UI for interacting with GPT-2, and using Colab to fine-tune the model to create toy-like experiences. It was pure novelty and not very compelling. But I was stunned. The model was completing sentences that - sort of? - made sense. It didn’t take an overly active imagination to see what could come next.
Tim’s post highlights this tweet from Sam, taking issue with Sam’s use of ‘trust’, and stating “it is clear to me that these opinions are not anchored in reproducible observations”.
One may look at the progress over the past 3 years, from GPT-2 to ChatGPT, and decide that they need more evidence before they’ll extrapolate. Fortunately, that evidence exists in the form of scaling laws (initially published by OpenAI in 2020, revised by DeepMind in early 2022). Much like other technology trends, things are going to get better in forecastable ways for a while.
Where We Are and What’s to Come
Tim references Carlota Perez’s framework for technological revolutions and financial capital, which is useful as a way to think about our place in time and what’s to come.
I believe we’re in the midst of the irruption phase and that the frenzy is still to come.
Apparently half of the current YC class is building with ChatGPT, which is exactly what we should expect at this point (as Replit’s Amjad rightly argues). We’re still figuring out what’s possible with this technology and developing the tools and infrastructure that make it easier to build with it. All while the underlying technology is rapidly advancing. Which is to say, it’s still early.
AI-enabled products will dramatically change the world, but not as soon as some think. While most applications built in the short-term will be inane, some will become transformative companies. AI is already good enough to deliver meaningful utility with certain product experiences. And while the degree of novelty remains high today… trust the exponential.
Follow me on Twitter: @fraser
Join 1,400 other readers of Entropy and follow along as we try to make sense of modern AI and AI-enabled products: