We’re Still Early on The Exponential
Jascha Sohl-Dickstein recently encouraged attendees at his Stanford talk to play with frontier models, as imperfect as their interfaces might seem, by referencing an Epoch chart to explain just how early we are on The Exponential. While I knew from experience the importance of keeping up with the newest models, I did not deeply understand the Epoch chart, which was from 2024. How could it still be early if the chart he’s pointing to is already two years old and Claude Code was not even a thing back then? To properly absorb this chart’s implications, I had Codex and Claude build me a better explainer, which is at the end of this post. Maybe others will also find it useful so that we may also be like Jascha, and truly believe that we are still early.
The short version of what the box plot tells us is that GPT-4 is really nothing because the model of 2030 could use 10,000x more computeI have not looked into how Epoch reaches their compute estimates, but it might be in their report. for its training. This is also a proxy of a model’s intelligence; more compute generally leads to smarter models and, if GPT models jump in integer value roughly every year from hereon, then this 2030 model is GPT-10. I requested a small addition to the chart from my agents; they show where GPT-6 lies on it. It used 45 times more compute to train than GPT-4, and is a helluva lot smarter. Yet GPT-6 sits 200x below Epoch’s 2030 line, so GPT-10 will easily dwarf it. That 200x is what explains why we are early.
Think about it: the jump from Opus 4.5 to both Fable 5 and Opus 5.5 felt tremendous. I don’t know what exactly this GPT-10 will manifest as, but I expect to feel the difference when I see it.
Jascha thinks we shouldn’t miss out, but after getting to spend time with this chart, I have some other questions: what if Epoch’s estimates are conservative? What if more than 10,000x compute gets thrown at training runs — maybe 1,000,000x — by 2030Probably something worth looking at in a new analysis of space data centers someday over here.? If so, we should be more than a little worried; missing out on the next 4-5 years would be costlier than Jascha suggests. Of course, playing with these models is no guarantee that you will personally capture the exponential’s value. But it is better to have felt the exponential and fallen off than never to have felt it at all.
So how do you not miss out? The easiest mistake one could still make is not playing with the newest models despite being able to afford them. In the developed world a subscription costs less than your internet bill, or about five bougie flat whites in London.1
Okay, with that we can go into the animation of Epoch’s plot, which estimates the largest training runs possible by 2030 and the four constraints on such training runs. Of these, I was unfamiliar with latency wall, which is described as so:
Latency wall. The latency wall represents a sort of “speed limit” stemming from the minimum time required for forward and backward passes.
-
But we should also spare a thought for those who remain without access to coding agents and LLMs as AI subscriptions are expensive for those in many other parts of the world. They have access to smartphones and can still be offered the chance to feel the benefits of AI through some of us. One clear example to me is by making smarter educational materials available for free online, like the interactive textbook for orbital mechanics that I discussed in my last essay. It embeds Sonnet agents that guide you through the book’s contents based on your response to questions in it. The agent’s costs are borne by me (through a generous Anthropic gift: $500 of API tokens, monthly — thanks!). I’m hoping more folks use the book and that even more create such resources in their fields of expertise. The cost is marginal to you, as an author, and will give those with less financial flexibility an education at no cost (especially compared to a university education). ↩