My Mental Model of AI Broke on September 8 – Rough Ideas

Rough Ideas

Home Blog

My Mental Model of AI Broke on September 8

09 Sep, 2026

September 8, 2026 feels different to me, assuming the reported mathematical proof is correct. I don't mean that AI passed another benchmark or became better at a task humans already know how to do. Most benchmarks can be beaten by some very smart human somewhere, so the comparison is still human performance versus machine performance. A genuinely correct solution to a previously unsolved Millennium Prize problem feels fundamentally different. If no human had the solution before, then the important fact is not that AI beat a human score, but that an AI system may have produced new mathematical knowledge that humanity did not have before. I am setting aside, for now, the drama around who produced the proof and who should receive credit. My point is about what this capability would mean if the proof is correct. That feels like a much more important boundary.
My belief system was shattered the day the proof was announced. I had held a very different view of AI: that it was basically a stochastic parrot, an advanced search engine, an autocomplete system, or a next-token predictor. I remember early systems struggling with something as simple as 2+2. Then, within just a few years, we went from that to systems achieving IMO gold-medal-level performance and now, assuming this proof is correct, to a Millennium Prize problem. That completely changes how I think about the trajectory. I can imagine future AI benchmarks becoming less about solving problems humans have already solved and more about making genuinely novel discoveries. If every major AI lab starts competing to demonstrate its capabilities through new mathematics, science, and other discoveries beyond what humans currently know, that race could move extremely fast. And honestly, that is what scares me.
There are still important assumptions here. I am assuming the proof is actually correct, survives independent verification, and is genuinely a new result rather than a reconstruction of something already known. Those questions need to be settled carefully. But if those assumptions hold, I don't think "AI is just a tool" is a sufficient description anymore. A calculator performs calculations we already know how to specify. A system that can discover a correct solution to a problem nobody had solved before is something fundamentally different.
That is why I have started thinking about a possible crisis in the coming years. If AI moves from helping humans use existing knowledge to producing new knowledge at the frontier, many systems built around human intellectual scarcity could face serious pressure. Education, research, careers, and even our idea of what makes human intelligence special could change faster than society can adapt. I am no longer comfortable thinking about AI as merely another software tool. Maybe September 8 will turn out to be less important than it looks today. But if the proof survives serious scrutiny, we may look back at this period as the beginning of a very different relationship between humans and machines.
I hope humanity finds a way to turn this transition toward a better future, where these new capabilities improve human life rather than leave people behind.
References:
1. https://news.ycombinator.com/item?id=49605915
2. https://news.ycombinator.com/item?id=49613262
3. https://simonwillison.net/2026/Sep/8/on-navier-stokes/
4. https://mastodon.social/@tristanbuckmaster/117233413705701198
5. https://openai.com/index/navier-stokes-solution/

Powered by Bear ʕ•ᴥ•ʔ

The announcement of a purported mathematical proof fundamentally altered the author's mental model of artificial intelligence. The significance of this event is not assessed by comparing machine performance against existing human benchmarks, as human ingenuity can often surpass machine performance on known tasks. Instead, the critical shift lies in the possibility that an artificial intelligence system may have generated genuinely new mathematical knowledge, potentially solving a previously unsolved Millennium Prize problem. This distinction moves the focus away from who produced the proof toward the profound implications of such a capability.

The author reflects on a previous view of AI, which characterized it as merely a stochastic parrot, an advanced search engine, or a next-token predictor. This perspective was rooted in the understanding that AI operates within the boundaries of human-known problems. However, the potential for AI to discover novel solutions alters this framework entirely. If such a capability is accurate, the trajectory of AI development shifts from assisting humans with existing knowledge to actively generating new knowledge at the frontier of mathematics and science. This scenario suggests a very rapid acceleration if major AI laboratories compete to demonstrate such novel discoveries, raising concerns about the pace of change.

The author posits that if the foundational assumptions regarding the proof—its correctness, independent verification, and novelty—hold true, the description of AI as simply a tool is no longer sufficient. While traditional tools like calculators execute known specifications, a system capable of discovering entirely new solutions represents something fundamentally different. This realization leads the author to contemplate a potential future crisis stemming from this advancement. If AI transitions to a role of producing novel knowledge, systems currently structured around human intellectual scarcity, such as education, research paths, and career structures, may face significant pressure. There is a concern that the very definition of what constitutes human intelligence may be challenged and redefined more rapidly than societal structures can adapt.

Ultimately, the reflection serves as an appeal for humanity to navigate this transition. The author expresses hope that humankind can manage this evolution to ensure that these new capabilities serve to improve human life rather than exacerbate existing inequalities or leave segments of the population behind. The event is viewed as marking the beginning of a profoundly different relationship between humans and machines, demanding careful scrutiny regarding the nature of machine capability and its societal consequences.