Sutskever and LeCun: Scaling LLMs Won't Yield More Useful Results
Recorded: Nov. 27, 2025, 1:02 a.m.
| Original | Summarized |
Ilya Sutskever, Yann LeCun and the End of “Just Add GPUs” — abZ Global Skip to Content Open Menu Reviews Portfolio Squarespace projects Shopify Projects Angular Projects Services Products Blog Squarespace Blog Shopify Blog Web Development Blog Unboxing Blog About Login 0 0 Contact Reviews Portfolio Squarespace projects Shopify Projects Angular Projects Services Products Blog Squarespace Blog Shopify Blog Web Development Blog Unboxing Blog About Login 0 0 Contact Open Menu Reviews Folder: Back Squarespace projects Shopify Projects Angular Projects Services Products Folder: Back Squarespace Blog Shopify Blog Web Development Blog Unboxing Blog About Login Contact Ilya Sutskever, Yann LeCun and the End of “Just Add GPUs” Nov 26 Written By Sorca Marian When two of the most influential people in AI both say that today’s large language models are hitting their limits, it’s worth paying attention.In a recent long-form interview, Ilya Sutskever – co-founder of OpenAI and now head of Safe Superintelligence Inc. – argued that the industry is moving from an “age of scaling” to an “age of research”. At the same time, Yann LeCun, VP & Chief AI Scientist at Meta, has been loudly insisting that LLMs are not the future of AI at all and that we need a completely different path based on “world models” and architectures like JEPA.As developers and founders, we’re building products right in the middle of that shift.This article breaks down Sutskever’s and LeCun’s viewpoints and what they mean for people actually shipping software.1. Sutskever’s Timeline: From Research → Scaling → Research AgainSutskever divides the last decade of AI into three phases:1.1. 2012–2020: The first age of researchThis is the era of “try everything”:convolutional nets for visionsequence models and attentionearly reinforcement learning breakthroughslots of small experiments, new architectures, and weird ideasThere were big models, but compute and data were still limited. The progress came from new concepts, not massive clusters.1.2. 2020–2025: The age of scalingThen scaling laws changed everything.The recipe became:More data + more compute + bigger models = better results.You didn’t have to be extremely creative to justify a multi-billion-dollar GPU bill. You could point to a curve: as you scale up parameters and tokens, performance climbs smoothly.This gave us:GPT-3/4 class modelsstate-of-the-art multimodal systemsthe current wave of AI products everyone is building on1.3. 2025 onward: Back to an age of research (but with huge computers)Now Sutskever is saying that scaling alone is no longer enough:The industry is already operating at insane scale.The internet is finite, so you can’t just keep scraping higher-quality, diverse text forever.The returns from “just make it 10× bigger” are getting smaller and more unpredictable.We’re moving into a phase where:The clusters stay huge, but progress depends on new ideas, not only new GPUs.2. Why the Current LLM Recipe Is Hitting LimitsSutskever keeps circling three core issues.2.1. Benchmarks vs. real-world usefulnessModels look god-tier on paper:they pass examssolve benchmark coding tasksreach crazy scores on reasoning evalsBut everyday users still run into:hallucinationsbrittle behavior on messy inputsurprisingly dumb mistakes in practical workflowsSo there’s a gap between benchmark performance and actual reliability when someone uses the model as a teammate or co-pilot.2.2. Pre-training is powerful, but opaqueThe big idea of this era was: pre-train on enormous text + images and you’ll learn “everything”.It worked incredibly well… but it has downsides:you don’t fully control what the model learnswhen it fails, it’s hard to tell if the issue is data, architecture, or something deeperpushing performance often means more of the same, not better understandingThat’s why there’s so much focus now on post-training tricks: RLHF, reward models, system prompts, fine-tuning, tool usage, etc. We’re papering over the limits of the pre-training recipe.2.3. The real bottleneck: generalizationFor Sutskever, the biggest unsolved problem is generalization.Humans can:learn a new concept from a handful of examplestransfer knowledge between domainskeep learning continuously without forgetting everythingModels, by comparison, still need:huge amounts of datacareful evals to avoid weird corner-case failuresextensive guardrails and fine-tuningEven the best systems today generalize much worse than people. Fixing that is not a matter of another 10,000 GPUs; it needs new theory and new training methods.3. Safe Superintelligence Inc.: Betting on New RecipesSutskever’s new company, Safe Superintelligence Inc. (SSI), is built around a simple thesis:scaling was the driver of the last waveresearch will drive the next oneSSI is not rushing out consumer products. Instead, it positions itself as:focused on long-term research into superintelligencetrying to invent new training methods and architecturesputting safety and controllability at the core from day oneInstead of betting that “GPT-7 but bigger” will magically become AGI, SSI is betting that a different kind of model, trained with different objectives, will be needed.4. Have Tech Companies Overspent on GPUs?Listening to Sutskever, it’s hard not to read between the lines:Huge amounts of money have gone into GPU clusters on the assumption that scale alone would keep delivering step-function gains.We’re discovering that the marginal gains from scaling are getting smaller, and progress is less predictable.That doesn’t mean the GPU arms race was pointless. Without it, we wouldn’t have today’s LLMs at all.But it does mean:The next major improvements will likely come from smarter algorithms, not merely more expensive hardware.Access to H100s is slowly becoming a commodity, while genuine innovation moves back to ideas and data.For founders planning multi-year product strategies, that’s a big shift.5. Yann LeCun’s Counterpoint: LLMs Aren’t the Future at AllIf Sutskever is saying “scaling is necessary but insufficient,” Yann LeCun goes further:LLMs, as we know them, are not the path to real intelligence.He’s been very explicit about this in talks, interviews and posts.5.1. What LeCun doesn’t like about LLMsLeCun’s core criticisms can be summarized in three points:Limited understanding LLMs are great at manipulating text but have a shallow grasp of the physical world. They don’t truly “understand” objects, physics or causality – all the things you need for real-world reasoning and planning.A product-driven dead-end He sees LLMs as an amazing product technology (chatbots, assistants, coding helpers) but believes they are approaching their natural limits. Each new model is larger and more expensive, yet delivers smaller improvements.Simplicity of token prediction Under the hood, an LLM is just predicting the next token. LeCun argues this is a very narrow, simplistic proxy for intelligence. For him, real reasoning can’t emerge from next-word prediction alone.5.2. World models and JEPAInstead of LLMs, LeCun pushes the idea of world models – systems that:learn by watching the world (especially video)build an internal representation of objects, space and timecan predict what will happen next in that world, not just what word comes nextOne of the architectures he’s working on is JEPA – Joint Embedding Predictive Architecture:it learns representations by predicting future embeddings rather than raw pixels or textit’s designed to scale to complex, high-dimensional input like videothe goal is a model that can support persistent memory, reasoning and planning5.3. Four pillars of future AILeCun often describes four pillars any truly intelligent system needs:Understanding of the physical worldPersistent memoryReasoningPlanningHis argument is that today’s LLM-centric systems mostly hack around these requirements instead of solving them directly. That’s why he’s increasingly focused on world-model architectures instead of bigger text models.6. Sutskever vs. LeCun: Same Diagnosis, Different CureWhat’s fascinating is that Sutskever and LeCun agree on the problem:current LLMs and scaling strategies are hitting limitssimply adding more parameters and data is delivering diminishing returnsnew ideas are requiredWhere they differ is how radical the change needs to be:Sutskever seems to believe that the next breakthroughs will still come from the same general family of models – big neural nets trained on massive datasets – but with better objectives, better generalization, and much stronger safety work.LeCun believes we need a new paradigm: world models that learn from interaction with the environment, closer to how animals and humans learn.For people building on today’s models, that tension is actually good news: it means there is still a lot of frontier left.7. What All This Means for Developers and FoundersSo what should you do if you’re not running an AI lab, but you are building products on top of OpenAI, Anthropic, Google, Meta, etc.?7.1. Hardware is becoming less of a moatIf the next big gains won’t come from simply scaling, then:the advantage of “we have more GPUs than you” decreases over timeyour real edge comes from use cases, data, UX and integration, not raw model sizeThis is good for startups and agencies: you can piggyback on the big models and still differentiate.7.2. Benchmarks are not your productBoth Sutskever’s and LeCun’s critiques are a warning against obsessing over leaderboards.Ask yourself:Does this improvement meaningfully change what my users can do?Does it reduce hallucinations in their workflows?Does it make the system more reliable, debuggable and explainable?User-centric metrics matter more than another +2% on some synthetic reasoning benchmark.7.3. Expect more diversity in model typesIf LeCun’s world models, JEPA-style architectures, or other alternatives start to work, we’ll likely see:specialized models for physical reasoning and roboticsLLMs acting as a language interface over deeper systems that actually handle planning and environment modelingmore hybrid stacks, where multiple models collaborateFor developers, that means learning to orchestrate multiple systems instead of just calling one chat completion endpoint.7.4. Data, workflows and feedback loops are where you winNo matter who is right about the far future, one thing is clear for product builders:Owning high-quality domain dataDesigning tight feedback loops between users and modelsBuilding evaluations that match your use case…will matter more than anything else.You don’t need to solve world modeling or superintelligence yourself. You need to:pick the right model(s) for the jobwrap them in workflows that make sense for your userskeep improving based on real-world behavior8. A Quiet Turning PointIn 2019–2021, the story of AI was simple: “scale is all you need.” Bigger models, more data, more GPUs.Now, two of the field’s most influential figures are effectively saying:scaling is not enough (Sutskever)LLMs themselves may be a dead end for real intelligence (LeCun)We’re entering a new phase where research, theory and new architectures matter again as much as infrastructure.For builders, that doesn’t mean you should stop using LLMs or pause your AI roadmap. It means:focus less on chasing the next parameter countfocus more on how intelligence shows up inside your product: reliability, reasoning, planning, and how it fits into real human workflowsThe GPU race gave us today’s tools. The next decade will be defined by what we do with them – and by the new ideas that finally move us beyond “predict the next token.” Sorca Marian Founder, CEO & CTO of Self-Manager.net & abZGlobal.net | Senior Software Engineer Next ContactAt the moment, the minimum budget for a project is $1k.marian@abzglobal.netFrequently Asked Questions Task Manager with AIPoze haioase |
This document, authored by Sorca Marian, provides a crucial and timely analysis of the current state and future trajectory of Artificial Intelligence, particularly focusing on the perspectives of Ilya Sutskever and Yann LeCun. It meticulously breaks down the shift in AI development, moving away from the dominant "scale is all you need" approach toward a renewed emphasis on fundamental research, architectural innovation, and a deeper understanding of intelligence itself. Here’s a detailed summary of the key arguments: **The Rise and Fall of “Just Add GPUs”:** Marian outlines the historical narrative of AI development, framing it as a period heavily reliant on simply scaling existing Large Language Models (LLMs). From 2012-2020, innovation primarily revolved around experimenting with new architectures and training models on increasingly large datasets, often with limited regard for fundamental understanding. The 2020-2025 era solidified this trend, fueled by the assumption that scaling parameters and tokens would consistently deliver superior performance. This lead to the creation of models like GPT-3 and 4, driving the current AI product landscape. However, this approach is nearing its limits, with diminishing returns on scale and mounting issues with benchmark performance versus real-world reliability. **Sutskever’s Perspective – A Shift Back to Research:** Ilya Sutskever, now heading Safe Superintelligence Inc., argues for a return to a research-driven approach. He identifies three key challenges: 1) **Benchmarks vs. Usefulness:** Current LLMs perform exceptionally well on synthetic benchmarks but struggle with real-world tasks like hallucinations, brittle behavior, and making practical mistakes. 2) **Opacity of Pre-training:** The vast amounts of data used in pre-training are often opaque, making it difficult to diagnose the root cause of problems. 3) **Generalization Remains a Bottleneck:** Despite their capabilities, LLMs still struggle to generalize knowledge to new domains and contexts, falling short of human-level adaptability. Sutskever believes progress will depend not solely on bigger numbers of GPUs, but on fundamentally new training methods and objectives. **LeCun’s More Radical Critique – World Models as the Future:** Yann LeCun, VP & Chief AI Scientist at Meta, takes an even stronger stance, arguing that LLMs *as they currently exist* are not the path to artificial general intelligence. He focuses on four core issues: 1) **Limited Understanding:** LLMs lack a true understanding of the physical world, failing to grasp concepts like objects, physics, and causality. 2) **Product-Driven Dead End:** LeCun sees LLMs as excellent product technologies (chatbots, assistants) but believes they are approaching the limits of their capabilities. 3) **Simplicity of Token Prediction:** The architecture of LLMs, relying solely on predicting the next token, is inherently limited. 4) **Need for Persistent Memory & Planning:** He advocates for “world models” – architectures like JEPA – that learn through interaction with the environment, build persistent memories, and can reason and plan. **Key Takeaways & Implications:** The document powerfully communicates a shift in the AI landscape. The “just add GPUs” approach is fading, acknowledging that simple scaling is no longer sufficient. The primary drivers of future AI innovation will be: 1) Architectural advancements (like world models), 2) More advanced training methods, and 3) A deeper appreciation for fundamental concepts of intelligence. It emphasizes the importance of user-centric metrics over synthetic benchmarks and highlights the need to build pragmatic systems, focused on reliability and useful workflows, rather than chasing scale alone. Marian correctly frames this as a turning point, suggesting that research and theory will once again take center stage. **Strategic Implications for Developers & Founders:** The writing stresses a decreasing reliance on simply purchasing more powerful hardware. Instead, it advises a refocus on building robust UIs, developing reliable data pipelines, and prioritizing practical applications of AI. The shift also implies a broader exploration of novel architectural approaches, potentially leading to specialized AI systems designed for specific tasks like robotics or complex planning. The document effectively positions this as an opportunity for agile startups and smaller companies to differentiate themselves and compete with larger organizations by focusing on intelligent system design rather than simply leveraging the largest available models. It ultimately recognizes an urgent need for engineers and founders to move beyond the current trend and embrace a more nuanced and theoretically-grounded approach to building intelligent systems. The document is a very insightful and timely perspective. |