Gwern proposes overtraining giant models on small datasets to unlock human-like AI

In 2022, OpenAI published research demonstrating a phenomenon called "grokking." When models are trained on simple datasets like mathematical operations and kept training long after performance plateaus, they suddenly undergo a massive capability leap. The mechanism appears to involve two distinct phases: first, rote memorization of training data compressed into model weights; second, when regularization pressure (such as constraints on weight magnitude) accumulates, the model is forced to discover simpler, more general ways of compressing that data. At the critical moment, the model recognizes that the data can be expressed through the underlying mathematical operation itself, producing an instant jump in capability. OpenAI named this process after Robert Heinlein's neologism for "gaining a deep, intuitive and fundamental understanding."

Gwern, an anonymous but influential researcher with a proven track record of AI prediction, argues that current large language models lack this deeper generalization capacity. While LLMs excel in narrow domains and match human-level performance in specific tasks, they make errors that any human as intelligent as the model would never make—a failure of generalization. His theory rests on a counterintuitive inversion of current practice: frontier AI labs have spent years acquiring ever-larger datasets and training relatively small models on them. Kimi-K3, an open-source baseline for frontier model size, has just under three trillion parameters with fifty billion active parameters—substantial in absolute terms but trainable in days on the largest clusters. Gwern proposes instead training a single hundred-trillion-parameter model on a constrained dataset, forcing the model to ruminate on limited material and discover deeper generalizations rather than simply memorizing novel information.

The reasoning is that when a model has access to unlimited data, it can improve indefinitely by memorizing new examples or drawing simple connections. A model forced to extract all possible insight from a small dataset must keep searching for deeper patterns. A very large model maximizes memorization capacity—each piece of memorized data can serve as raw material for subsequent generalization. The largest existing model is probably Claude Mythos, which is not yet at the hundred-trillion-parameter scale required; the engineering challenges of training at such scale likely remain unsolved. However, frontier labs possess both the resources and technical talent to attempt this.

The political and organizational barriers may exceed the technical ones. A training run following this approach would show zero improvement in test performance for weeks or months while consuming billions of dollars before suddenly succeeding or failing—a level of risk tolerance that may not exist in current corporate structures. Recent AI history has already shifted away from pure scaling: OpenAI's "even bigger version" of GPT-4 underperformed and was released as GPT-4.5 rather than GPT-5. The breakthroughs since have come from reasoning and improved automated reinforcement learning, neither of which appears a plausible path to machine superintelligence. Gwern's proposal represents one of the few genuinely ambitious ideas circulating with a concrete mechanism for achieving transformative capability.

Gwern's essay builds on his established credibility in AI prediction. He published "The Scaling Hypothesis" in 2020, immediately after GPT-3's release, correctly anticipating the trillion-dollar GPU cluster arms race and continued scaling through the decade—predictions made two years before ChatGPT and the broader AI boom. This track record lends weight to his current proposal, even though it contradicts the dominant scaling orthodoxy of recent years.

The post identifies a genuine empirical gap: LLMs demonstrably fail to generalize as broadly as humans, despite matching human performance in narrow domains. The question Gwern raises—why neural networks should be capable of any generalization but stop at the level of current LLMs—is difficult to dismiss on pure principle. The open question is whether language and reasoning contain the kind of deep, discoverable rules that grokking has demonstrated in simple mathematical domains, or whether human generalization relies on architectural features neural networks cannot replicate.

The practical obstacle is as much cultural as technical. A lab would need to commit billions of dollars and engineering resources to an experiment that offers no visible progress for an extended period. The 2024 plateau in pure scaling (where larger GPT-4 variants underperformed) and the subsequent success of reasoning and automated RL suggest the field has already abandoned simple scaling. Whether it will embrace Gwern's radically different approach remains an open question.

No comments yet. Be the first to share your thoughts!