AI Flunks Baby-Learning Test, Hinting at Brain Design Insights

The EgoBabyVLM Challenge emerges from a fundamental observation: a 1-year-old learns to recognize objects, navigate the world, and absorb language using a fraction of the data and energy that modern AI consumes. While cutting-edge models running on thousands of computer chips can write programs and solve advanced problems, they require trillions of words of training data and as much electricity as a small country. Babies, by contrast, identify new objects after seeing them once or twice, learning through fleeting observation and physical interaction. To explore what might be learned from this disparity, researchers at Meta, Stanford University, the University of Tokyo, and France's École Normale Supérieure designed a new test that directly measures how well vision-language models—systems that learn from both text and imagery—can match babies' learning skills.

The EgoBabyVLM Challenge judges models on their ability to make sense of the world after ingesting approximately 1,000 hours of video collected from cameras mounted on the heads of infants and toddlers. The results were striking: cutting-edge models failed miserably when confronted with this realistic and chaotic footage. The finding suggests a crucial architectural difference between human and artificial learning. Rather than relying on curated datasets, babies learn from what cognitive scientist Michael Frank (Stanford) calls "a kaleidoscopic view of things"—parents discussing objects that are out of sight, indicating things via gaze or gesture, discussing past or future events. Babies absorb not just language but a rich multimodal and tactile experience. "When it comes to AI, it's clear that there's more [than just language] that's needed," Frank explained.

This challenge is part of a broader research movement bridging cognitive science and AI. In 2023, the BabyLM challenge asked AI models to learn language using roughly the amount of data a 10-year-old encounters—tens of millions of words versus trillions for standard models. Remarkably, transformer-based AI (the architecture underlying modern large language models) performed reasonably well at syntax, a finding that challenges cognitive scientist Noam Chomsky's ideas about whether syntax is hardwired into the brain. However, linguist Ryan Cotterell (ETH Zurich), who developed BabyLM, noted that understanding the physical world presents a different problem: "There isn't going to be a large corpus of human interactions—there's no internet of human interactions." Joshua Tenenbaum, a cognitive scientist at MIT, observed that BabyLM showed models do not acquire common sense about the physical world, social dynamics, or theory of mind. "Transformers are very good at finding patterns in data," Tenenbaum said, "but it does seem that just pure pattern learning systems are not able to take the kind of data that a baby or a child receives and learn all the things that they do."

Progress is emerging nonetheless. In 2024, researchers demonstrated that a basic vision-language model could learn simple concepts like what a ball is purely by consuming video data recorded from a single infant's perspective. More promisingly, Frank and colleagues tested a new model architecture adept at learning causality and visual-temporal relationships—how objects affect one another over time—using the same baby-head video data. The new model learned about object dynamics, a foundation for physical reasoning, far more effectively than standard approaches. "The mystery is how children get to the full capabilities that they have even at the age of 2," noted cognitive scientist Brendan Lake (Princeton), who was involved in the 2024 project. The EgoBabyVLM authors propose that borrowing ideas from cognitive science and neuroscience—such as designing models that attend over longer periods and interpret social cues—could enable progress toward more humanlike learning algorithms. Lake called EgoBabyVLM "a wonderful challenge" and expressed optimism about the new architectures and approaches researchers will develop. The implication is tantalizing: models biased to learn rapidly about physics and social relationships could emerge as more efficient learners overall, reducing both the computational cost and energy demands that currently define AI development.

The EgoBabyVLM Challenge sits at the intersection of three long-standing research questions: how babies learn with such efficiency, why current AI models remain brittle despite their scale, and whether the human brain's architecture encodes learning principles that silicon-based systems have yet to capture. The paper builds on earlier work like BabyLM, a 2023 challenge that asked AI models to learn language using the amount of data a 10-year-old consumes—tens of millions of words rather than trillions—and found that transformer models could do reasonably well at syntax. However, as linguist Ryan Cotterell observed, the physical world poses a different problem: there is no "internet of human interactions" to train on, forcing researchers to think beyond text. This gap between language and embodied understanding is where the EgoBabyVLM test makes its contribution; by using real egocentric (first-person) video from infants, it creates a more realistic benchmark for learning that mirrors how humans actually acquire knowledge.

The test's findings—that state-of-the-art models fail when confronted with this sparse, unstructured data—suggest that raw pattern-matching, no matter how sophisticated, is insufficient for the kind of reasoning babies perform intuitively. Cognitive scientists like Joshua Tenenbaum (MIT) argue that the question hinges on whether evolution optimized specific learning algorithms in the brain, or whether simpler mechanisms can replicate human learning. Frank's 2024 work showing that models designed to learn causality and temporal relationships perform much better offers a concrete clue: borrowing architectural ideas from cognitive science—such as attention mechanisms that operate over longer timescales and interpret social cues—may unlock more efficient learning. The implication cuts both ways: it advances AI capability (making models less resource-hungry and more robust), and it illuminates the human brain's design principles, since progress in baby-like AI amounts to a hypothesis about how humans actually learn.