Human children long stood alone as the only entities that can learn a language to native fluency. Modern large language models (LLMs) — such as OpenAI’s GPT family, Anthropic’s Claude and other contemporary systems — can now produce fluent, flexible language. Yet they require vastly more training data than children do: current LLMs are typically pretrained on many orders of magnitude more text than a child hears while acquiring their mother tongue.
The data efficiency gap and why it matters
This gap — often called the data efficiency gap — raises both scientific and engineering questions. Understanding how children reach linguistic competence from far less input could enable more data-efficient AI, easing training for multimodal tasks and improving performance in minority-language settings. It would also let cognitive scientists test enduring hypotheses about how humans acquire grammar and whether innate biases are required.
How big is the gap?
A child in a language-rich home may hear roughly 100 million words by preteen age; including literacy, that might rise to around 300 million words by age 20. By contrast, Meta’s Llama 3.1 reportedly consumed about 15 trillion tokens during pretraining, and frontier models may use ten times as much data. To make the scale concrete: if you printed the words used to train a modern LLM, the paper stack could exceed the International Space Station; a preteen’s 100 million words would form only about a 20-meter-high stack.
Theoretical background: Chomsky and statistical learners
Noam Chomsky argued in the 1950s that children require innate grammatical knowledge because the linguistic input is too impoverished for learning syntax purely from experience. That debate with behaviorist perspectives (e.g., B.F. Skinner) shaped decades of linguistics and early AI. Neural networks that learn statistical patterns seemed unlikely, but transformers and large-scale data in the 2010s changed that: models such as BERT and GPT-2 (and later ChatGPT) demonstrated that extensive statistical learning can produce syntactic competence of a sort, even though LLMs are fundamentally non-biological statistical learners.
BabyLM: training models at child-scale
In August 2022 Alex Warstadt and colleagues launched the BabyLM competition to probe what models can learn from developmentally plausible amounts of data. BabyLM provides corpora of about 100 million words for the main track (10 million for a toddler-scale track), sourced from storybooks, dialogue, movie subtitles, Simple English and regular Wikipedia, and child-directed speech transcripts. Models are evaluated with grammar benchmarks similar to those used in psycholinguistics.
Findings surprised some expectations. Curriculum learning — presenting data from simple to complex, mirroring human teaching methods — was widely used but did not bring the expected advantages. The 2024 BabyLM winner, GPT-BERT, combined next-token prediction and masked-language objectives and, after pretraining on roughly 100 million words, outperformed Meta’s Llama 2 70B on at least one BabyLM benchmark despite Llama 2 70B’s vastly larger pretraining corpus. Nonetheless, BabyLM models are not on par with full-scale LLMs: many do not generate fluent text, and those that do remain far less capable than commercial models.
What might models be missing?
Children do not learn from text alone. They experience the world through vision and hearing, actively explore, experiment, and learn socially. Developmental research indicates that children select and create informative experiences, they know what they don’t know and seek to fill gaps, and they interpret information differently when an adult is trying to teach them. Models, by contrast, learn passively from static corpora.
Multimodal training on child-centered video has been explored but remains challenging. Brenden Lake and colleagues trained a model on 61 hours of SAYCam headcam footage and showed it could learn simple object-word associations, but it did not attain two-year-old linguistic competence. One obstacle is that existing headcam datasets capture limited slices of children’s lives (a few hours per week), so they may lack the continuity and scope of actual childhood experience.
Larger, more realistic datasets
Newer efforts aim to collect far denser naturalistic data. Uri Hasson’s Princeton project recorded roughly 12 hours per day in communal living spaces for 17 children across their first 1,000 days, producing a dataset that would have been infeasible to analyze before advances in transcription and video processing. Such datasets provide the kind of input some researchers argue is needed to close the data gap.
Motivations and potential applications
Reducing the data efficiency gap has practical incentives: it could democratize AI research (allowing institutions without hyperscale resources to train competitive models), support smaller languages with limited corpora (for which available tokens may be on the order of tens of millions), and decrease reliance on massive internet-scale datasets. For instance, David Samuel and collaborators worked on GPT-BERT partly to improve models for languages with small data budgets.
Scientific payoff: learning about ourselves
Beyond engineering benefits, studying models trained on child-scale or multimodal data can inform theories of human language acquisition. Although brains and LLMs differ fundamentally — brains are embodied, living, and continually plastic — researchers such as Michael Frank, Alex Warstadt, Richard Futrell, Brenden Lake and others are using models as experimental stand-ins to test hypotheses that would be difficult or impossible to test on children. When models achieve linguistic behaviors once thought impossible without innate structure, that challenges assumptions and stimulates new lines of inquiry.
Conclusion
Modern LLMs demonstrate striking linguistic abilities, but they do so after ingesting staggering quantities of data. Children reach comparable milestones from far less input. Ongoing efforts — from BabyLM competitions to large-scale headcam corpora and multimodal modeling — aim both to build more data-efficient AI and to shed light on human language acquisition. Progress on the data efficiency gap may yield both practical technological gains and deeper scientific understanding of how people learn language.
