What Happens When AI Has Read Everything? The Data Wall, Model Collapse and What Brains Do With Far Less

On 20 August 2026 the data-science blog ar-Raqmi published What Happens When AI Has Read Everything Humans Ever Wrote? Its subtitle sums up the argument: the real bottleneck is not compute, but the finite stock of human wisdom (ar-Raqmi, 2026).

The essay is about data, not brains. It is in our Neuromorphic section because the question it raises has a biological answer that it never mentions. A child learns language from a tiny fraction of the text a language model needs, and runs on a fraction of the power.

The essay's argument

The essay moves in five steps:

1. The hidden bottleneck. The open web was once treated as free, unlimited training data. Now good human text is finite and contested. The essay's references cite Epoch AI's estimate that models will be trained on datasets roughly the size of the whole stock of public human text sometime between 2026 and 2032 (Villalobos et al., 2024).

2. The inbreeding problem. As more of the web is written by models, models train on their own output. The essay argues the distribution narrows, outliers wash out, and errors repeat with more confidence. The research behind this is real. A 2024 Nature paper showed that training indiscriminately on model-generated data causes irreversible defects, with the rare tails of the original distribution disappearing (Shumailov et al., 2024).

3. The synthetic frontier. Synthetic data makes mediocre text cheap, but text as good as an expert's is hard to generate. The essay says the real gate is judging which data is worth training on, a data-engineering skill rather than a hardware one.

4. The wisdom ceiling. Very few people can produce data that would make a model better than the best human experts. The scarce input is not text but elite judgement.

5. ʿIlm and ḥikmah. The essay ends on a distinction from the Islamic tradition. ʿIlm is knowledge, which can be written down and scraped. Ḥikmah is wisdom, which the author describes as the right placing of what you know. Wisdom lives in experience, study and mentorship, and is mostly never written down. Synthetic data can only remix what a model already holds.

The closing warning is about hallucination. A model trained on its own confident errors compounds them.

Where it is strongest, and where it overreaches

The strongest point is about curation. Choosing what to train on, and keeping model-written text from quietly diluting human text, is now a central engineering problem. Our earlier post on Karpathy and model collapse reaches a similar conclusion from a different direction: models drift toward the bland centre unless something restores variety.

The overreach is treating human text as the only source of new knowledge. Where a task gives a reliable signal of success, systems can generate their own experience. AlphaZero reached superhuman play in chess, shogi and Go from self-play alone, without human games (Silver et al., 2018). That is the argument Rich Sutton makes, covered in our reading of the Sutton–Dwarkesh interview. The data wall is a wall for imitation, not necessarily for learning.

The Neuromorphic angle: brains do more with less

The essay's ḥikmah, knowledge gained through experience rather than reading, is also the best description of how brains learn. The numbers make the point:

Michael Frank estimates that language models receive around four to five orders of magnitude more language data than children. The children's figures are deliberately generous upper bounds, and tokens are not exactly words (Frank, 2023).

That gap is a research agenda in its own right. The BabyLM Challenge asks teams to pretrain language models on no more than 100 million words, roughly what a child might hear, to find what makes learning sample-efficient (Warstadt et al., 2023). Frank lists candidate explanations for the gap:

• children's pre-existing conceptual knowledge;

• language grounded in a multimodal world of sights, sounds and touch;

• learning through interactive, social exchanges rather than passive prediction;

• the possibility that children and models are simply evaluated on different things.

The brain is efficient with energy as well as data. It makes up about 2% of body weight but uses about 20% of the body's energy at rest, a budget that is tiny next to a training cluster (Raichle & Gusnard, 2002). Neuromorphic engineering mostly chases the energy side, with event-driven, spiking hardware; our post on why brain-like computers are hard explains the obstacles. The data side is a question of learning algorithms. On both, the brain is the existence proof.

What follows

Read together, the essay and the biology point the same way. The next gains are unlikely to come from scraping harder. They are more likely to come from three things:

• Better curation of the human data that exists, as the essay argues.

• Learning from experience, where the world, not the web, supplies the signal.

• Learning more like brains do: grounded, interactive and sample-efficient.

The essay asks who we trust, what we keep and whom we train on. Neuroscience adds a fourth question: how a learner gets so much from so little.

References

ar-Raqmi. (2026, August 20). What happens when AI has read everything humans ever wrote? ar-Raqmi. https://ar-raqmi.net/post/what-happens-when-ai-has-read-everything-humans-ever-wrote

Frank, M. C. (2023). Bridging the data gap between children and large language models. Trends in Cognitive Sciences, 27(11), 990–992. https://doi.org/10.1016/j.tics.2023.08.007

Meta. (2024, April 18). Introducing Meta Llama 3: The most capable openly available LLM to date. https://ai.meta.com/blog/meta-llama-3/

Raichle, M. E., & Gusnard, D. A. (2002). Appraising the brain's energy budget. Proceedings of the National Academy of Sciences, 99(16), 10237–10239. https://doi.org/10.1073/pnas.172399499

Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), 755–759. https://doi.org/10.1038/s41586-024-07566-y

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., & Hassabis, D. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140–1144. https://doi.org/10.1126/science.aar6404

Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2024). Will we run out of data? Limits of LLM scaling based on human-generated data (arXiv:2211.04325v2). arXiv. https://doi.org/10.48550/arXiv.2211.04325

Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., & Cotterell, R. (2023). Findings of the BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning (pp. 1–6). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.conll-babylm.1