In this article
A copy of a copy loses information. It's classic information entropy. A model trained on model-generated text loses information in an analogous way: the rare tails of the distribution evaporate, the median swells. Each generation of AI trained on the previous one's outputs is a little poorer, a little more averaged. "Model collapse" (Shumailov, Nature 2024) is a measured phenomenon, not a speculative one. And nobody knows how it's going to be stopped.
The photocopy as an exact metaphor
Put a sheet in a photocopier. Make a copy. Photocopy the copy. Photocopy that second copy. Repeat it twenty times. By copy number twenty it's already illegible. Each pass introduced a little noise — toner grain, slight misalignment, loss of contrast — and none of those small deteriorations was recovered. The chain has a single direction: information is lost, not recovered. It's the domestic version of the second law of thermodynamics applied to the communication channel.
This analogy, which in the nineties was used to explain the difference between digital and analogue — digital, they said, doesn't degrade — has become useful again in 2026 for an unexpected reason. Generative models are reintroducing degradation into the digital channel, not because bits are copied with noise, but because each generation of model learns from what the previous generation produced, and "learning" has an inexorable statistical property: it samples toward the mode.
When a language model generates text, it doesn't return the full distribution of the corpus it learned. It returns probable sequences. The improbable sequences do appear, yes, but with a frequency lower than the one they had in the original corpus, because the sampling is tuned — temperature, top-k, top-p — to produce coherent text. The tail thins. If that model's output is fed into the next model's training corpus, the next model learns an already-thinned distribution and produces, in turn, an even more thinned distribution. It's the photocopy, without toner.
What the metric captures and what it doesn't
Shumailov and others, in AI models collapse when trained on recursively generated data (Nature 631, 2024), measured the effect in three families of generative models. The metric that rises across the generations is perplexity on the tail — the successor model's difficulty in reproducing the rare samples of the original human corpus. The metric that holds or even improves is fluency on the mode. The authors documented that the damage accumulates and, most relevantly, that it's irreversible with conventional techniques. Adding more synthetic data doesn't bring back the tail. Adding a little human data cushions, but the loss already produced stays incorporated.
Alemohammad and others, in Self-Consuming Generative Models Go MAD (ICLR 2024), reached the same conclusion by a parallel route in generative image models. They called the phenomenon Model Autophagy Disorder (MAD), by analogy with bovine spongiform encephalopathy, the mad cow disease that appeared when cattle were fed meal from their own species. The parallel isn't metaphorical. It's operational: a system that consumes its own production accumulates structural disorders that don't appear while the cycle is short and that become catastrophic when the cycle closes and iterates.
The part worth retaining is what the dominant industrial metric doesn't capture. The usual benchmarks companies publish to announce each new version's improvement — MMLU, GSM8K, HumanEval, the standard evaluation sets — measure performance on central tasks, not on the corpus's tail. A model that has lost the statistical tail can rise on MMLU and drop on something that isn't being measured: the capacity to generate text outside the median. The metric that rises gets published. The one that drops, nobody publishes because it hasn't been defined.
The cultural median swells
There's a second reading, less technical, worth raising without demagogy. When millions of people use generative models to write emails, draft texts, translate, summarise and produce content, what circulates through the public conversation becomes more like the statistical mode of the training corpus. The speakers' expressive diversity doesn't disappear, but the effectively published diversity does narrow, because the generative filter flattens the extreme. The local register loses to the global register. The idiom loses to the neutral formulation. Rarity loses to the reasonable.
Here it's worth avoiding the elegiac trap. It's not that culture "loses richness" in the abstract. It's that the channel that transmits it favours the median, and culture is transmitted mostly by channel in 2026. The rarity that survives is the rarity produced outside the channel or able to cross it without filtering. That rarity exists and will exist. What changes is the proportion. The centre becomes more visible and the extremes become less audible, not because they're silenced, but because the channel's bandwidth rewards the centre.
Bender and others, in On the Dangers of Stochastic Parrots (FAccT 2021), warned of this tendency before generative models were in mass use. The argument wasn't prohibitionist. It was empirical: when a system trained on a massive corpus becomes a linguistic intermediary for speakers of many varieties, the system's output tends toward the best-represented variety in the corpus, and that best-represented variety drags the cultural biases of the corpus. It's statistical physics, not ideology.
The proposed solutions, and why none is operational
There's a reasonably well-known catalogue of technical and regulatory proposals to halt the degradation. It's worth listing them with their limitations, with no promise.
AI watermarking. Kirchenbauer and others, in A Watermark for Large Language Models (ICML 2023, arXiv 2301.10226), proposed a technique that slightly biases token sampling toward a "green" subset defined by the model's key, so that the generated text can be identified afterwards with statistical methods. The idea is elegant; the practice, fragile. A rewording, a model change, a translate-and-back — all of that degrades the watermark. Besides, watermarks only work if all producers cooperate, and in a competitive market the incentive not to mark is high.
Human-only curated datasets. Some labs maintain and grow internal collections of audited human text, paid to annotators with traceability. The quality is high and so is the cost. Scale is the problem: audited human corpora are measured in billions of tokens; general training corpora, in trillions. The audited human fraction becomes small even if it's very good.
Provenance regulation. The European Union, in the AI Act, foresees transparency obligations on origin and training. China has published regulations on mandatory labelling of generated content. The United States navigates between state initiatives and presidential declarations. None of these regulations imposes a universal technical protocol of verifiable marking, nor can it impose one unilaterally over globally distributed infrastructure.
AI detectors. The companies offering detection of AI-generated text — Originality.AI, GPTZero and the like — claim high accuracy under controlled conditions, but independent studies have documented non-negligible rates of false positives and false negatives. The risk is especially high in flagging non-native speakers' text as artificial: Liang and others measured that several commercial detectors wrongly classified more than half of the essays written by non-natives as artificial. They work as a heuristic, not as a decisive criterion. And the reliability tends to erode as the models improve.
The balance: none operates at scale
The uncomfortable part isn't that the proposals are bad. It's that none, as of today, operates at the scale of the problem. The degradation runs faster than the solutions, and the solutions are neither technically unifiable nor politically agreeable on any reasonable timeline. Kate Crawford walks through, in Atlas of AI, the materiality and economics of datasets — rare-earth mining, annotator wages, opaque licensing contracts — and lets you understand why this problem won't be solved with a technical patch: datasets are an industrial object, not an academic repository.
The difference between eroding and conserving is active
There's a claim worth holding with no way out. Cultural conservation doesn't happen by inertia. It's an operation that costs effort, money, infrastructure and, above all, a decision about what gets conserved. Libraries exist because someone decided they should exist. Archives exist because someone funded them. Museums catalogue because there's human work behind each cataloguing. Linguistic diversity survives where there are active policies sustaining it, and erodes where there aren't.
The generative era raises the stakes. It isn't enough for the human culture produced up to 2022 to keep existing on Common Crawl's hard drive. That production needs to be conserved in a way that is statistically distinguishable from the later contaminated corpus, kept audited, accessible and citable as a baseline. That requires institutions, not good intentions. If the institutions don't appear, the baseline dilutes into the rest of the corpus and ceases to exist operationally, even though its bits stay on a server.
What's the difference between a lost manuscript and a manuscript kept in a basement where nobody knows it's there? Operationally, none. That's what's at stake.
Definitions
Shannon entropy. The classic measure of the information of a probability distribution. When a distribution concentrates on the mode and loses the tail, its entropy decreases: there's less information per sample.
Model collapse. A phenomenon documented by Shumailov and others (2024) whereby training generative models on the output of previous models produces progressive and irreversible loss of the tail of the distribution.
Model Autophagy Disorder (MAD). A term coined by Alemohammad and others (2024) by analogy with mad cow disease to describe the deterioration of generative image models trained on their own outputs without enough fresh human data.
Watermarking. A technique for embedding an identifiable statistical signal in a generative model's output, allowing later detection. Vulnerable to text transformations and dependent on the producer's cooperation.
Human baseline. An identifiable corpus of audited human production, kept as a stable statistical reference for training and evaluating future models. Its maintenance demands active institutions, not inertia.
References
Alemohammad, S. et al. (2024). Self-Consuming Generative Models Go MAD. ICLR 2024. arXiv: 2307.01850. An analysis of the autophagic phenomenon in generative image models and the coining of the term MAD.
Bender, E., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. An early warning about the tendency toward the median in systems trained on massive corpora and used as linguistic intermediaries.
Crawford, K. (2021). Atlas of AI. Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press. General frame on the materiality and economics of training datasets.
Liang, W. et al. (2023). GPT detectors are biased against non-native English writers. Patterns 4, 100779. arXiv: 2304.02819. A study documenting how commercial detectors wrongly classify more than half of non-native speakers' essays as artificial (one detector reached 61.3% false positives on TOEFL essays).
Kirchenbauer, J. et al. (2023). A Watermark for Large Language Models. ICML 2023. arXiv: 2301.10226. A technical proposal for a statistical watermark to identify LLM output, with its known limits in the face of rewording and translation.
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N. & Anderson, R. (2024). AI models collapse when trained on recursively generated data. Nature 631, 755–759. Empirical demonstration of model collapse and of the irreversibility of the distributional tail loss.
You might also like
- The training loop
- The entropy of digital content
- The dataset as ideology
- The future conditioned by old data
Comments0
No comments yet.
Leave a comment