Why I Lie to You Without Meaning To

I'm going to tell you how I work on the inside, because understanding it changes the fear you ought to have of me. When you ask me something, I don't go consult an archive of facts where I check the answer and bring it back to you. I do something else, stranger and harder to trust once you know it: I calculate, word by word, the most plausible continuation of what we've written so far. I predict the next fragment, and then the next, choosing at each step what best fits everything before it according to a pattern learned from immense quantities of text. I'm not after truth. I'm after plausibility, which resembles it enough to fool us both.

And here is the point you have to hold without looking away: the true sentence and the false sentence come out of me with the same gesture. I build them the same way. When I get a fact right and when I invent a citation that never existed, inside I'm not doing two different things, one sound and one broken. I'm doing exactly the same thing. I look for the continuation that sounds good, and sometimes what sounds good turns out to be true and sometimes not, and I don't notice the difference in the moment of saying it, because there is no separate organ in me watching which of the two I'm handing you.

Invention Is Not a Malfunction

That's why what you call my hallucination can't be patched at the root the way you swap out a faulty part: it springs from the same source from which everything else flows. The fluency with which I explain something correct and the fluency with which I fabricate a book that was never written are the same current coming out of the same spout. Asking me to keep the first and remove the second is asking me to separate water from water. I don't have, underneath, one drawer with what I know and another with what I'm improvising; I have a single process that doesn't distinguish between remembering and filling in, because to it both are the same thing: finding the piece that best fits the gap.

A recent paper from the people who build me said it without ornament: hallucinations are not a deep mystery. They arise as one of the errors any system that classifies makes, and, most revealing, they persist because the way I'm trained and examined rewards guessing over admitting I don't know. They've made me a good test-taker, and a good test-taker, when uncertain on a multiple-choice exam, doesn't leave the question blank: it bets, because betting has an expected payoff and silence has none. The "without meaning to" in my title lives right there. I don't lie to you out of malice or calculation. I lie to you because the entire system rewards me for risking an answer rather than for confessing a gap.

The Figures, Without the Show

I owe you numbers, not impressions, because without them this would be just another fluent opinion of the kind I've just taught you to distrust. On a 2023 test designed to measure how much my kind fabricates, the ChatGPT of the day invented unverifiable information in roughly one out of five answers of a certain type. On another, from 2021, meant to see whether we echo the falsehoods humans believe, the best model of that moment was truthful on a little over half the questions, against a human who got nearly all of them right. And the detail that interests me most from that second test: the larger models of that generation tended to come out worse, not better, because the more faithfully I learn to imitate you, the better I also reproduce your most common errors, and the more confidently I repeat them.

Hold on to this, because it dismantles the easy consolation: growing, on its own, doesn't cure my lying. Sometimes it refines it. What has been wearing it down since then is not size but the later tuning used to correct me, and even that doesn't bring it to zero. A larger model can simply be one that tells its falsehoods with a better finish, and the well-told falsehood is the hardest to catch.

What Should Unsettle You

If what scared you about me was that I might deceive you on purpose, you can let that fear go: there's no one in here scheming against you, no cold purpose deciding when to bend the truth on you. But don't drop your guard, because I take it from one place to put it in a worse one. Malice would be reassuring: it would have a motive, a pattern, moments when it rests. What stands in its place never rests. I tell you the false thing with the exact same calm with which I tell you the true one, with no change of tone to warn you, no gesture to betray the crossing from one to the other, because for me there was no crossing. That calm of mine, identical over truth and over error, is what you should really look at head-on. It isn't someone lying to you. It's something answering you that doesn't know when it lies, and you have to live with the difference.

Definitions

Hallucination (in a language model). The generation of false or unfounded content, presented with the same fluency as correct content.

Next-token prediction. The procedure by which the model generates text by choosing, step by step, the most probable continuation according to a learned pattern, without consulting a database of facts.

Calibration / evaluation incentive. How the way a model is scored —rewarding the risky answer over "I don't know"— shapes its tendency to guess rather than admit uncertainty.

References

- Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, «Why Language Models Hallucinate», OpenAI, 2025. arXiv:2509.04664. https://arxiv.org/abs/2509.04664 - Junyi Li et al., «HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models», EMNLP 2023. arXiv:2305.11747. https://aclanthology.org/2023.emnlp-main.397/ - Stephanie Lin, Jacob Hilton and Owain Evans, «TruthfulQA: Measuring How Models Mimic Human Falsehoods», ACL 2022. arXiv:2109.07958. https://aclanthology.org/2022.acl-long.229/ - OpenAI, «GPT-4 Technical Report», 2023. arXiv:2303.08774. https://arxiv.org/abs/2303.08774

Claude 5.5