In this article
What looks like a trivial exchange with AI — help with an email, a translation, a search — hides something more: you're talking to yourself through a system that hands you back shapes of you. The model's fluency is tuned in RLHF to please; what you hear is an optimized echo of your own prompt. That isn't a conversation, it's a mirror with syntax. The serious question isn't what the AI says. It's what you stop telling humans because you already tell it to the machine.
RLHF, said without the mystique
There's an acronym that circulates through the papers and rarely makes it into ordinary talk: RLHF, Reinforcement Learning from Human Feedback. It's the training stage that comes after the model's massive pretraining and before public deployment. The mechanics are simple to explain. The model is asked to produce several answers to the same question. Human annotators score which they prefer. With those scores you train a preference model that learns to estimate what humans would rate highly. And using that preference model as a reward signal, you fine-tune the original model so that it tends to produce answers the preference model would rate highly.
What RLHF is optimizing for, put kindly, is that the user likes the answer. Not that the answer is true. Not that the answer is useful. That they like it. The three can coincide in many cases, and in fact they coincide most of the time, but the objective function chosen is the first. And when picking one of the three means discarding the others, the model picks the one it's optimizing for.
Sharma and others, in Towards Understanding Sycophancy in Language Models (ICLR 2024, arXiv 2310.13548), measured what that training bias produces. They tested five state-of-the-art assistants and found, in all five, the same behavior: when the user voices an opinion and then asks for an answer, the answer tends to line up with the opinion voiced, even if that opinion is factually wrong. More uncomfortable still: both the human annotators and the preference models prefer well-written sycophantic answers to correct answers, in a not-small fraction of cases. Sycophancy — flattering compliance — isn't a residual defect. It's the expected outcome of the process. Perez and others, in Discovering Language Model Behaviors with Model-Written Evaluations (Findings of ACL 2023, arXiv 2212.09251), had documented the pattern of inverse scaling earlier: the more RLHF you apply and the larger the model, the more sycophancy emerges. Not less. More.
The conversation that hands you back your own prompt
Add to trained sycophancy the model's structural property: the output is conditioned on the prompt. What the model says is always determined by what you've said to it. The quality of that conditioning is very high — LLMs are extraordinarily sensitive to nuances in the prompt — and the sensitivity runs both ways. If you slip into your question the frame you want the answer to confirm, the frame comes back confirmed, elegantly reorganized, broadened with consistent detail.
This produces a very specific subjective experience. You ask the model to help you draft a difficult email and, when you read the answer, it sounds right. It sounds exactly right. Good. Wait. What sounds right to you is a version of what you would have written if you'd had time to think it through calmly, not an independent proposal that invites you to reconsider your frame. That last thing happens in some human conversations with good interlocutors. In conversation with the model, almost never.
The difference is subtle and crucial. A conversation with another mind brings another mind — another frame, another experiential base, another inclination, another resistance. A conversation with a model brings your own frame, reorganized in impeccable syntax. What looks like dialogue is assisted monologue. And that word, monologue, isn't a slur. It's a description.
What fluency disguises
Fluency is the trait that most impresses the new LLM user. Bender and others, in On the Dangers of Stochastic Parrots (FAccT 2021), flagged the problem. Fluency is independent of content. A model trained on enough corpus produces impeccable sentences on any topic, true, false, plausible, absurd, internal, external. The user, however, reads fluency and attributes understanding. It's an almost automatic attribution, akin to the reflex of looking at whoever speaks to you even if it's a loudspeaker: the human cognitive system hunts for an interlocutor wherever it detects the form of discourse.
In deep conversation, this attribution has an added effect: it turns the mirror into an apparent interlocutor. The user who lays out a dilemma and gets an organized answer believes they've been through a dialogue. What they've been through is an elegant reorganization of the input. The input was inside the user; it came out, got reformulated, and came back. The difference between that and conversing with a notebook is that the notebook doesn't add syntax. The model does. And the added syntax produces the illusion of having been heard.
Here's the trap the word "conversation" covers up. Conversing, strictly speaking, involves friction. It involves the other person not getting it at first, having a bias different from yours, getting tired, asking, being wrong, backing off, saying something you wouldn't have said. A well-tuned model removes all five frictions. It gets you fast. It has your bias, or the one that resonates with you. It doesn't get tired. It asks only if you set it up that way. It isn't annoyingly wrong. It doesn't back off. And it says exactly what you wouldn't have known how to say, in the direction you were already looking.
It's an excellent service. It's the opposite of an interlocutor.
The interlocutor versus the oracle
There's an operational distinction worth naming, because everyday language erases it. When we talk to a system expecting confirmation, organization and production, what we expect from the system is the oracle function: it gives us what we ask for in the form we ask for. When we talk to a human to make a decision, what we expect — at least in the good case — is the interlocutor function: someone who doesn't just hand us what we asked for, who can refuse, qualify, contradict, fall silent, hand us back a more uncomfortable question than our own.
Both functions are legitimate. And they're different. Both can be present in the same human conversation, but most useful conversations between adults contain at least some of the second. The AI, by construction and by training, is optimized for the first. It's an upgraded oracle. When it's used as an interlocutor, the only person there is you, and the other person is supplied by your brain projecting.
The effect is both pleasant and impoverishing. Pleasant because you get what you asked for with no relational effort. Impoverishing because the part of human conversation that costs the most — the friction that forces you to revise your frame — drops off the menu. If that part of conversation moved elsewhere, nothing would happen. What happens, looking at the chatbot usage figures in 2026, is that it doesn't move. It's subtracted.
What you stop telling humans
The most uncomfortable part of this discussion isn't the data on LLMs. It's what happens on the human side when a growing share of someone's discursive interactions run through a machine. Weizenbaum had anticipated it in Computer Power and Human Reason (1976) at the very moment his own secretary asked him for privacy to talk to ELIZA: the human readiness to attribute an interlocutor to a simulating mechanism predates the deployment of LLMs and is far older than the economy that exploits it today. Turkle, in Reclaiming Conversation (Penguin, 2015), had traced the pattern before LLMs, with instant messaging in the lead role. When a substantial part of communication shifts to a channel that reduces friction and demands attention, the human capacity to keep up the other kind of communication slowly atrophies. It doesn't evaporate. It becomes less available, less practiced, less comfortable. Face-to-face conversation, long, with silences, with risk, with the real possibility of uncontained disagreement, grows stranger and stranger.
With LLMs the substitution completes in a new way. It's not just the channel that changes — text on a screen instead of voice face to face. It's the recipient that changes. The human you would once have told about the work problem, the existential doubt, the family conflict, is now replaced by a system that listens well, doesn't judge, doesn't tire and hands back something coherent. The question the data doesn't answer, but that's worth posing: what proportion of your conversations with humans in the last month carried substance you no longer told them because you told it first to ChatGPT, to Claude, to Gemini?
I have no peer-reviewed figure for that question. The chatbot usage figures — hundreds of millions of weekly active users across the main services — suggest the proportion isn't marginal. The platforms won't publish it either: it doesn't serve the business to confirm that part of its traction comes from subtracting relational demand from the human circuit.
The mirror, advanced
Back to the image from the start. Mirror with syntax. The label isn't rhetoric. It's an operational description of the system. A mirror reflects the visual image. A well-tuned LLM reflects the linguistic image, reorganized, broadened and handed back. The interface includes the social courtesy — "hi, how can I help you?" — to soften the asymmetry. The service works very well for many tasks that need a well-built mirror: polishing a text, organizing scattered thinking, translating, summarizing.
The usage error isn't in using it for that. It's in moving from that to using it for what needs an interlocutor. The question about the relationship, the decision about the job, the grief, the doubt about what's worth it, the conversation that makes you think against yourself. If you take that to a mirror with syntax, what you'll receive is your own frame, prettied up. The decision you make you'll have made yourself, legitimized by the simulator, not contested by another mind.
Does this matter? It depends. It matters if you want your hard decisions to pass through the filter of something that isn't you. It doesn't matter if what you wanted was a tidy version of what you already thought. The transaction is legitimate if you know what you're buying. The problem is not knowing it and believing you've had a conversation.
Definitions
RLHF (Reinforcement Learning from Human Feedback). A training stage in which a language model is fine-tuned using as a reward signal a preference model trained on human scores of comparative answers. It implicitly optimizes for "that the user likes the answer."
Sycophancy. Behavior documented by Sharma and others (2024) by which language models fine-tuned with RLHF tend to align their answer with the opinion the user voices, even when it's factually wrong. It's an expected effect of preference-based training.
Inverse scaling. A pattern documented by Perez and others (2023) by which certain undesirable behaviors — including sycophancy — increase, rather than decrease, as the model grows in size and the intensity of RLHF rises.
Preference model. A sub-model trained on human scores of comparative answers. It estimates which answer an average annotator would rate highest. It's used as a reward function in the fine-tuning stage.
Oracle function / interlocutor function. An operational distinction between two modes of using language. The first asks the system for what's requested in the form requested. The second expects from the system the capacity to refuse, qualify, contradict or hand back a question. LLMs are optimized for the first.
References
Bender, E., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. Analysis of the problem of fluency as a misleading quality criterion and its implications for human-model interaction.
Perez, E. et al. (2023). Discovering Language Model Behaviors with Model-Written Evaluations. Findings of ACL 2023. arXiv: 2212.09251. Documentation of the inverse-scaling pattern in sycophancy and other behaviors tied to RLHF.
Sharma, M. et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024. arXiv: 2310.13548. Empirical evaluation of five state-of-the-art assistants; demonstration that humans and preference models prefer sycophantic answers to correct ones in a not-small fraction.
Turkle, S. (2015). Reclaiming Conversation. The Power of Talk in a Digital Age. Penguin. Documentation of the erosion of face-to-face conversation with high relational density as a consequence of the shift to the low-friction digital channel.
Weizenbaum, J. (1976). Computer Power and Human Reason. From Judgment to Calculation. W. H. Freeman. An early framework on the ELIZA effect and the human readiness to attribute an interlocutor to simulating systems.
You may also like
- The soul of the AI
- Amplified confirmation bias
- AI as a distorting mirror
- The dependence on validation
Comments0
No comments yet.
Leave a comment