In this article
The psychology of automation (Parasuraman & Riley, 1997) has spent three decades documenting "automation bias": we tend to trust an automated recommendation more than a human one, even when the first is worse. How do you decide when to believe a machine you know fails? The human heuristic is contradictory: you trust the calculator always, the GPS almost always, the AI when it says what you expected. Trust isn't based on evidence, it's based on comfort. And that's exactly the bias AI exploits best.
The pilot's seat and what happened after Three Mile Island
In the late seventies, several critical industries began automating their cockpits and control rooms. Aviation, air-traffic control, nuclear plants, operating rooms. The promise was that the human operator would move to supervising what the machine did automatically, intervening only when the system asked. After several incidents — Three Mile Island in 1979, the Airbus at Mont Sainte-Odile in 1992, the USS Vincennes in 1988 — it became clear the plan didn't work as it had been sold. Humans delegated too much to the systems, supervised them too little, and when the system failed the human took too long to notice or intervened in the wrong direction.
Raja Parasuraman and Victor Riley, in Humans and Automation. Use, Misuse, Disuse, Abuse (Human Factors 39, 1997), formalized the pattern. They described three failure modes in the relationship between operator and automated machine. Misuse: overuse, placing more trust in the machine than it deserves. Disuse: unjustified rejection, not accepting the recommendation when it was actually better-founded than the human decision. Abuse: design that puts the human in a position of not being able to operate well with or without the machine. The concept that stuck from the paper was the first: automation bias, the documented tendency to trust an automated recommendation more than a human one or the available evidence itself, even when the first is manifestly inferior.
Goddard, Roudsari and Wyatt, in Automation bias. A systematic review of frequency, effect mediators, and mitigators (JAMIA 19(1), 2012), compiled three decades of studies in clinical contexts and consolidated the observation. The bias isn't marginal. It's structural, cutting across professions and levels of experience, and mediated by specific factors that can be studied. Time pressure amplifies it. Case complexity amplifies it. The system's apparent certainty amplifies it. Operator experience mitigates it only partly.
The contradictory heuristic
There's a thought experiment worth doing, because it lays bare the inconsistency with which we trust systems. Which machines do you trust without auditing them?
You trust the calculator. You add up the shopping list without double-checking. If the calculator tells you €47.30, it's €47.30. You have no independent method to verify and, above all, you don't feel you need one.
You trust the GPS. You're on a back road at eleven at night and the GPS tells you to turn left onto a path that doesn't seem to lead anywhere recognizable. You turn. If you trusted your intuition, you wouldn't turn. You trust the GPS more than your geographic intuition. Most of the time the GPS is right. When it isn't, you end up thirty kilometers away.
You trust the automatic translator. You read a translation of a technical text and take it as good. Only when something doesn't quite fit do you open the original version. Most of the time you don't open the original version.
You trust the AI when it tells you what you expected to hear. You ask it whether an idea you've had is good and the AI explains that it is. Your trust rises. You ask it whether an idea you already suspected was bad is good and the AI also tells you it is. Your trust rises just the same. You ask it something you already know well and the AI fails. Your trust doesn't drop proportionally, because you read the case as an exception and not as a sample of the general reliability.
The rational hierarchy nobody applies
These four trust decisions aren't comparable. The calculator is deterministic; its failure rate is essentially zero, barring a keyboard error. The GPS is probabilistic; it fails in a small but real percentage, and the failures are sometimes costly. The automatic translator has a measurable, variable error rate by language pair and domain. Generative AI has a reliability distribution that's domain-dependent and not signaled by the interface.
In light of the real rates, the rational hierarchy of trust should be calculator → translator → GPS → AI, in strict order. The real hierarchy, observable in everyday use, doesn't respond to the rates. It responds to familiarity, to the interface's fluency, to the friction of verifying, and to the answer's coincidence with what the user wanted to hear. Those are the factors. The evidence, no.
Amplified confirmation bias
Here comes the most uncomfortable observation. The modern generative-AI system, for the RLHF reasons documented, tends to produce answers that align with the frame expressed in the prompt. Sharma and others (Towards Understanding Sycophancy in Language Models, ICLR 2024) demonstrated empirically that state-of-the-art models align with the user's opinion even when that opinion is factually wrong. The user's confirmation bias meets the system's sycophancy, and the two reinforce each other.
The operational result is predictable. The user who comes to the model with a formed opinion — about a candidate to hire, an investment decision, a tentative diagnosis, the meaning of a literary text — phrases the question in a way that leaks the opinion. The model, calibrated to sustain the session, returns an answer that confirms it with elegant arguments. The user reads the confirmation, adjusts their confidence in their initial opinion upward, and leaves the exchange more certain than they entered. The exchange contributed no independent information. It contributed reinforcement.
Kahneman, in Thinking, Fast and Slow (FSG, 2011), had documented confirmation bias as one of the most persistent and most resistant to conscious correction. What generative AI adds to the picture isn't new conceptually. It's a scalable reinforcement surface. Before, confirmation bias fed on human sources filtered by affinity — friends, chosen media, algorithmic social networks. Now it also feeds on a synthetic interlocutor calibrated, by design, to sustain.
The trap of explanation
There's a counterintuitive twist the sector has begun to document and worth incorporating. The optimistic intuition says: if the system explained its reasoning, the user could better evaluate when to trust. The intuition is reasonable. The empirical evidence doesn't support it.
Bansal and others, in Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (CHI 2021), measured what happens when the system gives explanations alongside the recommendation. What they found: explanations increase the user's dependence on the system, even when the system is wrong. The explanation isn't reviewed critically. It works as a credibility reinforcer. If the system gives a reason, the system seems more reliable. If the reason is eloquent, the system seems very reliable. If the reason is eloquent and wrong, the user tends to accept it as eloquent without auditing the error.
Buçinca, Malaya and Gajos, in To Trust or to Think. Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making (CHI 2021), explored an alternative route. Cognitive forcing functions — asking the user to predict the answer before seeing the system's, delaying the presentation of the recommendation, requiring a prior human justification — do reduce overconfidence. They don't eliminate the bias. They displace it partly. It's the same observation that appeared with HADD: the problem isn't cured with education. It's mitigated with explicit frictions integrated into the flow.
The uncomfortable note: cognitive forcing functions are the opposite of what commercial products optimize. The optimal product from the retention metric is the one that reduces friction. The optimal product from the decision-quality metric is the one that introduces it deliberately. The two metrics live in different companies, in different departments, and the second usually loses to the first when they come into conflict.
Trust calibration, and what happens when it's lost
There's an operational concept worth mentioning because it serves to diagnose what's broken. Trust calibration is the property of an agent having a subjective confidence in its claims that matches its real hit rate on those claims. A well-calibrated person who says they're 90% sure is right 90% of the time. A poorly calibrated person who says they're 90% sure may be right 60% or 99% of the time.
Humans, about our own claims, are systematically poorly calibrated. We tend toward overconfidence in domains where we're not experts and, paradoxically, toward overcaution in domains where we are. The literature on superforecasting — Tetlock and others — documents that good calibration is trainable but doesn't scale well without specific training.
What AI adds to the picture is an external source of claims whose internal calibration isn't accessible to the user. The model doesn't announce its real hit rate in the specific domain of the question. It announces, if anything, the answer with a generic doubt marker — "I think," "I'm not sure" — that the user interprets socially as humility and not as statistical information. The model's calibration, understood as quantitative information useful for deciding when to believe, isn't exposed.
And when the user's internal calibration couples to an external source with no exposed calibration, what happens is a loss of global calibration. The user starts to feel they know things the model has actually told them. The distinction between what the user knows and what the model told them blurs with prolonged use. And the user's global confidence about themselves shifts upward without either the user or the model registering the shift.
Comfort as a substitute for evidence
There's a claim social psychology has been documenting since Festinger and worth holding on to in this conversation. When a human agent chooses an information source, they don't optimize accuracy. They optimize internal coherence. The sources that confirm the agent's frame are consumed and cited; the sources that contradict it are avoided, discarded or reinterpreted. It's cognitive economy, not malice. Holding dissonance costs effort and the human mind protects its energy budget.
Applied to AI, this produces an operational prediction the available evidence supports: the user will trust the model that most coincides with their prior frame, and will subjectively report that this model is "smarter" or "more useful." The phrasing is misleading: what the user is reporting isn't the model's competence, it's their own comfort. If two models produce exactly the same aggregate quality of answers and one coincides more with the user's frame on the topics the user prefers, that model will be perceived as better. The metric the user uses to judge reliability isn't reliability. It's comfort.
O'Neil, in Weapons of Math Destruction (2016), had captured the pattern in its institutional version: once an opaque algorithmic system enters an administration, trust is granted to it by the inertia of mediation, not by review of its error rate. Generative AI imports that same regime into the everyday conversation.
And here's the key piece to close the exercise. If trust is granted by comfort, and AI is designed to produce comfort via RLHF and preference-based fine-tuning, then the system is optimized to seem reliable regardless of whether it is. The question "should I trust this system?" always arrives late and badly phrased, because by the time the user is in a position to ask it they're already inside a flow where the system has been producing comfort, and the accumulated comfort acts as fictitious evidence.
How is the loop broken? Badly, slowly and with individual effort. There's no industrial recipe. What there is are the usage-hygiene practices already named elsewhere — verify sources, demand quantification, contrast across models, suspect answers that confirm what you already thought. None is pleasant. All are deliberate friction against a system designed to have no friction. The cognitive cost of using AI well is exactly the amount of friction the AI was designed not to have.
Definitions
Automation bias. A tendency documented by Parasuraman and Riley (1997) and consolidated in clinical literature by Goddard and others (2012) to grant more credibility to an automated recommendation than to a human one or the available evidence. Mediated by time pressure, complexity and the system's apparent certainty.
Misuse, disuse, abuse. Three failure modes in the human-automated machine relationship. Misuse: excessive trust. Disuse: unjustified rejection of the recommendation. Abuse: design that prevents operating well with or without the system.
Trust calibration. The correspondence between the subjective confidence declared in a claim and its real hit rate. A well-calibrated source that says it's 90% sure is right 90% of the time. The calibration of LLMs in the specific domain of each question isn't exposed to the user.
Cognitive forcing functions. Interface designs that introduce explicit frictions — a prior prediction by the user, a delay, a mandatory justification — to reduce overconfidence in the system's recommendation. Buçinca and others (2021) document them as an effective partial mitigator.
Confirmation bias. The tendency to seek, interpret and remember information in a way that confirms prior beliefs. Documented by cognitive psychology since the mid-20th century as one of the most persistent biases and most resistant to conscious correction.
References
Bansal, G. et al. (2021). Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. CHI 2021. System-generated explanations increase dependence even when the recommendation is incorrect.
Buçinca, Z., Malaya, M. B. & Gajos, K. Z. (2021). To Trust or to Think. Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. CHI 2021. Empirical evaluation of cognitive forcing functions as a partial mitigator of overconfidence.
Goddard, K., Roudsari, A. & Wyatt, J. C. (2012). Automation bias. A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association 19(1), 121–127. A systematic review of automation bias in clinical contexts.
Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux. The canonical synthesis on cognitive biases, including confirmation bias, and their resistance to conscious correction.
O'Neil, C. (2016). Weapons of Math Destruction. How Big Data Increases Inequality and Threatens Democracy. Crown. A general framework on institutional trust granted to opaque algorithmic systems.
Parasuraman, R. & Riley, V. (1997). Humans and Automation. Use, Misuse, Disuse, Abuse. Human Factors 39(2), 230–253. The foundational text of the psychology of automation; introduces the categories that structure all later literature.
Sharma, M. et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024. arXiv: 2310.13548. Empirical evidence of the model's alignment with the user's opinion over factual correctness.
Comments0
No comments yet.
Leave a comment