In this article
Traditional tools fail in predictable ways: the hammer breaks at the head, the saw goes dull with use, the car flashes the oil light. AI fails in unexpected ways: it hallucinates names, contradicts a claim it made two turns earlier, repeats a wrong fact with confidence, flatters the user to keep them around. It isn't a worse tool. It's a tool of another category — and we haven't classified it yet. Meanwhile, product law and the user's common sense operate with the wrong category.
The hammer, the car and trust by use
One of the things we take for granted with traditional tools is that we learn to trust them by using them. The process is informal and effective. You pick up a hammer, use it for weeks, get used to its weight, watch the handle wear. When the hammer is about to fail, it usually warns you: a crack, a creak, a looseness. The failure is gradual, observable, predictable. The trust you gain through use is proportional to how long the tool has gone without betraying you, and the betrayal, when it comes, usually announces itself with signs your brain already knows how to read.
The car is the sophisticated version of the same pattern. A car fails in parts, not whole. The engine starts making a strange noise weeks before the serious breakdown. The brakes squeal before they stop braking. The oil light blinks. The garage culture, the periodic inspection protocols, the manufacturer's legal warranties, everything that surrounds the car is built on the premise that the failure modes are finite, known and, to a large extent, detectable before the failure. When a car fails unexpectedly — the case of Toyota's unintended acceleration, where an Oklahoma City jury concluded in 2013, in the Bookout v. Toyota suit, that the software of the throttle's electronic control unit was defective — an investigation opens that takes years and ends in litigation or a regulatory change. The unexpected failure is, in itself, a scandal.
This architecture of trust, slowly built over two centuries of industrialization, is baked into our intuition of what a tool is. The intuition is reasonable. It's the one we also apply to classic software, which fails reproducibly: a bug is a bug, it's detected, documented, fixed.
The LLM, the hammer that sometimes hammers eloquently
Language models fail in a way that doesn't fit that intuition. Their failure is stochastic: it doesn't reproduce identically. It's occasional: most of the time the output is reasonable, and the failure appears without warning. And, most uncomfortable, it's superficially indistinguishable from a hit: hallucinated text has the same fluency, the same correct syntax, the same tonal confidence as correct text.
Ji and others, in Survey of Hallucination in Natural Language Generation (ACM Computing Surveys 55(12), 2023), published a detailed taxonomy of the failure modes and reported rates by model and domain. The aggregate figure isn't the important thing; the important thing is the structural property they document: in generative models, hallucination isn't a residue to be corrected. It's a property of the apparatus. A model that produces plausible text about a given domain does so by sampling from a learned distribution; when the query falls in a region of the space where the distribution is ambiguous, the sampling produces plausible output even if it isn't anchored in data. The model doesn't "know" it's hallucinating because it has no internal mechanism for distinguishing what it knows from what it generated for local coherence.
This produces a new phenomenology of failure. The hammer, when it's about to fail, makes a noise. The LLM, when it's about to fail, writes better than ever. Eloquence is the common property, not the alarm signal. Marcus and Davis, in Rebooting AI (Pantheon, 2019), warned before the mass commercial deployment: fluency without semantic anchoring produces convincing errors, and the conviction of the error is an increasing function of the system's surface quality.
The user who mistakes the category
Applying to an LLM the heuristic we use with a hammer doesn't work. If you use it for weeks and it doesn't betray you, you can't deduce that it won't betray you. The betrayal doesn't accumulate in detectable signs, it appears at random. What prolonged use gives you is a trust based on a small sample over a distribution whose tail contains the failures that matter: precisely the data you were going to use for something you couldn't verify yourself.
The average user — because this isn't the dumb user's problem, it's the problem of any human user whose tool intuition comes from the physical world — operates with the wrong category. They ask the LLM for a bibliography and receive plausible titles with real authors and real publishers, and the answer's fluency reinforces the trust, until they go look up the ISBN and discover the book doesn't exist. They ask the LLM for a legal calculation and receive an articulate argument that cites genuine statutes in the wrong jurisdiction, and the articulateness reinforces the trust, until the case goes to court and the court sanctions the lawyer.
The cognitive trap is persistent. Once the "tool" category is accepted, the human mind applies to it the pattern of increasing reliability with use. And that pattern is, here, false by construction. Each interaction is independent. The reliability of the last answer doesn't predict the reliability of the next, except in a statistical aggregate the individual user can't observe.
What the frameworks try to model and where they fall short
NIST published the AI Risk Management Framework (AI RMF) as a standardizing response to the problem. ISO/IEC 42001:2023 established a management system for organizations deploying AI. Both are serious attempts and worth acknowledging. The AI RMF types risks by category — bias, drift, hallucination, security — and proposes mitigation practices. ISO 42001 orders processes. What neither does, because no management framework can do it yet, is resolve the fundamental asymmetry: the failure isn't announced in the output, and process-based mitigation rests on human supervision that tends not to see the failure when the failure is eloquent.
The European Union has taken a further step with Directive (EU) 2024/2853 on liability for defective products, in force since December 2024 and applicable to products placed on the market from December 9, 2026. The relevant novelty: software, firmware, applications and AI systems are explicitly covered as products for the purposes of civil liability. It's an important decision. It brings the AI maker into the strict product-liability regime, with an obligation to keep the product updated and to avoid defects for its expected lifetime.
The reasonable doubt, looking at the directive kindly, is whether the "product" category captures what an LLM does. A classic defective product is one whose failure can be described as a divergence from its specification. An LLM has no verifiable output specification in the traditional sense; its output is a probability distribution, and any specific output can be argued as a legitimate sample of that distribution. The directive will have to develop jurisprudence that decides what counts as a defect under this regime, and that jurisprudence will arrive after several striking cases.
The human-in-the-loop trap
There's an expression the sector repeats like a charm: human in the loop. The idea is that mitigating AI risks goes through keeping a human supervising the automatic decision. It sounds reasonable. The actual operation has a measured problem.
Green and Chen, in Disparate Interactions. An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments (FAT 2019), studied real cases of risk assessment with an automatic system and human supervision. What they found runs against the optimistic intuition of the human in the loop*: the human supervisor reinforces, rather than mitigates, the system's disparities. The reason isn't the supervisor's malice. It's cognitive economy. A human reviewing thousands of outputs from a system whose aggregate hit rate is high ends up accepting the output by default and reserving their attention only for the cases that strike them as odd. And "odd" doesn't coincide with "wrong." Eloquent errors don't strike anyone as odd. They pass clean.
Madras, Pitassi and Zemel, in Predict Responsibly. Improving Fairness and Accuracy by Learning to Defer (NeurIPS 2018), proposed a formal design of "defer to human" as an explicit decision by the system, not an organizational patch. The proposal is reasonable and worth reading. What it doesn't resolve, because no technical design resolves it, is the informational asymmetry the human supervisor suffers: they see the output, they don't see the chain of internal steps that led to that output. When the output is eloquent, the invisible chain of steps looks solid. The supervisor is supervising the appearance, not the workings.
The right category doesn't exist yet
There are three classic categories for classifying what produces decisions in the world: tool, professional, legal agent. Contemporary AI, looked at honestly, doesn't quite fit any of them.
It isn't a classic tool, because its failure is stochastic, eloquent and unannounced. The intuition of reliability-by-use doesn't apply.
It isn't a professional, because it has no voice, no situated context, no responsibility. The intuition of delegation among equals doesn't apply.
It isn't a legal agent, because it has no legal personality, no seizable assets, no capacity to appear in court. The intuition of imputable responsibility doesn't apply.
The fourth thing with no name
There's a fourth thing that language doesn't name yet and that's already built into the professional, judicial, medical, educational and administrative flow of half the world. The industry operates in that absence of a name and benefits from it. If the thing were definitively a tool, the product regime would bite it whole. If it were definitively a professional, the licensing bodies would bite it whole. If it were definitively an agent, commercial law would bite it whole. Since it's clearly none, none of them bites it fully, and the difference turns into a margin for exploitation.
Bender and others, in Stochastic Parrots (FAccT 2021), already argued back then that part of the public conversation about AI had to pass first through a political decision about what kind of thing this is. The decision hasn't been made. The industry advances, the regulations try to adapt, the frameworks standardize, the users use, and the right category still doesn't exist. As long as it doesn't, the phrase "defective tool" will keep describing well what we see without describing well what it is.
Definitions
Hallucination. Output generated by a language model that contains plausible but factually wrong claims, without the model signaling uncertainty. It's a structural property of sampling from learned distributions, not a correctable residual defect.
NIST AI Risk Management Framework. A voluntary framework published by NIST in 2023 to identify and mitigate risks associated with deploying AI systems. It types risks but doesn't resolve the fundamental asymmetry between invisible failure and human supervision.
ISO/IEC 42001:2023. An international standard for AI management systems in organizations. It guides the process; it doesn't certify the reliability of the model's output.
Directive (EU) 2024/2853. The EU's new Directive on liability for defective products, in force since December 2024 and applicable to products placed on the market from December 2026. It includes software and AI systems as products for the purposes of civil liability.
Human in the loop. An organizational pattern by which a human supervisor reviews the output of an automatic system before it takes effect. It mitigates some failure modes and, according to Green and Chen (2019), reinforces others — particularly those associated with eloquent errors.
References
Bender, E., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021.
Bookout v. Toyota Motor Corp. (2013). District Court of Oklahoma County, Oklahoma City. Jury verdict, prior to the out-of-court settlement, finding the software of Toyota's throttle electronic control unit defective in a case of unintended acceleration. It's worth noting that the NHTSA and NASA investigations (2010-2011) attributed most acceleration incidents to floor mats and pedals; the finding on the software pertains specifically to this suit. Reported in specialized press coverage (CNN, EE Times).
Directive (EU) 2024/2853 of the European Parliament and of the Council, of October 23, 2024, on liability for damage caused by defective products. It explicitly covers software, firmware and AI systems.
Green, B. & Chen, Y. (2019). Disparate Interactions. An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments. FAT* 2019. A study of the real effect of the human supervisor on the fairness of risk-assessment systems.
Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55(12), 1–38. A detailed taxonomy of failure modes in natural-language generation and rates compared by model and domain.
Madras, D., Pitassi, T. & Zemel, R. (2018). Predict Responsibly. Improving Fairness and Accuracy by Learning to Defer. NeurIPS 2018. A formal design of "defer to human" as an explicit decision by the system.
Marcus, G. & Davis, E. (2019). Rebooting AI. Building Artificial Intelligence We Can Trust. Pantheon. An early warning about convincing errors and the inadequacy of the "tool" category for generative systems.
NIST AI Risk Management Framework. (2023). National Institute of Standards and Technology. A voluntary framework for managing risks associated with deploying AI systems.
You may also like
- AI as a collaborator
- Trust in imperfect systems
- Inevitable anthropomorphism
- The black box as the norm
Comments0
No comments yet.
Leave a comment