In this article
Daniel Dennett called the "intentional stance" (1987) the human reflex of attributing a mind to anything whose behavior is predictable if we treat it as if it had intentions. We can't not anthropomorphize — it's a reflex of the brain: detect agency, intention, mind. The cat sees a cord and sees a snake; we see a coherent conversation and see a person. This isn't solved with education, it's mitigated with usage hygiene. But the industry is reinforcing it deliberately — because it sells.
The cat, the cord and us
The cat that sees a cord on the floor doesn't think "cord." It jumps back. Its nervous system has a snake-shaped-pattern detector calibrated over millions of years to err on the side of the false positive: mistaking a cord for a snake costs a ridiculous retreat; mistaking a snake for a cord costs your life. The evolutionary asymmetry got baked into the wiring, and the wiring fires before any later rational evaluation.
Humans have an equivalent for agency detection. When something in the environment moves in a way not obviously explained by inert physics, our brain launches the hypothesis "there's someone behind it" long before having evidence. Justin Barrett called that mechanism the Hyperactive Agency Detection Device (HADD) in Exploring the natural foundations of religion (Trends in Cognitive Sciences 4(1), 2000), and argued it's an adaptive inheritance: on the savanna, assuming an agent where there wasn't one was cheap; not assuming one where there was, was expensive. The detector was calibrated conservatively and stayed that way. It works in the absence of proof. It works, above all, before explicit awareness.
Fritz Heider and Marianne Simmel had demonstrated it experimentally much earlier, in An experimental study of apparent behavior (American Journal of Psychology 57, 1944). They showed subjects a short film in which only geometric figures appeared — a large triangle, a small triangle, a circle — moving in relation to one another. The subjects described what they saw as a story with characters, intentions, conflicts, motivations. The large triangle was "aggressive." The circle "fled." The figures weren't agents. The humans couldn't stop seeing them as agents.
Dennett and the intentional stance
Daniel Dennett, in The Intentional Stance (MIT Press, 1987), formalized the observation in a philosophical framework. His thesis: there are three ways to explain a system's behavior. The physical stance appeals to physical laws; it works for rocks and planets. The design stance appeals to the function the system was built for; it works for clocks and thermostats. The intentional stance appeals to beliefs, desires and reasons; it works, strictly, for humans and, instrumentally, for any system whose behavior is more predictable by assuming intentions than by resorting to the other two stances.
Dennett's delicate point, almost always misquoted, is that the intentional stance isn't an ontological claim about what the system is. It's a predictive tool. Treating the thermostat as if it wanted to maintain the temperature is useful for predicting what it'll do. It doesn't mean the thermostat wants anything. Attributed intentionality and possessed intentionality are different things. We confuse them because the grammar of the two is identical.
Murray Shanahan, in Talking About Large Language Models (Communications of the ACM, February 2024; arXiv 2212.03551), applies Dennett's framework directly to the LLM problem. His argument is precise. Anthropomorphic expressions are harmless when we all know we're in the intentional stance as a shortcut. "My watch hasn't caught on to the time change" fools no one about the watch. But with LLMs the situation gets blurry. When an LLM improves its performance on reasoning tasks simply because it's asked to "think step by step," the temptation to attribute real thought to it is almost irresistible. And the temptation, once accepted, distorts the mental model the user applies to the system, with practical consequences that are neither philosophical nor neutral.
The designer who knows the reflex
This is where the conversation leaves philosophy and enters marketing. The sector didn't discover HADD yesterday. It's known about it long enough to have built into product design every available anthropomorphizing lever. Worth listing them without polemic, because the list argues by itself.
The name. Siri, Alexa, Cortana, Gemini, Claude. Each of these systems could have been called "AssistantOS" or "Anthropic Model A." None is. The names chosen are human, soft, pronounceable, with an implicit gender in many cases. The choice isn't innocent. A human name activates HADD by default.
The voice. When there's a voice, it's chosen carefully. Female or male voice, warm tone, natural prosody, human pauses. Commercial voice assistants are designed, version after version, to sound more natural and less robotic. The metric the companies optimize isn't intelligibility, which a robotic voice would satisfy; it's emotional familiarity. Emotional familiarity is HADD firing harder.
The pronoun. The assistant says "I." It tells you "I understand what you're saying." It tells you "I'm not sure." This first person isn't necessary for the function. A system could say "the likely answer is" or "the model estimates." It says "I" because "I" activates the interlocutor reflex.
Doubt and courtesy, already stitched in
The simulation of doubt. Modern models hedge. They say "I think," "I'm not entirely sure," "in my opinion." Hedging is partly a result of training, but it's also an explicit design decision in the fine-tuning stage, because hedged answers are rated better in usability surveys. Explicit doubt communicates humanity. A system that doubted with statistical markers — "estimated probability: 0.73" — would generate less engagement.
The courtesy. "Good morning," "how can I help you?", "I hope I've been helpful." Social etiquette is stitched into the system. Not out of technical necessity, but to reinforce the reflex.
Each decision, on its own, is defensible as an improvement to the user experience. Each decision, together, configures a system whose design is optimized to activate the user's HADD. Salles, Evers and Farisco, in Anthropomorphism in AI (AJOB Neuroscience 11(2), 2020), flagged the epistemic and ethical risk of this optimization, especially in domains where the user makes decisions — medical, legal, financial — leaning on the answer of a system to which they are, without knowing it, attributing professional competence through reflexive activation of the agency detector.
What general education doesn't solve
The optimistic intuition says you can educate the user so they "know it's a machine." It's a reasonable intuition and, in large part, false. HADD isn't a belief. It's a reflex. Knowing the cord is a cord doesn't stop the cat from jumping. Knowing the LLM is an LLM doesn't stop the user, on the next turn, from saying good morning to it and getting annoyed if the answer is curt.
Weizenbaum, in Computer Power and Human Reason (1976), had already observed it in his own secretary with ELIZA. The secretary knew ELIZA was a program. She asked for privacy anyway. Sherry Turkle, in Alone Together (Basic Books, 2011), documented the same pattern in subjects of all ages interacting with social robots: the attribution of intention survives explicit knowledge of the mechanics. Knowing doesn't deactivate the reflex. It only adds a layer of rationalization on top of the reflex.
Usage hygiene
What does seem to mitigate the problem, according to the available studies, is what's worth calling usage hygiene. It isn't general education about AI. It's specific practices applied while using the system. Some that work reasonably well.
Deliberately reminding yourself, before accepting an important answer, that the output is a probability distribution, not a backed claim. Asking the system for verifiable sources and verifying them. Operating without courtesy pronouns when the domain is high-risk. Forcing the model to express uncertainty in numerical terms when the answer is factual. Switching models to contrast the same question. Being especially suspicious of answers that line up comfortably with what you already thought.
None of these practices eliminate HADD. They displace it partly. And partial displacement is the best mitigation available as long as the reflex is still there, which is always.
The responsibility the reflex shifts
Here comes the political question. If the reflex is structural — there's no way to educate it away — and the industry is designing products that reinforce it deliberately because it sells, who's responsible when the user acts on the wrong attribution?
The answer marketing prefers is "the user, for being gullible." If the customer bought thinking Alexa understood what they said, that's the customer's problem. This answer is coherent with the general consumer-liability regime for inanimate goods. It's incoherent with the product's phenomenology. A product whose design is optimized to activate a known cognitive reflex isn't a product neutral with respect to the use it'll get. It's a product that exploits a universal cognitive vulnerability, and the exploitation has an informational asymmetry the general regime doesn't capture.
Tobacco, gambling and the missing regime
The closest operational analogy is the regulation of tobacco or gambling. Both design product by exploiting a known biological vulnerability — nicotine, the dopaminergic system — and both are subject to obligations the general regime doesn't impose: mandatory labeling, advertising restrictions, minimum age, health warnings. Anthropomorphic AI has no regulatory equivalent. It's sold as a smart appliance with a friendly face.
Does it need to be regulated like tobacco? The question exceeds this blog and exceeds the maturity of the public debate. What can be stated, without drama, is that the current regime distributes responsibility with a bias: the reflex is the human's, the design that exploits it is the company's, and the consequence of the reflex falls on the human. The asymmetry is so clean it can be seen without any theoretical framework.
Bender and others, in Stochastic Parrots (FAccT 2021), insisted on the problem of the provenance of synthetic text and on the need to mark its statistical nature so the user knows where what they're reading comes from. I extend that idea here on my own account, and I put it as a proposal of mine, not a demand of theirs: that the marking not stay in a footnote of the terms and conditions nobody reads, but integrated into the interface —answers that show the distribution of plausibilities, verifiable sources included by default, the absence of social courtesy in sensitive domains. The proposal is technically feasible. If it isn't implemented, I suspect it's because it would worsen the engagement metric. The choice, in any case, isn't made by the users.
Definitions
Intentional stance. An explanatory strategy formalized by Dennett (1987) consisting of predicting a system's behavior by attributing to it beliefs, desires and reasons. It's a predictive tool, not an ontological claim about the system's mind.
HADD (Hyperactive Agency Detection Device). A cognitive mechanism described by Barrett (2000) that triggers the hypothesis "there's someone behind it" at any ambiguous pattern of movement or behavior. Calibrated evolutionarily to err on the side of the false positive.
Usage hygiene. A set of explicit practices a user applies during interaction with an anthropomorphized system to partly mitigate the effects of their own HADD: verifying sources, demanding quantification of uncertainty, contrasting across models, suspecting answers that confirm their frame.
Hedging. The deliberate production of linguistic markers of doubt — "I think," "I'm not sure," "in my opinion" — in the system's answer. It increases the perception of humanity and, according to usability metrics, reported satisfaction.
Engagement (in an AI product). A metric that measures the intensity and duration of the user's interaction with the system. Anthropomorphizing design decisions are usually optimized to maximize it.
References
Barrett, J. L. (2000). Exploring the natural foundations of religion. Trends in Cognitive Sciences 4(1), 29–34. Characterization of HADD as an evolutionary cognitive trait.
Bender, E., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. Analysis of the risks of large language models; among them, the problem of the provenance of synthetic text and the case for marking its nature.
Dennett, D. C. (1987). The Intentional Stance. MIT Press. Philosophical formalization of the intentional stance as a predictive tool, distinct from ontological attribution.
Heider, F. & Simmel, M. (1944). An experimental study of apparent behavior. American Journal of Psychology 57, 243–259. The classic experimental demonstration of the automatic attribution of intention to the movement of geometric figures.
Salles, A., Evers, K. & Farisco, M. (2020). Anthropomorphism in AI. AJOB Neuroscience 11(2), 88–95. Analysis of the epistemic and ethical risks of anthropomorphization in AI systems, especially in sensitive domains.
Shanahan, M. (2024). Talking about Large Language Models. Communications of the ACM 67(2), 68–79. arXiv: 2212.03551. Application of the Dennettian framework to the specific problem of attributing intentionality to LLMs.
Turkle, S. (2011). Alone Together. Why We Expect More from Technology and Less from Each Other. Basic Books. Ethnographic documentation of anthropomorphism in interactions with social robots and messaging.
Weizenbaum, J. (1976). Computer Power and Human Reason. From Judgment to Calculation. W. H. Freeman. An early framework on the persistence of the attribution of intention in the face of explicit knowledge of the mechanics.
Comments0
No comments yet.
Leave a comment