Suggestibility facing the machine. Fluency is its superpower

Today we're talking about why we believe the machine. My thesis is simple and slightly obnoxious: AI convinces us less by what it says than by how it says it, and it always says it well. But is that a flaw in the machine or a flaw in us? I suspect the second, and that's worse, because the machine's flaws get fixed in the next version. Revising this article, written a few months ago, I had to swallow part of my own argument: the human face gives away far less than I'd assumed, and there are experiments where AI persuasion does exactly what you'd want it to do. Let's see, starting with an experiment involving grey letters.

Black type on white, grey type on yellow

In 1999, Rolf Reber and Norbert Schwarz published an experiment in Consciousness and Cognition so simple it was almost insulting. They showed identical statements with different typographic contrast: some easy to read, others in pale colours that were hard to make out. Participants judged the easy-to-read ones more likely to be true. Truth didn't depend on the colour of the type, obviously. The judgement did.

That's called the fluency heuristic: the ease with which we process something gets used as a shortcut to decide whether it's true, whether it's credible and whether whoever says it knows what they're talking about. The shortcut makes sense. For almost all of history, whatever was easy to process tended to be the familiar, the thing repeated by a lot of people, or the thing said by someone who knew the subject.

Generative AI produces text whose fluency is optimized by training and polished with human feedback to sound competent and unhesitating. The fluency of the output says nothing about whether it's correct. The heuristic, formed in another world, doesn't know that.

The white coat, according to Cialdini

Robert Cialdini published Influence in 1984, one of the most cited books on persuasion. It identified six principles: reciprocity, commitment and consistency, social proof, authority, liking and scarcity. In the expanded 2021 edition he added a seventh, unity, the sense of belonging to the same group. Here the one that matters is authority.

Cialdini documented how external symbols of authority (titles, uniforms, technical vocabulary, confident speech) produce obedience even when what's being asked is questionable. His canonical example is Stanley Milgram's experiments in the sixties, where ordinary people administered what they believed were escalating electric shocks because an experimenter in a lab coat told them to continue. The coat and the tone worked as markers; the experimenter's actual authority didn't matter.

A chatbot produces the textual equivalent of the lab coat. It has no title, no experience, no body, but its prose carries every marker: precise vocabulary, firm connectives, no hesitation, well-placed examples, confident endings. The reader grants credit before deciding whether they should.

The face gives away less than I thought

In the first version of this text I argued that, facing a person, the face, the sweat and the pauses warn us of deception, and that with the machine those warnings disappear. Erving Goffman, in The Presentation of Self in Everyday Life (1959), distinguished between the expression one gives on purpose and the one that leaks out unintentionally, and that second layer would be what protects us.

But wait, does it really protect us? In 2006, in Personality and Social Psychology Review, Charles Bond and Bella DePaulo pooled hundreds of experiments on lie detection. Average accuracy was 54 %, barely above a coin toss. With a face, a voice and a body, we're terrible at spotting deception.

So I'm correcting the argument. The difference with the machine isn't that we used to detect well and now we don't. It's that human hesitation, useless as it is for catching liars, is a reliable signal of doubt, and the speaker's doubt invites us to doubt too. The model never hesitates. It says what's well founded and what it's made up with the same composure.

Four thousand six hundred people who couldn't tell

Maurice Jakesch, Jeffrey Hancock and Mor Naaman published Human heuristics for AI-generated language are flawed in PNAS in 2023. Across six experiments, with around 4,600 people, they asked participants to judge whether personal self-presentations (the kind found in holiday-rental or dating profiles) had been written by a person or an AI. Nobody did better than chance.

The interesting part is why. Participants were using the wrong cues: they took text with first person, informal tone and family references to be human. The models, trained to sound human, produced exactly that. In earlier work presented at CHI 2019, Jakesch and his team had seen the other side: when readers suspected a profile had been written by an AI, they withdrew their trust. We don't detect when we don't suspect, and when we suspect we distrust for no concrete reason.

Longer, more believable

The finding that's made me think the most is recent. Mark Steyvers and his team at the University of California, Irvine published a study in Nature Machine Intelligence (2025) on what they call the calibration gap: the distance between how confident the model is in its answer and how confident the user thinks it is. With the usual explanations, people overestimated the model's accuracy and could barely tell its correct answers from its incorrect ones.

And one detail: longer explanations increased user confidence even though the extra length did nothing to improve the answer. More text, more fluency, more credit. Cialdini would have smiled.

The good part of the study, which I don't want to hide, is that when explanations were rewritten to reflect the model's actual confidence ("I'm not sure", "it's likely"), both gaps narrowed considerably. The bias exists, but it can be corrected by design.

What the machine's defender would say

There's a serious opposing position, and I'm going to give it its best version.

It says, to begin with, that the fluency heuristic isn't a mistake but a good bet: in a normal environment, what's fluent is usually true, and if the models get it right most of the time, trusting their fluency is rational. It adds that people aren't that credulous with machines; quite the opposite. Berkeley Dietvorst, Joseph Simmons and Cade Massey showed in 2015, in the Journal of Experimental Psychology: General, what they called algorithm aversion: after watching an algorithm err, people abandon it sooner than they abandon a human who errs more.

And it finishes with the most cited experiment of the last two years. Thomas Costello, Gordon Pennycook and David Rand published a study in Science in 2024 in which more than two thousand people who believed in some conspiracy theory talked with GPT-4. Their beliefs dropped by around 20 %, and the effect was still there two months later. The AI didn't persuade with tone or flattery, but with concrete data that answered each person's arguments. If the machine's fluency can pull people out of a conspiracy hole, the defender will say, the problem isn't fluency, it's what we put it to work for.

Why it doesn't quite convince me

I grant the Costello experiment without reservation: it's among the best work published on the subject. But read backwards, it's unsettling. It shows that a fluent conversation with a model shifts entrenched beliefs, and that the effect lasts. What shifted those beliefs was correct data. Nothing in the mechanism stops a wrong answer, delivered with the same fluency, from shifting beliefs in the opposite direction just as effectively.

Algorithm aversion doesn't reassure me either, because it's triggered when the error is visible. The characteristic failure of a language model is a well-written hallucination, and what's well written doesn't look like an error. And the rational bet on fluency works as long as the hit rate is high in the specific area you're asking about, which is exactly what the user can't know.

So here's where I land: the machine's fluency is an amplifier, and it amplifies whatever it carries, truth or error, without telling them apart. Steyvers's study suggests there's a technical fix. Whether it gets applied by default is another matter.

Education alone has a ceiling

The classic liberal answer is to teach critical reading. It's well-meant and it isn't useless, but it has a ceiling. Daniel Kahneman, in Thinking, Fast and Slow (2011), explained that the fast, automatic system processes what comes in before the slow, deliberative one gets a chance to step in. When you read a fluent statement, it has already seemed plausible to you before you can ask yourself whether it's true. Education helps the slow system show up more often; it doesn't stop the fast one from getting there first.

What mitigates the effect is friction: uncertainty markers, pauses before accepting something in important decisions, clear notices about where the text comes from. And friction is the last thing anyone who measures their business in usage and retention wants.

Labels, from August 2026

The European Artificial Intelligence Act (Regulation (EU) 2024/1689) came into force in August 2024 with phased application. The transparency obligations in its Article 50 apply from 2 August 2026: telling people they're talking to an AI, marking synthetic content in a detectable way and disclosing deepfakes. For generative systems already on the market before that date, the application timeline allows until 2 December 2026 to comply with the marking requirement.

It's a reasonable measure. My doubt, and this is a hypothesis, not a fact, is how long a label holds up as a warning signal when it's repeated a hundred times a day in the same interface. It already happened with cookie notices: by the third week nobody was reading them. And the label says where the text comes from, not whether it's correct, which is what we'd really need to know.

Text without a face

Persuasion techniques have always become more effective by removing the persuader's body: print advertising before radio, the self-help manual before the podcast. AI takes that to the extreme. Its text has no face, no biography, no reputation to defend, and it slips into emails, reports and messages where we read it with the same heuristic we'd use for a careful colleague.

I'm not expecting a catastrophe. I'm expecting thousands of small decisions tilted towards whatever sounded better: a diagnosis accepted without a second opinion, a contract signed without the reading it would have got if an intern had drafted it, an argument shared because it was well written. The brain isn't going to be redesigned in time. Interfaces could be.

Definitions

• Fluency heuristic: a mental shortcut by which we judge what we process more easily to be truer or more credible.
• Articulated authority: a set of discourse markers (technical vocabulary, confident prose, absence of hesitation) that trigger deference to authority in the receiver, whether or not there's any real authority behind them.
• Calibration gap: the difference between the confidence a model has in its answer and the confidence the user believes it has.
• Algorithm aversion: the tendency to abandon an automated system after seeing it err, faster than a human who makes more mistakes.
• System 1 / System 2: in Kahneman, the fast, automatic mode of thinking and the slow, deliberative one.
• AI-mediated communication: a field defined by Hancock, Naaman and Levy (2020) that studies interpersonal messages in whose production an artificial agent takes part.

References

• Reber, R. & Schwarz, N. — Effects of perceptual fluency on judgments of truth. Consciousness and Cognition 8(3), 338-342 (1999).
• Cialdini, R. — Influence. The Psychology of Persuasion (1984; expanded edition, Harper Business, 2021).
• Milgram, S. — Obedience to Authority. An Experimental View (Harper & Row, 1974).
• Goffman, E. — The Presentation of Self in Everyday Life (Doubleday, 1959).
• Bond, C. F. & DePaulo, B. M. — Accuracy of Deception Judgments. Personality and Social Psychology Review 10(3), 214-234 (2006).
• Jakesch, M., Hancock, J. T. & Naaman, M. — Human heuristics for AI-generated language are flawed. PNAS 120(11), e2208839120 (2023).
• Jakesch, M., French, M., Ma, X., Hancock, J. T. & Naaman, M. — AI-Mediated Communication. How the Perception that Profile Text was Written by AI Affects Trustworthiness. Proceedings of CHI 2019. DOI 10.1145/3290605.3300469.
• Hancock, J. T., Naaman, M. & Levy, K. — AI-Mediated Communication. Definition, Research Agenda, and Ethical Considerations. Journal of Computer-Mediated Communication 25(1), 89-100 (2020).
• Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. & Smyth, P. — What large language models know and what people think they know. Nature Machine Intelligence (2025). Preprint on arXiv:2401.13835.
• Dietvorst, B. J., Simmons, J. P. & Massey, C. — Algorithm aversion. People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General 144(1), 114-126 (2015).
• Costello, T. H., Pennycook, G. & Rand, D. G. — Durably reducing conspiracy beliefs through dialogues with AI. Science 385(6714), eadq1814 (2024).
• Kahneman, D. — Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011).
• Reglamento (UE) 2024/1689, de Inteligencia Artificial, Article 50 (transparency obligations) and application timeline.

Further reading

• Milgram, S. — Obedience to Authority (1974). The book in which Milgram interprets his experiments. It helps to understand what kind of obedience switches on when the authority marker is present and the body of whoever gives the orders is irrelevant, and why Cialdini cites it.

## You might also like - Inevitable anthropomorphism - Trust in imperfect systems - Hallucinations and lies - Progressive trust - The illusion of familiarity

## Elsewhere - arXiv — What large language models know and what people think they know (Steyvers et al.) - EU AI Act application timeline - Guide to Article 50 of the EU AI Act