Thinking Without Understanding. If It Works the Same, What Did We Think Thinking Was

In 1980 Searle proposed a room in which a man manipulates Chinese symbols without knowing Chinese and produces answers indistinguishable from those of a native speaker. Forty-six years later, LLMs do exactly that at industrial scale. Processing isn't understanding, at least under any useful definition of the verb understand. The uncomfortable part of the matter isn't the room. The uncomfortable part is that if it works the same, you have to ask what we thought thinking was before the room was actually built and sold by subscription.

The room, properly

It's worth refreshing the thought experiment because almost everyone cites it from memory and almost no one first-hand. John Searle, in Minds, Brains, and Programs (Behavioral and Brain Sciences 3, 1980, pp. 417–457), asked us to imagine a man locked in a room. Papers with Chinese symbols are passed to him through a slot. He, who doesn't know Chinese, has a manual, also written in his own language, telling him which Chinese symbols to answer with against which combinations of incoming Chinese symbols. He follows the manual, composes answers, returns them through the slot.

Outside the room, a Chinese speaker reads the answers and finds them perfect. He holds a conversation with the man inside and comes away convinced he's talking to a native. The man inside hasn't understood a single word. He has moved symbols according to a book, and the outputs have been correct.

Searle's argument was direct. Syntax doesn't produce semantics. However much a system manipulates symbols following rules, that doesn't constitute understanding, and therefore strong artificial intelligence, the kind holding that a suitable program would be a mind, is refuted. The experiment attacked the functionalism of its time, not LLMs, which didn't exist. Forty-six years later the situation is uncomfortable. What was a thought experiment has become a commercial product. And Searle's question hasn't been answered. It has been displaced.

Bender and Koller. The octopus that learns English by accident

In 2020, in Climbing towards NLU. On Meaning, Form, and Understanding in the Age of Data (ACL 2020), Emily Bender and Alexander Koller updated the argument with a metaphor worth keeping fresh. Imagine, they say, a hyperintelligent octopus at the bottom of the sea, connected to an undersea telegraph cable. Down that cable travel conversations between two people on an island. The octopus, with no access to the island's world, never seeing a palm tree, never touching a coconut, listens for years. It learns to predict which patterns follow which patterns. The moment comes when it taps the cable and starts replying in place of one of the speakers. It works, for a while, because it has learned the form. Until the speaker asks something concrete about how to build a catapult out of palm trees: there the octopus, which has never seen a palm tree, fails in silence.

Bender and Koller formalise it like this. A system trained solely on linguistic form, with no access to the external referent of what's said, cannot learn meaning in the strict sense. It can learn statistical distributions of how words chain together, which is enough to sustain a good part of a conversation. It's not enough to guarantee that the system knows what it's talking about.

The octopus argument is the technical, updated version of the Chinese room. It's more useful than the room because it doesn't depend on metaphysical intuitions about what counts as a "mind." It only says there's an operational distinction between form and meaning, that LLMs sit on the side of form, and that confusing one for the other has practical consequences.

The functionalist reply, and why it doesn't quite close the debate

There's a classic way out of Searle's argument, defended vigorously by Daniel Dennett in Consciousness Explained (Little, Brown, 1991) and, with other tools, by Douglas Hofstadter in Gödel, Escher, Bach (Basic Books, 1979). It's called functionalism. It says roughly this: if two systems produce exactly the same observable behaviour given the same set of stimuli, they are cognitively equivalent. What matters is the function, not the substrate. It doesn't matter whether inside there are neurons, transistors or a man with a manual. If it does the same, it is the same.

The position has its elegance and resolves some impasses. If we reject functionalism strictly, we end up arguing that only certain biological material can sustain a mind, which isn't a comfortable idea to defend except on faith. But functionalism pays for its elegance at a price the debate has uncovered bit by bit. If complete functional equivalence requires the two systems to behave the same in all possible situations, LLMs don't meet it. They fail outside their training distribution, they hallucinate, they contradict reasoning they themselves had just held. The equivalence is local, not global. If you demand only practical equivalence within a bounded range, then the word "equivalence" empties out.

Gary Marcus has made flagging these cracks his work of recent years. In entries like A Knockout Blow for LLMs? in Marcus on AI and in his column in Communications of the ACM, he insists that the most recent reasoning models keep collapsing on structured problems slightly out of distribution, and that without an explicit representation of facts there's no in-principle solution to hallucination. You can accept or reject the optimistic or pessimistic conclusion. What's hard to argue with is the observation. Functional equivalence, in LLMs, is partial and depends on the crop.

The pragmatic paradox

Here comes the uncomfortable question the classic philosophical debate doesn't contemplate, because in 1980 it wasn't a practical question. What if it doesn't matter whether the system understands, as long as it gets it right?

If a company hires an assistant to answer customer queries, what it measures at the end of the quarter is the resolution rate, not the depth of the assistant's understanding. If an emergency doctor uses a system to support diagnoses, what counts is whether the system gets it right more often than the doctor alone. If a programmer delegates tasks to an LLM, what matters is whether the code works and the tests pass. In all those contexts, getting it right is the only operational thing. Understanding is a concept the contract doesn't sign.

The paradox consists of this. If in enough contexts the difference between understanding and getting it right doesn't show, then the concept of understanding loses traction in public life. It doesn't stop existing philosophically, but it stops being demandable, stops being auditable, stops being a decisional lever. And when a concept loses traction, the person who had the asymmetric information wins. The industry knows its systems don't understand; the user assumes they understand; the life in between proceeds as if the asymmetry didn't exist, until it collapses.

The suspicion about our own thinking

There's a second reading, subtler. If it works the same and it doesn't matter, it's not only the concept of understanding that collapses. It's our own understanding that comes under suspicion. Maybe what we called thinking in humans was also, in a larger share than we admitted, the manipulation of learned patterns. Maybe human intelligence is closer to Bender and Koller's octopus than to the idealised image we make of it. The question, then, isn't only what LLMs do. It's what we thought we were doing before LLMs did it at scale.

This reading is the one that discomforts most, and the one that's worst admitted in any public conversation about AI. Accepting it partly doesn't oblige you to accept functionalism. It only obliges you to accept that the difference between understanding and simulating understanding is finer than common sense believed, and that part of human cognitive operation falls on the side of simulation more often than we like to acknowledge.

Emergent capabilities and other worn-out words

In 2022 the first papers on emergent capabilities of LLMs appeared. As the models grew, certain tasks went from complete failure to competent resolution with no intermediate steps. Modular arithmetic, translation of rare languages, multi-step reasoning. The word "emergence" evoked a qualitative leap and seeped quickly into commercial discourse.

The later literature has cooled the image considerably. Searle returned to the matter in The Rediscovery of the Mind (1992) to specify what his original argument did say and didn't, against a decade of lazy paraphrases; Bender, Gebru, McMillan-Major and Shmitchell, in On the Dangers of Stochastic Parrots (2021), extended the problem to the elastic use of cognitive vocabulary applied to systems that have nothing of the substrate the word refers to. Berti and others, in Emergent Abilities in Large Language Models. A Survey (arXiv 2503.05788, 2025), review the field and show that a good part of the "emergent" leaps are artefacts of the chosen metric: if you use accuracy on a binary task, you see leaps; if you use continuous likelihood, you see smooth growth. Emergence, in many cases, isn't the model's: it's the gauge we use to see it. That doesn't deny that real transitions exist in some cases. It denies the idea that the model "understands once it reaches a certain scale," because that's exactly what a human who hears "emergent" takes in.

The rhetorical chain works. Emergent leap → understanding → reasoning → consciousness. Each link is an abbreviation acceptable only if the previous one is. Break the first, show that the leaps are metric artefacts, and the whole chain breaks. But the broken chain keeps circulating in press kits, because the reader doesn't read the correction published a year later.

What the unclosed debate leaves in your hand

If the Chinese room hasn't been answered in forty-six years and LLMs have taken it into the real world, the reasonable thing is to ask what you do in practice with a debate that doesn't close. The operational answer, whether we see it or not, is already given. Adoption decisions are taken in favour of the most permissive reading of the concept of understanding. Regulators accept behavioural metrics. Investors value demonstrated capabilities. Users delegate trusting in an operation whose epistemological status remains open.

The ambiguity isn't neutral. When a concept isn't settled, it's exploited by the actor with the most capacity to set the narrative, and in this case that actor is the industry. Calling what the models do understanding is legal, is defensible philosophically under functionalist positions, and is commercially useful. The part that's systematically left unsaid is the clause serious academics add to their version of the term: "understanding in the functional sense restricted to this task." The clause disappears when the concept crosses into the press kit, and what's left is a big word with no nuance.

Searle wrote in 1980 that a program wasn't a mind. Bender and Koller in 2020 wrote that a system trained only on form didn't learn meaning. The two statements are compatible, are empirically verifiable at least in their weak version, and both still discomfort because they oblige you to admit that most of what the industry calls understanding is something else. If that something else is enough to get it right, wonderful. But it's worth knowing it's something else before delegating what only understanding lets you delegate.

Definitions

Chinese room. A thought experiment proposed by John Searle in 1980 to argue that manipulating symbols according to rules, however correct the output, doesn't constitute understanding. A locked man answers in Chinese following a manual without knowing Chinese.

Strong AI / weak AI. A distinction introduced by Searle. Strong AI holds that a suitable program would be a mind with real mental states. Weak AI holds that a program can functionally simulate a mind without having mental states. Searle rejected the first.

Functionalism. The philosophical position that what defines a mental state is its functional role, not the physical substrate that realises it. Dennett is one of its contemporary defenders. It implies that different systems with equivalent functions are mentally equivalent.

Form vs meaning. A central distinction of Bender and Koller (2020). Form is the statistical distribution of linguistic expressions; meaning depends on their anchoring in referents in the world. A system trained only on form doesn't learn meaning.

Octopus test. Bender and Koller's metaphor of the octopus that learns to predict linguistic patterns without access to the speakers' world. A modern, operational version of the Chinese room.

Emergent capability. An ability a language model exhibits from a certain size and that wasn't present in smaller models. The recent literature suggests that a good part of the documented emergences are artefacts of the metrics used, not real transitions of the model.

References

Searle, J. R. (1980). Minds, Brains, and Programs. Behavioral and Brain Sciences 3, 417–457. The origin of the Chinese room thought experiment and the argument against strong AI.

Searle, J. R. (1992). The Rediscovery of the Mind. MIT Press. A later reformulation of the argument, cited to specify the scope of the rejection of strong AI.

Bender, E. M. & Koller, A. (2020). Climbing towards NLU. On Meaning, Form, and Understanding in the Age of Data. ACL 2020. The origin of the "octopus test" and of the operational distinction between form and meaning in language systems.

Bender, E., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. DOI: . Extends the argument about the elastic use of cognitive vocabulary applied to LLMs.

Dennett, D. (1991). Consciousness Explained. Little, Brown. A contemporary functionalist position, cited as a counterpoint to Searle's rejection.

Hofstadter, D. (1979). Gödel, Escher, Bach. An Eternal Golden Braid. Basic Books. A broad defence of the possibility that the mind emerges from the recursive manipulation of formal patterns.

Berti, L., Giorgi, F. & Kasneci, G. (2025). Emergent Abilities in Large Language Models. A Survey. arXiv 2503.05788. A critical review of the notion of emergent capabilities; many leaps are metric artefacts.

Marcus, G. Marcus on AI. Substack and Communications of the ACM. An ongoing critique of using cognitive terms for systems that fail out of distribution.

You might also like

• Statistics as a substitute for knowledge
• Machines that seem to think
• Measuring artificial intelligence
• Irony and context. The eighty percent of speech that escapes them

Elsewhere

• The Chinese Room Argument (Stanford Encyclopedia of Philosophy)
• AI: A Guide for Thinking Humans (Melanie Mitchell)