Deconstructing the Chinese Room
How intuition pumps burst when given faulty pipes
Hello! And – Happy New Year! Hopefully happier than the last.
This is some belated Extra Reading within my Dissolved Distinctions series. Previous entries can be found here:
Intro: Welcome to Dissolved Distinctions
Part One: Is "What Is Consciousness" the Wrong Question?
Extra Reading: Whose Qualia Is It, Anyway?
Part Two: Theorizing Thinking: What Is Cognition?
Part Three: Mind Watching Mind, All the Way Down (Exploring Theories of Metacognition)
Part Four: Enacting Dualisms (subtitle: How Enactivism Betrays Its Own Logic)
Back during part two, in my discussion of theories of cognition, I digressed on enactivism by noting:
“It’s also a theory that prizes autonomous and normative action within organisms as directly harmonizing with various thought experiments that were posed to computationalist perspectives, such as the infamous Chinese Room experiment formulated by John Searle (which will be covered in a future Extra Reading).”
As we ring in 2026, it’s time I follow through. Truth be told, any reader of my blog almost certainly knows my position on the Chinese Room already – I’ve discussed how there’s no clear metaphysical distinction between sufficiently sophisticated simulation or mimicry and “true” reality, and that biological-necessity claims toward understanding and intelligence ultimately rest on unfalsifiable, undefended (or poorly defended) essentialisms. That said, a brief summary of the Chinese Room is in order, along with some collected critical thoughts. 1
Setting the Scene
The classic formulation by famed philosopher John Searle2 goes like this:
The world’s top minds invent/program an artificial intelligence that its creators insist truly, genuinely understands Chinese. This machine “understands” Chinese by taking Chinese characters as input and then carrying out an instructional checklist within the programming that leads to Chinese character output. This output is, linguistically, flawless, and an interlocutor engaging with the computer would have no idea they’re not speaking to a person if they weren’t given some visual or verbal indication beforehand.
Searle argues: If following a step-by-step process leads to fully comprehensible Chinese output, then he could do it with simple pen and paper, “communicating” with someone despite having no idea what’s being conveyed and having no understanding of Chinese in the slightest.
In fact, the typical formulation of the thought experiment has Searle imagining himself operating entirely within the titular “Chinese Room,” replacing the computer entirely within a contextless room save for consulting a rulebook of pure syntax, no semantics, to output the Chinese through a slot. If he could pass the Turing test this way, presuming the computer itself did, then this reveals something crucial, something missing, in translating understanding to artificial minds. This lack of intentionality (or “aboutness,” directedness towards objects) leads to the conclusion that a “strong AI” theory of consciousness and intelligence via digital computation must be false: Purely computational AI necessarily lacks “mind” as associated with humans.
Approaches to the Chinese Room attempt to do one of four things:
1. Disprove the main thrust of the Room entirely, by attempting to find some configuration of capabilities other than biological naturalism that would ground understanding: robotics for the artificial intelligence’s embodiment (in an echo of enactivism), or a redesigned room or machine (such as a perfect simulation of every neuron in the brain);
2. Reject the Room entirely as based on false or misleading premises, such as disputing that such a room could ever possibly be “contextless” as Searle rather casually states, and evoking backgrounds of commonsense knowledge that provide that context within an otherwise bare room;
3. Accept the Room straightforwardly;
4. Accept the premises of the Room but reject a categorical denial of understanding occurring within the thought experiment. Such approaches argue understanding is still present in some conglomeration of the actors and scenery even if not in the machine or Searle as individual units. In this sense, it would be the “system” – Searle, the notebook, the processes he follows, and the room itself – that understands Chinese.
4) is the most famous critique of the Chinese room. However, Searle offered defenses against all three types of critiques and rejections, and each should be given their due time.
Drilling Down
First off, Searle rejects embodiment and redesign arguments as simply another layer of abstraction. He notes that the computer could be outfitted with a camera and hands in order to manipulate the symbols in its internal programming, and understanding still wouldn’t occur. He also insists that “human mental phenomena [are] dependent on actual physical–chemical properties of actual human brains” (Searle, 1980, p. 13) and evokes the image of a man operating elaborate water pipes in a perfect match with a Chinese person’s neurons to produce the same output; such a man would still lack understanding.
Searle agrees that common sense background knowledge exists but denies that it can ever be built into programs based on a syntax/semantics distinction. Essentially, for as long as a machine is only ever operating on a purely syntactical ruleset, it cannot do the work of understanding. In Searle’s words:
“Formal symbols by themselves can never be enough for mental contents, because the symbols, by definition, have no meaning (or interpretation, or semantics) except insofar as someone outside the system gives it to them.”
(Motzkin & Searle, 1989, p. 45)
A related argument that evokes category 2) on a broader meta-level has been posed by illusionists such as Daniel Dennett. Naturally, if you’re an illusionist, you think consciousness is already not nearly as real as it seems, and the thought experiment is thus malformed from the start. Searle is generally not very interested in debating these things; he identifies consciousness as a necessary existent, even if only via inference, that allows us to differentiate things like thermostats from humans.
Having sketched these out, we may already have an intuition of how Searle approaches the systems theory approach. It’s clear from all this that he is a biological naturalist: he thinks consciousness is a real thing and can only be instantiated within a “living” physical subject having something that’s at least like a human brain, something that produces the resultant phenomenology and intentionality that identifies a mind. It’s also important to note that Searle’s general mindset was quite consistent in this regard; he was quite critical of Bruno Latour’s social constructionist approaches to science, as well. As such, we already know he’s going to reject the systems theory.
To do so, he works to dismiss it ad reductio. He asks: What’s the simplest possible way to frame this? The answer: Rather than a rulebook the man just has all the ruleset memorized in his head, and the “system” collapses to a single item, the man himself. To Searle, any attempt at resolving this must necessarily be circular, in that the Chinese Room is fundamentally about whether consciousness is purely information processing, and any argument that resorts to information processing to justify systems theory cannot then stand on its own two feet. (Searle, 1980, p. 6)
Searle, of course, is completely correct with this conclusion – but the fact that he’s correct says a great deal about what the Chinese Room really is as a thought experiment.
What we have here is a classic, paradigmatic display of an intuition pump. Coined by Dennett, an intuition pump is an argument that follows well-worn grooves in human cognition and is sufficiently evocative as to do all the work of an argument without having to engage in consistent logical work. Intuition, in other words, fills the logical gap. To recognize this, it’s necessary to take a step back and survey what the thought experiment is doing.
Searle, as established, is a biological naturalist. He has downright ontological commitments toward crafting an argument that can reveal any position contrary to biological naturalism as unintuitive, unfalsifiable, and/or impossible. He does this to an excellent effect when he concludes that any attempt to prove understanding via informational theory rests merely on its own assertions, and he’s adroit when he notes that embodiment solutions can’t prove an inner, intentional mind. This is the hard problem of consciousness, translated and weaponized.
What he doesn’t do, at any point, is turn his own observations back on himself.
The Diagnosis
Searle asserts an unfalsifiable conclusion when he says that physical-mental properties of human brains ground human mental phenomena, and we only accept it because of how intuitive of an inference it is to have. That’s not to say that his conclusion is wrong, but that we continue to have zero way of establishing any empirical proof to ground it. He does the same thing when he insists that syntax is fully separable and does independent work from semantics and relies on further inferences when insisting on consciousness as necessarily existent in order to distinguish thermostats from people. At every step of the way, these inferences are eminently reasonable and commonsensical even as they fail to show any genuine argumentative work.
Consider: What would it take for Searle to admit defeat?
Searle himself is generous enough to offer implicit criteria: Either syntactical symbol manipulation needs to be sufficient for semantic meaning and understanding, or a machine needs to possess genuine intentionality via programmatic operations. The problem is, these criteria are deliberately impossible to meet, because Searle has already pre-defined syntax as insufficient for semantic meaning and pre-defined intentionality in a way that nothing, beyond biological naturalism, can meet. No wonder the Room seems so inescapable! It can abstract itself to any situation at hand, expanding and reducing while still being interpretable as mere symbol manipulation at every turn.
Never mind, of course, that there are formulations of illusionism that maintain a pragmatic distinction between thermostats and toasters and persons, as I’ve previously sketched out; indeed, it’s Searle’s responses to Dennett et al. that are easily the weakest, because they lay bare how much the substrate relies on mere assertion from the involved parties.3 As Searle states:
“[...] the study of the mind starts with such facts as that humans have beliefs, while thermostats, telephones, and adding machines don’t ... what we wanted to know is what distinguishes the mind from thermostats and livers.” (Searle, 1980, p. 7)
This is nothing more than confidence that the study of mind must necessarily involve correct facts. Without investigation and scrutiny, this reads more as faith than argument.
So where does this leave us? Well, certainly nothing said so far is capable of disproving Searle, and that’s by Searle’s own design. Nothing said by him can really prove him either, despite his reticence to say as much, and that itself is a necessary consequence of his own design. In a situation like this, we may seem to be at an impossible crossroads. Where can we turn without empirical proof, without epistemic certainty?
Our best recourse is to examine the internal logical consistency of our arguments. One scientist gives us the opportunity to do just that.
Margaret Boden, who recently passed away last July, was a research professor of cognitive science and informatics at the University of Sussex. She wrote a paper entitled “Escaping from the Chinese Room” (1988) in which she disputes a key element of how the Room is framed.
She rejects that understanding isn’t present within the room. Rather than positioning it in terms of systems theory, however, she keeps things on a level that’s both intuitive and logically reasonable: surely the man understands both English and the syntactical rulebook he’s drawing from, from which outputs are produced. Moreover, she ties this as functionally identical to how the machine operates, as a translative process from natural language to programming language to lower-level compiler code. Within this process, there’s no way to separate syntax from semantics – semantic natural language is rendered into syntactical programming language, and the programming language itself encodes a semantics that the compiler interprets in a further syntactical display. Understanding, then, is non-binary when it comes to any syntactical/semantic distinction. Boden implies this in a wonderful passage I feel compelled to quote in full:
“If the system is a human, animal, or robot, it may have causal powers which enable it to refer to restaurants and beansprouts (the philosophical complexities of reference to external, including unobservable, objects may be ignored here, but are helpfully discussed by Sloman). But whatever the information-processing system concerned, the answers will sometimes describe purely internal computational processes - whereby other symbols are built, other instructions activated. Examples include the interpretative processes inside Searle-in-the-room’s mind (comparable perhaps to the parsing and semantic procedures defined for automatic natural language processing) that are elicited by English words, and the computational processes within a Schankian text-analysis program. Although such a program cannot use the symbol ‘restaurant’ to mean restaurant (because it has no causal links with restaurants, food and so forth), its internal symbols and procedures do embody some minimal understanding of certain other matters - of what it is to compare two formal structures, for example.
One may feel that the ‘understanding’ involved in such a case is so minimal that this word should not be used at all. So be it. As Sloman makes clear, the important question is not ‘When does a machine understand something?’ (a question which misleadingly implies that there is some clear cut-off point at which understanding ceases) but ‘What things does a machine (whether biological or not) need to be able to do in order to be able to understand?’ This question is relevant not only to the possibility of a computational psychology, but to its content also.”
(p. 264-265)
This itself makes plenty of intuitive sense when one walks into it at the right angle. Is there any way to express human language without the cooperation of syntax and semantics? How can we definitively assert a programming language doesn’t encode semantic meanings? An absolutist answer, as Searle is wont to give in these matters, simply doesn’t capture experiential reality on any pragmatic, observable level. The Chinese Room is an elaborate attempt to box off a gradient – but it’s an arbitrary box, and there’s no obligation for the universe to play by its rules.
This might be seen as unsatisfying for those looking for a definitive resolution or grounding for a categorical position on understanding when it comes to artificial intelligence and the concept of digital minds. After all, we all must place our marks and put our feet down somewhere, and definitions are needed to facilitate this.
Prior entries in this series have discussed some tentative definitions to this effect. My definitions share commonalities with all others when it comes to understanding, consciousness and mind. They are unfalsifiable, for instance, and it’s highly unlikely that they don’t reduce to circularity. As I’ve noted before, it so far seems rather impossible to develop definitions that are non-circular and falsifiable while still checking out logically. If one were developed, I would have very little to talk about!
That said, we still have useful and pragmatic moves to make within this extremely fraught epistemic space within philosophy of mind. We have obligations to tighten the circularity of our claims so that they are fully internally consistent rather than avoidant. We also have obligations to ensure the circularity is generative and virtuous, rather than vicious; it needs to continually refine its operations so as to track experiential reality, rather than abstract and ossify it all away.
Searle does not provide this. The Chinese Room is inert. For all the rhetorical force given, the thought experiment performs no identifiable logical work, just as it claims the machine performs no identifiable understanding. It has long been time to move to less bounded rooms.
Addendum:
There is a wonderful 1984 televised debate between Searle and Boden on what was the British television show “Voices,” hosted by the late philosopher Ted Honderich, in which the Chinese Room is discussed. Searle was an excellent showman, and many will likely walk away thinking he won the argument. Either way, I’d like viewers to ask themselves: How much of the debate, in general, is being carried through force of imagery and rhetoric rather than logic? Are the two interlocutors talking past each other, or not?
I don’t say “refutation” for a reason; as will be discussed, there is little pragmatic utility in attempting to prove or disprove the thought experiment.
Including in Dennett’s case! Dennett, after all, ultimately insists that functionalism is all that mind boils down to - itself a claim that’s hardly unimpeachable.