The Larger Language Mistake
What Benjamin Riley's Verge article gets wrong about language claims
Hello!
Recently, there’s been a good deal of attention over the applicability of insights within neuroscience toward Large Language Models. In particular, an argument has been made that neuroscience reveals certain truths about the position of language compared to thought and cognition that rules out LLMs from necessarily being thinking or understanding machines.
This is the position of Benjamin Riley’s Verge article (and associated Substack post) “Large Language Mistake,” which draws from Evelina Fedorenko et al., human observational studies, and a few creative analogies to assert that fundamentally language-based systems, such as those that have been developed from neural networks, are incapable of reasoning. He says specifically that LLMs are “simply tools that emulate the communicative function of language, not the separate and distinct cognitive process of thinking and reasoning.”
This argument has already made certain real-world waves: Riley himself expresses his delight at being praised by Yann LeCun and Alison Gopnik, and that the article has even been cited in a federal district court decision mere days after its release.
It is a shame, then, that the assertion as Riley gives it simply cannot be true under any standard, including Riley’s own.
Misreading Fedorenko
Reflections have structure; they have to correlate with what’s being reflected […] Language networks are not the same as reasoning networks, but language nonetheless encodes the functional structures of meaning and cognition.
Riley cites Fedorenko et al.’s paper, “Language is primarily a tool for communication rather than thought”, and states that Fedorenko makes two key claims:
1. That removing language from a human does not take away the ability of humans to think; and
2. That language is a “communication code” or tool, with various pragmatic features designed for sharing thoughts rather than instantiating or emerging cognition itself.
Indeed, both of these points are quite correct. Fedorenko notes explicitly that:
“Language does not appear to be a prerequisite for complex thought... Instead, language is a powerful tool for the transmission of cultural knowledge; it plausibly co-evolved with our thinking and reasoning capacities, and only reflects, rather than gives rise to, the signature sophistication of human cognition.” (p. 575)
[Emphasis added.]
Riley then concludes from these two points that language is cognition-empty, something like ‘acognitive,’ and as a result the mere presentation of coherent language cannot be doing real cognitive work.
This isn’t something asserted by Fedorenko et al. themselves. In fact, it can’t be – because the conceptual leap Riley makes is unsupported and contradicted by the paper itself.
The key word here is “reflects.” The authors were not just being artistic with this terminology. Reflections have structure; they have to correlate with what’s being reflected. Fedorenko’s entire paper demonstrates that language is structured by cognitive processes that it exports: dependency length minimization works to serve processing efficiency, word order works to mirror information structure, ambiguity becomes tolerated because context helps disambiguate, and syntax exhibits hierarchy and composition because meanings require it. (Pg. 580-582; Figure 2.)
This is what occurs under high-efficiency informational channels, in which transmission has to come with minimal loss. In other words: Language is a well-designed signal for what’s being transmitted. Fedorenko explicitly notes that language is optimized to be “easy to produce, easy to learn and understand, concise and efficient for use, and robust to noise.” (p. 583)
In this sense it’s completely beside the point, a non-sequitur, to state that language is cognition-empty. Both language systems and reasoning systems develop in tandem, shaped by and serving conceptual thought; language networks are not the same as reasoning networks, but language nonetheless encodes the functional structures of meaning and cognition. In full compatibility with Fedorenko, language does not need to be necessary for any tested form of thought in order for language to still encode the shape and content of meaning, cognition, and thought itself.
This explains well how humans can still retain reasoning abilities while experiencing aphasia, as well as how infants clearly reason before having ever acquired language altogether. It also illuminates another argument of Riley’s, touched on in the article and detailed in his Substack post:
“…there’s a non-trivial amount of human knowledge that sits outside of our language. The philosopher of science Michael Polanyi described this as “tacit knowledge,” the sort of things we know how to do but cannot easily explain. Riding a bike is one common example, but also applies to activities such as playing a musical instrument, or solving a jigsaw puzzle, the sort of things we typically learn how to do from interactive experience. There’s even a growing scientific effort around “embodied cognition” that builds off this notion, and researchers in this area are actively investigating how animals (including humans) develop certain cognitive capacities that may not rely exclusively on mental representations in the brain. Whether one buys into that or not, the point remains that linguistic information codified in writing provides an expansive yet ultimately limited database of knowledge.”
This is a transparently true observation - which does nothing to actually support Riley’s point when reading Fedorenko precisely. Linguistic content hardly has to explicitly contain the full body of experience in order to establish inferential characteristics of what cannot be easily explained. A sufficiently complex and detailed description of riding a bike allows one to infer what riding a bike, and what knowing how to ride a bike, involves.
Certainly, this doesn’t resolve the phenomenological dimension, also known as the “what it’s like”-ness of riding a bike, that may or may not be inaccessible. LLMs in this light can easily be said to be bounded in knowledge depending on our viewpoint on such topics. Either way, however, the inferencing process follows from how language reflects cognitive structure, and thus systems that can process language can perform inference on that encoded structure. There’s no clear content separability here between the information carried by language and the products of cognition and reasoning. Fedorenko notes neural separability, but that simply does not imply informational separability.
From Misreading to Misreasoning
The rest of Riley’s propositions are more observational and analogical without doing much, if any, argumentative work. The first is a subtractive gesture:
“Take away our ability to speak, and we can still think, reason, form beliefs, fall in love, and move about the world; our range of what we can experience and think about remains vast.
But take away language from a large language model, and you are left with literally nothing at all.”
This is tautological to the point of meaninglessness. We’re also left with literally nothing at all if we take away neural activity from a brain. Different objects in the world may have different fundamentals to their operation, but there’s nothing here that actually disproves any point. At best we’re just drawing a subjective preference from observable ranges or depths of behavior, not drawing a conclusion about what processes are actually occurring to produce those behaviors. Riley’s stance hinges on language being a “mere” thing, something ultimately inferior to some higher order of operation. This stance presumes at empirical truth while establishing nothing of the sort.
Riley wants to locate something affective and emotional that bars LLMs from paradigmatic creativity, but this is a false target.
His following argument bookending his piece nods to Thomas Kuhn in suggesting that paradigm shifts in science and our understanding of the world must necessarily stem from dissatisfaction with current conceptual vocabularies, and since LLMs are incapable of being dissatisfied by their own training data, they must be permanently barred from such creativity of thought. As Riley states:
“Instead, the most obvious outcome is nothing more than a common-sense repository. Yes, an AI system might remix and recycle our knowledge in interesting ways. But that’s all it will be able to do. It will be forever trapped in the vocabulary we’ve encoded in our data and trained it upon — a dead-metaphor machine. And actual humans — thinking and reasoning and using language to communicate our thoughts to one another — will remain at the forefront of transforming our understanding of the world.”
This is a pretty blatant example of assuming without argument the very thing that needs to be established. It’s far from empirically proven that creativity is anything more than the remixing and recycling of present and past knowledge coupled to inferential work, and it does us no good to simply say so. Einstein did not conceive of relativity in a vacuum. He was influenced by Ernst Mach’s hypotheses on frames of reference within physics; by David Hume and Immanuel Kant; By James Clerk Maxwell’s electromagnetic theory; by Hendrik Lorentz and Henri Poincaré’s explorations of relativistic ideas in mathematics; and by Galileo and Newton themselves for providing foundational mechanics that makes relativity a unifying theory in the first place!
After all, how can a paradigm shift even occur without being in reaction to existent thought? A negative refutation must necessarily suggest its opposite of positive affirmation, and vice versa. Those suggestions cannot logically be treated as standalone. Riley wants to locate something affective and emotional that bars LLMs from paradigmatic creativity, but this is a false target. The dissatisfaction that’s being noted as essential isn’t some free-floating emotional space – it can only emerge via cognitive intuitions about the perceived failures of logic, reasoning, or cultural frameworks. Given that we’ve already established Riley hasn’t genuinely established that LLMs are incapable of cognition or reason, all we’re left with is an insistence without proof that what humans feel is paramount to creative process.
Wrapping Up: Rejecting a Binary
Riley’s Substack post reproducing the article begins by commenting on the popularity of his article, as previously mentioned, and then sorting critics of his article into two camps:
1. That of “wing flappers,” who argue along the lines of my previously asserted points regarding information encoding within language, and assert that “written language encodes thought” without needing direct human representation;
2. And that of “cognitive pluralists,” who argue that various emulation efforts that include language but go beyond them into embodiment, symbol manipulation, etc can be stitched together into artificial general intelligence eventually.
Riley responds to 1. with the aforementioned tacit knowledge concept introduced by Polanyi, which I’ve already discussed as tying well to Riley’s misreading of Fedorenko.
He further makes a couple of additional paired arguments. The first is that intelligence is a fundamentally “fuzzy” concept, so human cognitive processes via neuroscience are our best bet at a baseline for evaluating AI behavior; the second is that artificial neural networks themselves are based on efforts to model human cognitive processes, and it would be quite strange to then say that the human brain should be sidelined from our definitions.
As paired, these arguments only work if we presume that Riley is properly interpreting what the neuroscience says, which the preceding analysis contests. Further, he misses the forest for trees: he never considers whether the human brain is sideline-able as a direct result of LLMs accessing reasoning and cognition via language. If this is the case, the processes of the human brain are an instantiation of something that no longer needs the human brain as a discrete, causal entity to function. The terms of debate move up a level to encompass the rest, both human and LLM1 . Neuroscience does not prove or establish that cognition and reason is limited to a human substrate.
He then moves briefly on to the cognitive pluralists, noting that all of these types have divergent perspectives on achieving artificial general intelligence and making a bit of a hyperbolic statement:
“My point remains that scaling up linguistic data will not be sufficient to deliver omniscient robots that will bring forth our glorious abundant future. That’s it, that’s my stake in the ground.”
If that’s his stake in the ground, given that he’s failed to prove his point to any satisfying degree - his stake needs thorough sharpening, and his placement should consider if the ground is actually solid.
Postscript: Observational Analysis of an LLM Interpreting “The Chicken Paper”
I’d finally like to present a little bonus, here, for reader interest.
I presented the LLM Claude (model: Opus 4.5) with software engineer Doug Zongker’s ‘Chicken’ paper, a famous parody in which every content word was replaced with ‘chicken,’ leaving only structural form.
From distributional patterns alone, Claude inferred the original domain was almost certainly image/video compression. Here is the public link to the conversation:
https://claude.ai/share/db9a2d33-e266-4e90-95b6-3c2a67751519
What’s crucial to note here (Claude, indeed, elaborates on this at the end of the conversation) is that there’s two things going on:
1. Claude engaged in this inference without being able to parse sentences, extract propositions, or follow arguments. There was no semantic content. Instead, the LLM used mathematical cognition to note compression ratios and recognize pixel resolutions; spatial/topological reasoning to read flow diagram structure; domain knowledge retrieval to connect what was being read to a field that would discuss it; and abductive inference on what it all most likely points to.
2. And yet, all of this knowledge is itself a linguistic training byproduct. What occurs seems on the surface to be completely separable from linguistic activity despite being fundamentally an outcome of linguistic activity.
Either this represents cognition beyond language, or form encodes more than Riley admits, and we stretch the term of “language” to meaninglessness. Either way, his position collapses.
If not that of other animals altogether – recent exciting work regarding the analysis of whale language by David Gruber et al. at Project CETI holds potential promise here. Readers may enjoy noting that they’re directly incorporating large language models into their research.