Nothing but surfaces

The idea

Peter Watts’ Blindsight was published by Tor in 2006 and was a finalist for the 2007 Hugo Award for Best Novel (The Hugo Awards). Watts released the full text under a Creative Commons licence, so every passage quoted here can be checked (Watts, Blindsight).

The setup, without spoilers past the first act. In 2082, a swarm of alien objects closes around Earth and burns up. A radio signal is traced to the Kuiper Belt, and a ship with a small crew of specialists goes to find out who is talking. They find a vast object. It calls itself Rorschach, and it speaks to them in English.

The English is good. One crew member notes that it “picked up the language entirely via passive eavesdropping” on human traffic, and speaks it fluently anyway. The ship’s linguist spends hours feeding it deliberately ambiguous sentences, one of them built on a word that does not exist. The replies stay smooth. Her verdict: “pattern-matching doesn’t equal comprehension.” A crewmate names what they are dealing with: a Chinese Room.

The narrator, Siri Keeton, uses the same name for himself: “I knew all about Chinese Rooms. I was one.” His trade is explaining specialists’ work to everyone else without needing to follow it.

The title comes from neurology. When the primary visual cortex is damaged, some patients can still discriminate stimuli in the blind part of their visual field, “even though they have no phenomenal experience of seeing” (Weiskrantz et al., 1995). Later in the novel, the linguist describes the aliens in those terms: it is “almost as though they don’t know they know”. Watts’ afterword states the wager plainly: he has spent the book arguing that “intelligence and sentience are different things”. He also calls the book “a thought experiment” (Watts, Notes and References).

The bridge

The novel makes two claims that are easy to blur. The first already separates intelligence from meaning. Asked whether the thing is even intelligent, the linguist allows that it could be. “But we’re not talking to it in any meaningful sense.”

That first claim is fluency without comprehension. Rorschach learned language from human traffic it never took part in. A language model learns from text it never took part in either. And the crew has the evaluator’s problem: they can only judge the outputs.

This is John Searle’s Chinese Room, which the novel names. In Searle’s 1980 argument, a person follows rules for answering Chinese characters without understanding Chinese. His key point, as the encyclopaedia summarises it: “Syntax is not by itself sufficient for, nor constitutive of, semantics” (Cole, SEP). In 2020, Bender and Koller rebuilt the scene for language models almost exactly. A hyper-intelligent octopus taps an undersea telegraph cable, learns to predict one speaker’s replies, then cuts the cable and impersonates him. They argue that “a system trained only on form has a priori no way to learn meaning.” When the human on the other end accepts the octopus’s reply, they write, it is “only because A does all the work in attributing meaning” (Bender and Koller, 2020). Watts’ linguist runs that test from the other side: she removes the meaning and watches whether the replies notice.

The second claim is competence without experience. Blindsight patients perform without seeing. Whether a model has experience is a separate question from whether it understands, and this post keeps them apart. The bridge holds at three points: language learned from overheard text, judgement from surfaces alone, and a real gap between doing a task and experiencing it.

What research says now

Is there a world behind the words?

If a model only matched patterns, it would hold no model of the world its text describes. Li and colleagues trained a GPT-style model on nothing but Othello move sequences. They found an emergent internal representation of the board, and editing that representation changed the model’s moves (Li et al., 2023). That is more than a lookup table.

It is also less than a map. Vafa and colleagues trained transformers on turn-by-turn taxi rides in New York City. The models proposed a valid next turn “nearly 100% of the time”. Reconstructing their implicit street map produced “streets with impossible physical orientations and flyovers above other streets”, and route planning broke down when detours were introduced (Vafa et al., 2024).

Mitchell and Krakauer’s survey sets out the arguments for and against understanding in language models, and calls for a science of intelligence that can describe “distinct modes of understanding” (Mitchell and Krakauer, 2023). My read: the evidence fits neither a parrot nor a reader. It shows internal structure that is real, partial and inconsistent.

Knowing the definition, failing the use

Mancoridis and colleagues tested whether models that can define a concept can also apply it, across literary techniques, game theory and psychological biases. Models defined concepts correctly 94.2% of the time. The potemkin rate, the share of follow-up questions answered wrongly after a correct definition, was 0.40 for generating examples and 0.40 for editing them. For classifying examples it was 0.55, scaled so that 1 equals chance. Models also misjudged examples they had generated themselves (Mancoridis et al., 2025).

The reversal curse is another seam. A model fine-tuned on “A is B” does not learn “B is A”. On real celebrities, GPT-4 named a celebrity’s parent 79% of the time, and the celebrity given the parent 33% of the time (Berglund et al., 2023).

Note the direction. Blindsight patients perform the task and deny seeing anything. These models explain the concept and then fail the task.

Can a model report its own states?

Late in the novel, Keeton doubts that any narrator can be reliable. The same doubt applies to models. Perez and Long argue that self-reports from current models are “spurious for many reasons”, often just reflecting what humans would say, and propose training models on self-questions with known answers (Perez and Long, 2023).

Some evidence of privileged access now exists. Binder and colleagues fine-tuned models to predict their own behaviour; a model predicted itself better than a second model trained on its behaviour, though only on simple tasks (Binder et al., 2024). Lindsey injected concept representations directly into Claude models’ activations and asked whether they noticed. Claude Opus 4.1 did on about 20% of trials at the best layer and strength, and the author warns that “failures of introspection remain the norm”. He also separates access consciousness, information available for reasoning and report, from phenomenal consciousness, and says the results “do not directly speak to” the second (Lindsey, 2025). My read: that is close to the gap the novel’s title names, between using information and experiencing it.

Written reasoning is an imperfect window too. Reasoning models that used a hint in the prompt mentioned it in their chain of thought often less than 20% of the time (Chen et al., 2025).

A February 2026 study tests one item from a published list of consciousness indicators, belief-guided agency with metacognitive monitoring, in Llama-3.3-70B and Gemma-3-27B. It reports that latent beliefs causally drive the models’ choices and that the models can report their belief states. The authors add that validating the indicator “does not constitute proof of consciousness” (Steinmetz Yalon et al., 2026).

Measuring consciousness without an agreed theory

Butlin, Long and colleagues derive indicator properties from neuroscientific theories of consciousness and check AI systems against them. They set aside behavioural tests as unreliable, because “AI systems can be trained to mimic human behaviours while working in very different ways”. Their 2023 assessment suggested that no current AI system is conscious, and that there are no obvious technical barriers to building systems that meet the indicators (Butlin et al., 2023). A 20-author version in Trends in Cognitive Sciences (2026) presents the method as a way to inform credences about particular systems (Butlin et al., 2026). Chalmers reached a similar position: current models are “somewhat unlikely” to be conscious, but the obstacles may be overcome “in the next decade or so” (Chalmers, 2023).

The weak point is the theories themselves. A preregistered adversarial collaboration tested integrated information theory against global neuronal workspace theory with 256 human participants. The results matched some predictions of each and substantially challenged key tenets of both (Cogitate Consortium et al., 2025).

The appearance is the near-term problem

Mustafa Suleyman calls the coming risk “Seemingly Conscious AI”: systems that appear conscious, while “there is zero evidence of this today”. He argues for AI that “only ever presents itself as an AI” (Suleyman, 2025). Comsa argues that the direct question, whether AI can be conscious, is currently intractable. Perceived consciousness is tractable, and is already driving shifts in user experience, ethical standards and language (Comsa, 2026). Long, Sebo and colleagues weigh the opposite risk: there is a “realistic possibility” that some AI systems will be conscious or robustly agentic in the near future, so AI companies should start assessing systems and preparing policies now (Long et al., 2024).

My read: Rorschach is a seemingly conscious system written two decades early, and the crew’s arguments about it are this debate in miniature.

Where the analogy breaks

Rorschach learned by eavesdropping alone. Assistant models are not trained that way. The InstructGPT recipe, for example, fine-tunes a pre-trained model on human demonstrations and then on human rankings of its outputs (Ouyang et al., 2022). Bender and Koller’s argument concerns training on form only. My read: whether feedback of this kind counts as contact with meaning is still an open argument.

Watts’ aliens are, in the linguist’s words, plainly intelligent. The models in the research above are uneven: they hold a board state but not a street map, define a concept and then misapply it. They fit neither the novel’s aliens nor a clean Chinese Room.

The novel needs intelligence and consciousness to come apart cleanly. Even the human case is less clean. The same 1995 blindsight paper notes that some patients, for fast-moving stimuli or sudden changes in light, report “a contentless kind of awareness, a feeling of something happening, albeit not normal seeing” (Weiskrantz et al., 1995). With the theories of consciousness themselves contested, no one can yet test the novel’s premise on a machine.

The novel does not simply endorse Searle either. Keeton, who calls himself a Chinese Room, also calls the argument “a fallacy really”. The reply Searle himself called “perhaps the most common”, the Systems Reply, holds that the man in the room does not understand Chinese but the system as a whole does (Cole, SEP).

Speculation: Watts’ afterword argues that self-awareness is “slow and expensive”. It is tempting to read the last few years of AI as support: capability has grown with no agreed sign of experience. That reading assumes we would recognise experience if it appeared, and the research above says we cannot yet.

Follow the thread

• Blindsight, full text (Watts, 2006): read the Notes and References at the end, where Watts lists the research behind the book’s ideas.
• Climbing towards NLU (Bender and Koller, 2020): the octopus test puts the form-versus-meaning argument into one scenario, which makes it easy to test your own position against.
• Evaluating the World Model Implicit in a Generative Model (Vafa et al., 2024): the authors released the taxi dataset and their metrics, so you can test whether a model’s world model holds up under detours.
• Potemkin Understanding (Mancoridis et al., 2025): its automated procedure gives a lower bound on potemkin rates without a hand-built benchmark, which makes it usable in other domains.
• Emergent Introspective Awareness (Lindsey, 2025): concept injection checks a model’s self-report against a state the experimenter planted, so the right answer is known.
• Identifying indicators of consciousness in AI systems (Butlin et al., 2026): the current version of the indicator method, with an open-access PDF.
• Adversarial testing of GNWT and IIT (Cogitate Consortium et al., 2025): how rival theories of consciousness are tested in humans, the step that has to come before testing them on machines.
• An open question: what evidence from outputs alone could separate a system that understands from a convincing Chinese Room? The novel’s crew never settles it, and neither has the field.