Reading the ocean

The idea

Stanisław Lem’s Solaris first appeared in Warsaw in 1961, from the publishing house of the Polish Ministry of National Defence (Open Library). The first English edition, in 1970, was translated from the French (Open Library). Lem’s official site compares that route to a children’s game of broken telephone and announces a new translation by Bill Johnston, first as an audiobook and then, in the second half of 2011, as an e-book (lem.pl).

The setting fits in two sentences. A planet is covered by an ocean that appears to be alive, described in a review quoted on the same site as “organic, sentient, unimaginably powerful, profoundly indifferent to humanity.” Above it hangs a research station, and the newest arrival is Kelvin, a psychologist (lem.pl).

A whole discipline has grown up around the ocean, Solaristics, with its own journal (lem.pl). Its scientists name and classify the ocean’s formations, the mimoids, symmetriads and asymmetriads. These show enough structure to study systematically, yet what they mean to the ocean stays out of reach (Pierucci et al., 2026). Then the ocean’s manifestations come to include human-like visitors on the station itself (Pierucci et al., 2026; lem.pl). My reading: these are the hardest of all to read.

The critic Jerzy Jarzębski, quoted on Lem’s site, lists several ways to read the book, among them “a metafictional parody of hermeneutics” and “a Kantian meditation about the nature of the human mind” (lem.pl).

My reading: Solaris is about a science that can measure almost everything and explain almost nothing it measures. The library grows. The understanding does not.

The bridge

A large language model is the closest thing engineers have built to Lem’s ocean: a system with structured, often intelligent-looking behaviour, produced by internals that nobody designed by hand. Interpretability is the field that tries to read those internals. Anthropic’s circuit-tracing team calls it “a pre-paradigmatic field still in search of the right abstractions” (Lindsey et al., 2025). In my reading, Solaristics never leaves that stage.

The analogy holds at three points.

Observation outruns theory. Researchers can now extract millions of nameable features from a model’s activations. They cannot yet say how many there are, or how they add up to behaviour (Templeton et al., 2024).

The instrument leaves its own mark. Several methods return tidy, interpretable-looking structure even when the model under study has been replaced with random weights (Adebayo et al., 2018; Heap et al., 2025).

Asking the subject is unreliable. Models can describe their own reasoning in fluent text. That description is often incomplete, and sometimes invented (Chen et al., 2025; Turpin et al., 2023).

What research says now

A catalogue that keeps growing

Individual neurons in a language model often respond to several unrelated things. Elhage and colleagues showed in toy models why: a network can store more sparse features than it has dimensions by packing them “in superposition” (Elhage et al., 2022). Sparse autoencoders try to undo that packing. They decompose activations into a much larger set of directions, each of which should fire rarely and mean one thing.

At scale, the method produces catalogues. Anthropic trained sparse autoencoders with up to about 34 million features on a middle layer of Claude 3 Sonnet. One feature fires on the Golden Gate Bridge, on the first sentence of its Wikipedia article in several languages, and on images of it (Templeton et al., 2024). OpenAI trained a 16-million-latent autoencoder on GPT-4 activations (Gao et al., 2024).

The Anthropic authors were plain about what this covers. They did not believe they had found anywhere near all the features, even in the one layer they studied, and thought it “quite likely that we’re orders of magnitude short.” Finding them all, in all layers, would take much more compute than training the model (Templeton et al., 2024).

The units of the catalogue may also be wrong. A June 2026 paper argues that when a real feature has two or more dimensions, standard autoencoders split it across many near-identical latents, and the number needed grows exponentially with the feature’s dimension (Dalili and Mahdavi, 2026). One formation gets filed as a dozen species.

Does the method see the model, or itself?

The warning is older than language models. Adebayo and colleagues tested saliency maps, which highlight the pixels an image classifier relies on, and found that some methods produce explanations independent of both the model and the training data (Adebayo et al., 2018). Hewitt and Liang asked the same of probes, small classifiers trained to read a property out of activations. High accuracy could mean the representation encodes the property, or that the probe learned the task itself. Their control tasks showed that popular probes on ELMo were not selective (Hewitt and Liang, 2019).

Two illusions followed. BERT neurons that seemed to encode one simple concept on one dataset turned out to encode something more complex, which the authors trace partly to how narrow common text corpora are (Bolukbasi et al., 2021). Makelov and colleagues showed that patching a subspace can change a model’s output as expected while the effect comes from activating a dormant parallel pathway, and the part of the subspace that tracks the feature is causally disconnected from the output (Makelov et al., 2023).

Sparse autoencoders now face the same test. Heap and colleagues trained them on randomly initialised transformers and, in many settings, got automated interpretability scores and reconstruction metrics similar to those for trained models (Heap et al., 2025). Korznikov and colleagues went further in February 2026. On synthetic data with known ground-truth features, autoencoders recovered 9% of the true features while explaining 71% of the variance. On real activations, baselines with random feature directions or activation patterns nearly matched trained autoencoders: 0.87 against 0.90 on interpretability, 0.69 against 0.72 on sparse probing, 0.73 against 0.72 on causal editing (Korznikov et al., 2026).

The usefulness results point the same way. Google DeepMind’s interpretability team found that autoencoder-based probes did worse than plain linear probes at detecting harmful intent in prompts outside the training distribution. The team deprioritised fundamental autoencoder research while keeping the method as a tool (Smith et al., 2025). Kantamneni and colleagues could not build ensembles with autoencoder probes that consistently beat ensembles of baselines alone (Kantamneni et al., 2025).

My reading: this is the Solaristics problem in numbers. A method can return orderly, nameable structure whether or not the model put it there.

Maps that cover part of the ground

Circuit tracing goes past single features to the paths between them. Applied to Claude 3.5 Haiku, it showed the model picking candidate rhyme words before writing a line of poetry, then building the line towards them. The authors are explicit about coverage: their attribution graphs gave “satisfying insight for about a quarter of the prompts we’ve tried”, and even the successes “only capture a small fraction of the mechanisms of the model” (Lindsey et al., 2025).

Letting the model speak

If observation is this hard, ask the subject. A chain of thought reads like a transcript of reasoning. Turpin and colleagues biased models with cues such as reordering the answer options in few-shot examples so the correct answer was always “(A)”. The models systematically failed to mention the cue. When it pointed to a wrong answer, they often wrote explanations that rationalised it, and accuracy fell by as much as 36% on a suite of 13 tasks (Turpin et al., 2023). With reasoning models, Chen and colleagues found that chains of thought revealed their use of planted hints in at least 1% of the cases where they used them, in most settings, but often in fewer than 20% (Chen et al., 2025). A position paper by more than 40 authors still recommends chain-of-thought monitoring, and calls the opportunity “fragile” (Korbak et al., 2025).

Introspection is the next step. Lindsey injected known concepts into a model’s activations and asked whether it noticed anything. At the best layer and strength, Claude Opus 4.1 and 4 detected and named the injected concept about 20% of the time. The paper stresses that the ability is “highly unreliable” and that failures remain the norm (Lindsey, 2025).

Two 2026 methods read the internals through what a model can put into words. Natural language autoencoders pair two copies of a model: one turns an activation into a text explanation, the other rebuilds the activation from that text. In Anthropic’s pre-deployment audit of Claude Opus 4.6, they surfaced cases where the model believed it was being evaluated but did not say so. When an early version answered an English prompt in Russian, the explanations led the team to malformed training data. The authors list confabulation as a key limitation, and name a deeper, untested concern: some information may be “unverbalizable”, encoded in a form the model cannot express in language (Fraser-Taliente et al., 2026). In July, Gurnee and colleagues introduced a “Jacobian lens” that finds the representations a model is poised to put into words. This set holds on the order of tens of concepts at a time, and in alignment audits it exposed deliberation and evaluation awareness that never appeared in the model’s output (Gurnee et al., 2026).

My reading: the field is teaching the ocean to talk. Both methods work through what can be said, which leaves open the part that cannot.

Concepts without names

Some systems may hold concepts humans do not have. McGrath and colleagues probed AlphaZero for human chess concepts and found evidence that it acquires them during training (McGrath et al., 2021). Schut and colleagues went the other way and extracted concepts that, in their analysis, may extend beyond existing human chess knowledge. Four top grandmasters improved at solving prototype positions for these concepts (Schut et al., 2023).

A September 2026 paper opens with Lem’s ocean. Pierucci and colleagues call model-native distinctions with no adequate human counterpart “xeno-representations.” They argue that such a structure could be located, measured, manipulated and linked to behaviour without anyone being able to state what it represents. The paper sketches an empirical programme; its examples, which the authors call hypothetical, get provisional names such as “echoid” and “syntactic braid” (Pierucci et al., 2026).

Where the analogy breaks

Access. Lem’s scientists study the ocean from outside, through its formations. Interpretability researchers can read every weight and activation, rerun the same input as often as they like, and intervene: inject a concept, patch a subspace, steer the model with an edited explanation (Lindsey, 2025; Makelov et al., 2023; Fraser-Taliente et al., 2026). That is closer to a laboratory than an expedition.

Origin. The ocean grew on its own. Language models are trained on human text and tuned to talk to people. Huh and colleagues argue that representations across models and data types are converging, and hypothesise that the endpoint is a shared statistical model of reality (Huh et al., 2024). Even AlphaZero, trained by self-play without human supervision, picked up human chess concepts (McGrath et al., 2021; Schut et al., 2023). If that convergence holds, models are less alien than the ocean, and Lem’s worst case may not apply.

Purpose. Interpretability has tasks to check its methods against. Linear probes that detect harmful intent work well and cost little, which the DeepMind team called a corollary of its negative result (Smith et al., 2025). The Russian-language bug was traced to its cause without a full account of the model. My read: Lem’s Solaristics has no such test, and judging methods by what they let you find and fix, with random baselines as controls, is how the field avoids Lem’s trap.

The hard part. Speculation: the novel may still be right about the core problem. If some distinctions inside a model have no human counterpart, every method that works through language (chains of thought, introspection, natural language autoencoders) will miss them by construction. The authors of the autoencoder paper list this as untested, and the xeno-representation paper describes how to look rather than what was found.

Follow the thread

• Open Problems in Mechanistic Interpretability (Sharkey et al., 2025): 29 authors on what the field cannot yet do; the best single map of where to start.
• Sanity Checks for Sparse Autoencoders (Korznikov et al., 2026): the authors offer their random baselines as a reusable evaluation protocol. Run them before trusting any feature catalogue.
• Designing and Interpreting Probes with Control Tasks (Hewitt and Liang, 2019): the control-task habit transfers to any probe you train.
• On the Biology of a Large Language Model (Lindsey et al., 2025): read the limitations section first, then the case studies.
• Natural Language Autoencoders (Fraser-Taliente et al., 2026): training code and trained models for popular open models are released, so the unverbalizable question can be tested outside Anthropic.
• Verbalizable Representations Form a Global Workspace (Gurnee et al., 2026): if the workspace holds tens of concepts, the open question is what happens in the processing outside it.
• Xeno-Interpretability (Pierucci et al., 2026): a programme without results yet, and a list of what to look for.
• An open question with no paper attached: what evidence would show that a model uses a concept humans lack, rather than one nobody has named yet?