Training a second model to explain the first adds another system you have to trust. Goodfire researcher Nick Cammarata's new interpretability essay proposes spending more effort inspecting a network's original neurons directly, a choice that could change how researchers investigate model failures.
The competing approach has produced measurable results. Anthropic's 2023 dictionary-learning paper extracted thousands of interpretable features from a small language model, including features that weren't apparent in individual neurons' strongest activations. The dispute concerns which components let researchers explain a model faithfully, and how much additional machinery they need to find them.
Why a neuron can be hard to name
A neuron is a computational component inside a network. A feature is a pattern researchers believe the network represents. Activations are the numerical signals produced as the network processes an input, giving investigators something to measure. Those units don't always line up. The 2020 Distill introduction to circuits treated features as directions in a model's activation space, which can involve combinations of neurons. It also described individual neurons that appeared to detect curves or dog heads. Researchers could sometimes trace how simpler detectors fed into more complex ones.
That work used several checks for its curve-detector interpretation. Synthetic curves tested which shapes activated a neuron, while rotations tested its preferred orientation. Investigators also examined the connections feeding into and out of it. They even wrote replacement weights by hand to reproduce curve detection. The Distill paper made understanding a component an experimental claim: an appealing collection of example images was only part of the evidence.
A later explanation for difficult neurons was superposition. In Toy Models of Superposition, published in 2022, researchers demonstrated how small networks could represent more features than they had dimensions. When features appeared sparsely, meaning only a few were present in a given input, the network could accommodate overlapping representations while tolerating some interference. The paper's simplest examples put two features into just one hidden dimension. Their behavior changed as the researchers varied feature importance and sparsity. A neuron could then participate in several features, making its activity hard to describe with one label.
The toy experiments established that this packing can happen under their conditions. Their authors explicitly left open how far the results generalized to real networks. That distinction survives the current debate: evidence that a mechanism exists in a controlled model doesn't establish how much it explains in a modern language model. The original paper is useful partly because its limited setting makes the mechanism testable.
Cammarata revisits the apparent mixtures
In the Goodfire essay, Cammarata reinterprets a famous cat-and-car neuron as detecting a shared V shape. He argues that related patterns expanding through learning explain much apparent confusion. Preliminary Qwen3-32B audits estimate that roughly 90% of neuron poles, their positive and negative sides, have one main job. He cautions that this percentage depends on thresholds and what counts as one job. The essay is a working hypothesis.
That disagreement reaches back to the premise of Towards Monosemanticity. Its researchers trained sparse autoencoders to reconstruct a language model's internal activations using a larger collection of sparsely active features. Their detailed analysis focused on 4,096 features extracted from a layer containing 512 neurons. Increasing the autoencoder's size could split a broad feature into more specific ones, changing the resolution at which researchers described the same network.
Those features also supported interventions. Activating an Arabic-script feature produced Arabic text, and activating a base64 feature produced base64 text, according to the 2023 study. Such experiments give the substitute components a stronger claim than readability alone. For someone choosing an inspection method, the useful comparison is whether it helps predict and change behavior in the original model.
A replacement model has its own engineering bill
The scale of that extra machinery is visible in Scaling and Evaluating Sparse Autoencoders, a 2024 paper. Its authors trained an autoencoder with 16 million latent features on GPT-4 activations across 40 billion tokens. They investigated how feature quality changed with autoencoder size, alongside the tradeoff between reconstructing the original activations accurately and keeping the representation sparse. A sparse representation uses relatively few features for each input. Reconstruction tests how much of the original signal the autoencoder can recover from those features.
The paper introduced metrics for recovering hypothesized features, explaining activation patterns and measuring how sparse downstream effects were. Those measures generally improved with autoencoder size. It also addressed dead latents, components that fail to activate usefully. The research shows why the choice of inspection method has practical consequences: the explainer has architecture and training decisions of its own. Its quality needs measurement alongside the network it describes.
A good label can still miss the mechanism
Automating the description of native neurons has a separate history. In OpenAI's 2023 neuron-explanation work, GPT-4 generated and scored explanations for all 307,200 neurons in GPT-2. Most explanations scored poorly. The researchers found that iterating on explanations and using a more capable explainer could improve scores, demonstrating that an inspection tool's limitations can affect how understandable a network appears.
The same report drew a sharper distinction between describing activity and explaining its purpose. A neuron firing on periods might help capitalize the next word or count sentences. An explanation based on input examples couldn't determine its downstream role. OpenAI also warned that even high-scoring descriptions could fail on unfamiliar text because they captured correlations without explaining the mechanisms producing them. For an auditor, that leaves a concrete question: what changes when the component is altered?
Anthropic's 2025 circuit-tracing study of Claude 3.5 Haiku approached that question using attribution graphs and interventions. Its graphs linked interpreted features to trace the steps between an input and a predicted output, generating hypotheses for follow-up experiments. In a case involving the capital of the state containing Dallas, it traced intermediate representations involving Texas and Austin. Inhibiting the Dallas-related features reduced Texas-related activity and changed the model's predictions. That connects an interpretation to a measurable consequence.
Yet the authors reported useful insight for only about a quarter of the prompts they tried. They described their replacement model as incomplete and their selected examples as success cases. The result gives both approaches a demanding comparison point: explaining a compelling example and covering a model's behavior broadly are separate accomplishments. An inspection method needs a record of where it fails, too.
Connections need testing as well
A more recent Anthropic study, published in August 2026, examined interference weights in a one-layer transformer. Here, virtual weights describe effective connections in a rewritten model whose components include tokens and extracted features. Even interpretable components can have misleading-looking connections between them. The researchers distinguished a connection's effect on outputs from its effect on prediction loss, measuring whether it helped or harmed the model's performance. Large weights alone didn't reliably identify the useful computation.
In that small model, removing the least effective 70% of virtual weights increased loss by just 0.01 nats. Loss measures prediction error, so that small increase meant predictions worsened only slightly on this measure. Helpful and harmful weights appeared across the range of weight sizes. The experiment therefore supplies another test for inspection tools: can they distinguish consequential connections from ones the network barely uses? Its limited scale prevents a verdict on large models, but its measurements make the question concrete.
Cammarata says a technical companion paper will detail the language-model experiments. Watch for tests that other teams can reproduce, especially whether explanations predict effects on model outputs. That would let researchers compare the methods on the same task, with failures visible alongside successes.