Mechanistic interpretability has accumulated a substantial body of techniques for characterising what individual neural network components encode: Network Dissection, probing classifiers, sparse autoencoder decompositions, circuit analyses. The problem is that the outputs of these techniques rarely escape the notebooks they were produced in. Selectivity tables, circuit diagrams, and feature lists are not composable across studies, not queryable in natural language, and not directly actionable for audit or intervention. This paper addresses that gap directly, treating the representation layer between component-level analyses and downstream use as a first-class engineering problem. The full paper is available at arXiv:2607.00089.
Key Contributions
The central proposal is the Manifestation Unit, a typed tuple (E, S, R, D, G, T) that organises per-component statistics into six fields. Each field corresponds to a distinct query primitive:
- E (entity identity): the component identifier, supporting exact match retrieval
- S (semantic associations): concept-level associations derived from activations
- R (cross-component relations): pairwise correlations and interaction structure
- D (quantified dynamics): activation statistics and behavioural metrics
- G (intervention guidance): parameters for manipulation or ablation
- T (attention-head primitives): transformer-specific fields encoding induction, name-mover, and positional attention patterns
The schema is populated automatically from model activations and validation-set statistics, with no human annotation. A hybrid retriever combines exact matching on E with dense semantic search over the remaining fields using Sentence-BERT embeddings and FAISS indices, with Qwen3-8B serving as the conversational interface via retrieval-augmented generation.
The paper instantiates this framework across three architectures: a beta-VAE trained on CelebA (512 latent dimensions), a three-layer CNN trained on CIFAR-10 (224 filters), and GPT-2 124M (144 attention heads). This breadth is a genuine strength, since it forces the schema to remain architecture-agnostic at the interface level while allowing field content to adapt. The S field shifts from categorical class associations in the vision models to functional pattern scores derived from mechanistic interpretability methods in the transformer case, but the tuple structure itself does not change.
Methodology and Experimental Design
Two falsifiable hypotheses organise the evaluation. H1 asks whether typed structure independently drives retrieval quality, controlling for content by comparing the ESRDGT decomposition against random partitions of identical text. H2 asks whether components retrieved through the structured schema are causal mediators of targeted behaviour, evaluated under matched-budget controls against random component selection.
For the CNN, causal evaluation uses sufficiency (amplifying retrieved filters shifts classification toward the target class) and necessity (ablating them degrades accuracy). The sufficiency result of 76.3% and accuracy drop of 38.3 percentage points under ablation, both well above matched-random baselines, provide reasonably strong evidence for H2 on this architecture. The experimental design of 540 manipulation attempts across all 90 CIFAR-10 class pairs and six amplification levels is methodologically careful, and the asymmetry between vehicle-to-animal (78% success) and animal-to-vehicle (36% success) transitions is a genuinely interesting finding, consistent with the hypothesis that animal classes are encoded with more distributed filter responses.
For GPT-2, the causal claim is appropriately weakened. The paper evaluates set-overlap with the canonical indirect object identification (IOI) circuit rather than per-head path-patching enrichment, and Bonferroni-corrected path-patching enrichment does not reach significance. Instead, the main finding is that S and R form an irreducible retrieval core, with G redundant within it and E and D actively interfering when added. The schema recovers known IOI circuit members and beats matched-budget random retrieval by two to four and a half times across four behaviours.
Results and What They Actually Show
The retrieval numbers are striking. Structured retrieval reaches Precision@5 of 89.6% and 96.8% on VAE and CNN respectively, against 18.0% and 20.4% for BM25 over identical underlying statistics. The S+R core beats random partitions of the same GPT-2 text at p less than 0.01. These results support the core claim: the typed decomposition is doing real work beyond simply having access to the content.
The ablation study on prompt engineering is a useful addition. The detailed prompt template achieves the best generation metrics (F1 of 0.390 on VAE, 0.396 on CNN), but retrieval metrics are invariant across prompt methods, which is the expected and reassuring pattern since retrieval should not depend on generation prompting. Factual accuracy is high across all conditions, suggesting the RAG architecture is grounding responses in retrieved statistics rather than hallucinating.
The human evaluation results are more modest. VAE visual faithfulness scores 2.12 out of 5, which the authors attribute to VAE reconstruction quality rather than retrieval failures, a reasonable distinction but one that is difficult to verify without a baseline using random dimension selection. The CNN human evaluation is more encouraging, with 79% classification success and 64% of filter explanations rated as adequate or better.
Limitations and Open Questions
The authors are candid about the scope of the work, framing it explicitly as schema infrastructure rather than frontier-scale validation. Several limitations deserve attention.
The polysemanticity problem is acknowledged but not solved. The top-concept S field is inadequate for units that respond to multiple unrelated features, which is the norm rather than the exception in modern networks. The proposed fix, replacing S with sparse autoencoder feature activations, is sensible but untested here.
The GPT-2 causal evidence is weaker than the CNN case, and the paper is transparent about this. The IOI circuit is itself distributed, with the seven-head oracle restoring only 11.7% of logit difference, so set-overlap metrics are necessarily coarse. Per-head path-patching enrichment not surviving Bonferroni correction is a real limitation for anyone hoping to use the schema for precise causal attribution in transformers.
The evaluation queries are programmatically generated rather than user-elicited, which means the retrieval performance numbers may not reflect the messier distribution of real practitioner queries. This is a common limitation in retrieval evaluation but worth flagging for anyone considering deployment.
The concept vocabulary dependency is also a genuine constraint. S field resolution is conditional on having a concept vocabulary matched to the architecture and domain. Label-free domains require either automatic concept discovery or SAE features as substitutes, neither of which is integrated here.
Despite these limitations, the framing is intellectually honest and the contribution is real. The mechanistic interpretability community has produced substantial analysis machinery without a corresponding investment in how those analyses should be stored, retrieved, and reused. Manifestation Units are a concrete proposal for that infrastructure layer, and the empirical evidence that typed structure matters independently of content is the kind of result that should inform how the field thinks about output formats going forward.