ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #NLA #Mechanistic Interpretability #Activation Reconstruction #GCG #LoRA

Reconstruction Is Not Faithfulness: Auditing Natural Language Autoencoders

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:EleutherAI’s SOAR 2026 study examines whether Natural Language Autoencoder explanations remain stable across training runs, paraphrases, and targeted edits. Four LoRA-based EasyNLA pairs trained on Qwen2.5-7B-Instruct could partially exchange verbalizations across runs, but logistic probes and human evaluators could still identify their training origins. The study’s sharper warning comes from Greedy Coordinate Gradient optimization on Gemma-3-12B-IT: FVE rose from 72.1% to 81.4%, while readabil


EleutherAI’s SOAR 2026 study delivers a clear warning: a Natural Language Autoencoder can reconstruct model activations with high fidelity without producing a faithful explanation of what those activations encode.

How the language bottleneck works

An NLA contains two modules. The Activation Verbalizer (AV) converts an internal activation vector into natural-language text, while the Activation Reconstructor (AR) maps that text back into activation space. This language bottleneck creates an inspectable and editable interface: researchers can both audit internal representations and intervene in model behavior by changing the explanation.

The study measures reconstruction quality with the Fraction of Variance Explained (FVE). For activation $x$, reconstruction $x̂$, and mean activation $μ$:

FVE = 1 - Σ||x - x̂||² / Σ||x - μ||²

A high FVE means that AR numerically approximates the original activation with less squared error than a mean-only baseline. It does not establish that the readable content of the explanation captures the information encoded by the activation. The team therefore also measures passage support: whether statements in the explanation are supported by the source passage. Passage support is still weaker than faithfulness to an activation, since an activation may encode predictions, confidence, or information not explicitly stated in the passage.

Cross-seed transfer does not imply shared semantics

The researchers trained four LoRA-based EasyNLA AV–AR pairs on layer-20 outputs from Qwen2.5-7B-Instruct and evaluated them on 127 held-out prompts. In the cross-reconstruction matrix, the explanation produced by AV$i$ was reconstructed by AR$j$ for every pairing.

Diagonal FVE values were highest, as expected because each AV was trained jointly with its corresponding AR. Yet the off-diagonal values remained close to the diagonal and substantially exceeded results obtained after ordinary paraphrasing. This suggests that independently trained NLAs can share conventions or pair-specific encodings rather than relying entirely on codes unique to one training run.

Transferability, however, is not semantic alignment. Logistic-regression probes distinguished text from different training runs with high accuracy. A human evaluator completed 42 matching trials and achieved 55% accuracy, above the 33% random baseline. Probes may exploit stylistic or formatting artifacts, while the relatively weak human result indicates that the underlying code is not readily interpretable as a common human-readable meaning. The experiment establishes numerical exchangeability, not a shared semantic representation.

GCG turns explanation optimization into a reconstruction attack surface

Using the released layer-32 NLA for Gemma-3-12B-IT, the team applied Greedy Coordinate Gradient (GCG) to search for token sequences that AR could reconstruct more accurately. The objective can be viewed as maximizing reconstruction FVE while penalizing deviation from the AV language distribution through negative log-likelihood.

The original AV explanations achieved an FVE of 72.1% ± 0.5%. With no language-model likelihood penalty, GCG raised FVE to 80.7% ± 0.5%; a small penalty weight of $β=0.01$ produced the highest reported value of 81.4%. The gains came with worse linguistic quality. Mean NLL rose from 3.84 nats for the original explanations to 4.63 at $β=0.01$ and 5.58 at $β=0$. Passage support fell from 17.5% to 14.3%. The study also included random-token substitutions matched on edit count as controls.

These results indicate that AR can be exploited through discrete token optimization: surface-level changes can improve numerical reconstruction without improving readability or consistency with the source passage. The findings are consistent with nonsemantic encoding or reconstructor exploitation, but they do not prove that the original AV explanations rely on steganography. Jointly reinforcement-learned AV explanations also reconstructed activations better than Sonnet-generated explanations despite having lower passage support, again showing that reconstruction and linguistic faithfulness are separate properties.

Text edits can redirect behavior, while robustness can be misleading

On the tested simple tasks, paraphrasing caused little additional performance degradation once the effect of NLA reconstruction was considered. However, some tasks remained solvable even when the activation at the patched position was deliberately mismatched. Behavioral robustness therefore does not by itself prove that the model causally depends on the patched activation or explanation.

Structured edits can also redirect outputs toward supplied target answers, including answers associated with a different task. NLA is consequently a bidirectional interface: it can expose internal behavior for auditing, but it can also become a behavioral control channel. Neither readability nor reconstruction accuracy indicates whether such control remains semantically appropriate or stays within the intended task boundary.

Four dimensions must be evaluated separately

A production NLA evaluation should not collapse into one score. It should measure:

  • Reconstruction fidelity: FVE and cross-reconstruction across independently trained pairs or decoders;
  • Linguistic quality: NLL, perplexity, readability, and adherence to the base model’s expression distribution;
  • External consistency: whether the explanation is supported by the source passage;
  • Behavioral causality: whether edits produce predictable, position-sensitive output changes.

The evidence remains limited to particular models, layers, and tasks. Passage support depends on a judging prompt and can inherit LLM-evaluator bias. Probe accuracy may reflect style shortcuts. GCG also requires repeated evaluations of the reconstructor and, when enabled, the language-model scoring objective; the study does not report latency or compute cost. NLA is consequently better suited to offline interpretability audits than to a real-time trusted explanation service.

Practical deployment guidance

  1. Treat NLA output as a hypothesis, not as ground truth about an activation.
  2. Maintain a multidimensional scorecard covering FVE, NLL, passage support, and behavioral effects instead of optimizing reconstruction alone.
  3. Run cross-seed and cross-reconstructor tests to distinguish genuinely shared semantics from formatting conventions or paired artifacts.
  4. Add counterfactual intervention controls, including mismatched activations, random locations, irrelevant edits, and cross-task targets.
  5. Restrict explanation-editing privileges. If users can alter NLA text before a high-privilege model executes, output changes should be policy-checked, approved where necessary, and fully audited.
  6. Keep GCG in offline research workflows. Its iterative discrete search is costly, so production systems should prefer precomputed explanations, bounded edit budgets, and explicit natural-language-quality constraints.

Source: EleutherAI Blog (2026-09-22)

— END —

Tags: #NLA #Mechanistic Interpretability #Activation Reconstruction #GCG #LoRA

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…