Tracking Internal Representation Dynamics and Error Lock-In in LLM Iterated Games
Summary:This paper uses iterated games with incomplete information to ask whether an LLM's visible chain of thought reflects the computation that actually drives its decisions. By examining layer activations alongside literal reasoning text, the authors report that the model builds and revises an internal representation of the game across turns. Once it commits early to a wrong belief, recovery is rare; the lock-in is visible in both generated tokens and activation trajectories. The contribution is a m
Core Finding
The study moves beyond asking whether an LLM can produce a coherent chain of thought to asking whether that visible trace corresponds to the computation actually driving its decisions. Its testbed is an iterated game with incomplete information: an agent acts from limited observations and must continually maintain a belief about hidden state. The authors report that the model forms and repeatedly revises an internal representation of the game while generating its CoT. Once it prematurely commits to an incorrect belief, recovery is rare, and the failed commitment appears in both the literal reasoning text and layer activations.
Reasoning traces therefore cannot be treated only as post-hoc explanations. In autoregressive generation, the hidden state used to produce the next token depends on text that has already been generated. CoT becomes part of the subsequent context and can influence later computation. The paper's central value lies in examining visible text and layer activations as two complementary evidence channels: whether the model's internal state encodes the same strategic belief as its words, when that belief is updated, when it becomes fixed, and when errors begin to amplify.
From Visible Reasoning to Internal State
At an architectural level, the hidden state at layer l and position t can be abstracted as h_l(t)=F_l(h_{l-1}(t), h_l(t-1), c_t), where c_t summarizes the current input and generated context. A probe could map a set of layer activations into an estimate of the latent game belief, producing an analytical representation such as B_t=g({h_l(t)}), which is then aligned with the reasoning token at the same position. The objective is not to require the model to verbalize the correct conclusion first, but to identify when its internal state diverges, whether later observations can correct it, and where commitment occurs.
The abstract says the authors used a series of mechanistic interpretability techniques, but it does not disclose whether these were linear probes, attribution methods, activation differencing, or direct interventions. It would therefore be premature to attribute a specific method to the paper. A safer interpretation is that the authors align CoT text, layer activations, and game turns as temporal signals, then use them to locate belief formation and commitment.
If activations change before the text does, the internal judgment may already have shifted without being verbalized. If the text changes first and activations follow, explicit reasoning may be shaping later computation. Distinguishing these causal directions requires counterfactual intervention; temporal correlation alone is insufficient.
Why Erroneous Beliefs Lock In
In an incomplete-information game, an agent's action changes the information available on the next turn. An erroneous commitment can create a feedback loop: the model forms a false belief, acts on it, and then interprets the environment's response as confirmation of that belief. The more active the strategy, the more readily the agent can generate selective evidence that appears to validate its original assumption.
Multi-agent settings can amplify this failure. An opponent may adapt to behavior that the model has already exposed, causing the incorrect belief to receive both environmental feedback and strategically adversarial pressure. The paper's claim that recovery is rare is therefore more than a text-quality issue. It suggests a path-dependent process in which later reasoning is organized around an existing hypothesis rather than remaining open to competing explanations.
Engineering Trade-offs and Empirical Evidence
Internal-state tracking is not free. If hidden states from every layer are retained at every step, FP16 hidden-state storage alone is approximately 2 × L × T × D bytes, where L is the number of layers, T the token count, and D the hidden dimension. This excludes attention and MLP intermediates. For L=32, T=2,000, and D=8,000, one trace approaches 1 GB. Concurrency, tool calls, and visualization data multiply that burden. A production-oriented implementation would sample critical layers, project activations into a compact representation, or aggregate them in a streaming fashion rather than retaining every tensor.
A probe can itself be lightweight, but three methodological risks remain:
- Shortcuts and overfitting: A probe may identify prompt templates, opponent names, or stylistic tokens instead of game beliefs. It must be tested on unseen games, counterfactual histories, and paraphrased prompts.
- Correlation is not causation: Without activation patching, ablation, or steering, the study can show that an erroneous belief and activation trajectory co-occur, but not that a particular activation causes the policy.
- Selective-example bias: The interactive visualizer makes it possible to inspect selected reasoning traces and is useful for qualitative diagnosis. Selected examples, however, cannot replace results over randomly sampled cases.
The supplied material does not identify model names or sizes, the number of games, opponent models, random seeds, prompt templates, recovery criteria, absolute accuracy, or confidence intervals. It also provides no comparison against outcome-only prediction, text-based CoT error annotation, or other diagnostics. The statement that recovery is rare therefore remains unquantified, and generalization across models and game families has not been established.
Practical Guidance
- Begin with offline diagnosis. Record observation histories, actions, CoT, model versions, and opponent policies, then extract only the layer activations needed for the target hypothesis. Do not label reasoning quality solely from final win or loss.
- Locate commitment with change-point detection. Combine a calibrated belief probe with trajectory change points to identify when the model shifts from revising a hypothesis to stably maintaining it, and test whether subsequent evidence actually changes that state.
- Build counterfactual replays. Hold wording constant while replacing hidden states, observation histories, or opponent types. Whether actions and activations respond appropriately is a stronger test than inspecting randomly selected traces.
- Treat interventions as hypotheses. Re-asking, parallel branches, memory rollback, and independent critique agents might break an erroneous commitment, but the abstract does not validate them. Paired experiments should measure recovery, game utility, and Token cost.
- Govern production observability. Activations and complete traces may expose private strategies, opponent information, and user data, requiring sampling, redaction, access controls, and retention limits.
Conclusion
The work reframes CoT debugging as internal-state debugging. The central question is not only whether a model writes down a conclusion, but when it forms a belief, when it stops revising it, and how an incorrect assumption becomes self-reinforcing across turns. The mechanistic perspective and trace visualizer are useful, but on the evidence currently available, the work is best treated as a strong research signal for multi-agent reasoning diagnostics rather than a production-grade reliability guarantee.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-19)
Tags: #LLM Agents #Mechanistic Interpretability #Iterated Games #Reasoning Traces #Representation Tracking
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments