Code as Explanations: Verifying LLM Behavior with Executable Programs

Summary:The paper introduces “code explanations”: instead of producing an unverifiable natural-language rationale for each input, an LLM generates one executable program that predicts behavior across a dataset. Running the program and comparing its outputs with the model defines a Consistency metric, which can rank candidate explanations and provide feedback for refinement. Experiments across three open-source code-capable LLMs, eight tasks, and Claude Opus 5.5 show that code explanations are 6–100× sh

---

From Natural-Language Rationales to Executable Explanations

This work does not attempt to generate a longer Chain-of-Thought (CoT). Instead, it changes the unit of explanation: the LLM produces one deterministic program that predicts behavior across an entire dataset. If the program’s outputs match those of the target model, agreement can be checked automatically by executing the program.

The central metric is Consistency:

[ C(P,M,D)=\frac{1}{|D|}\sum_{x\in D}\mathbf{1}[P(x)=M(x)] ]

Here, (P) is the generated explanation program, (M) is the model being explained, and (D) is the validation dataset. The metric measures how often the program reproduces observed model behavior; it does not establish that the program matches the model’s internal computation graph. Candidate programs can be ranked by this score, and inconsistent examples plus execution logs can be fed back to Claude Opus 5.5 for revision.

Why Programs Are More Testable

A conventional CoT is generated for one input at a time. It may describe a particular response, but it does not directly tell us whether the same rule applies to a new test case. A code explanation packages the rule as an executable artifact: inputs enter a function, outputs can be compared with labels or model answers, and the implementation can be subjected to unit tests, boundary cases, and counterexamples.

A practical system can implement a “generate–execute–compare–refine” loop:

1. Provide the model with inputs and the target task specification.
2. Ask for candidate programs, type signatures, and dependencies.
3. Execute candidates in a sandbox while recording exceptions, timeouts, and output differences.
4. Rank them by Consistency or task accuracy.
5. Feed failed cases and execution logs back into the model to generate or repair programs.

This turns explanation evaluation from manual reading into batch regression testing. Because the explanation is code, it can also enter static analysis, unit testing, and formal verification pipelines.

Results: Strong Compression, Uneven Fidelity

The study covers three open-source code-capable LLMs, eight tasks, and additional experiments with Claude Opus 5.5. Code explanations are 6–100× shorter than CoT and align better with observed model behavior in 9 of 24 model–task settings.

Shorter does not necessarily mean more faithful. On tasks with complex linguistic structure, consistency drops significantly and may fall near an uninformative baseline. Programs in these settings also rely more frequently on longer hard-coded patterns. This suggests that the model may be encoding surface features or lookup-like rules rather than extracting a compact, stable decision procedure.

Execution feedback improves consistency for Claude Opus 5.5 but does not remove the gap on complex tasks. Feedback can repair obvious errors; it cannot automatically solve the problem of modeling highly nonlinear or prompt-sensitive behavior with a deterministic program.

What Consistency Does—and Does Not—Prove

Consistency measures behavioral imitation, not causal explanation. A highly consistent program may still reproduce dataset-specific regularities or memorize labels through large lookup tables. Distinguishing genuine rules from overfitting requires additional checks:

• Consistency on unseen inputs and distribution-shifted data;
• Program size, embedded constants, string tables, and branch count;
• Whether perturbations affect the program and model in the same way;
• Synchronized failure rates on adversarial and counterfactual inputs;
• Runtime, memory use, dependency complexity, and sandbox reliability;
• Fair comparisons with rule learners, decision trees, and symbolic programs.

Consistency is therefore best treated as a metric for a verifiable behavioral surrogate, not standalone proof of the model’s true reasoning process. High consistency is necessary but not sufficient for a strong causal explanation.

Engineering Trade-offs

Executable explanations improve testability, but they also introduce operational costs.

Input cost: A meaningful test requires multiple inputs. A larger dataset improves statistical confidence but also increases execution time and infrastructure usage.

Output cost: Exact string matching can understate semantic agreement for open-ended text. Embedding similarity or an LLM judge is more flexible, but adds cost and nondeterminism. Structured labels, numerical tolerances, and semantic checks should therefore be applied at different levels.

Security: Generated programs must run inside a sandbox with no network access and strict CPU, memory, and execution-time limits. File, process, and system-call access should be restricted; an explanation system must not become an arbitrary code-execution endpoint.

Model cost: Feedback-driven repair requires additional inference tokens and tool-call rounds. In latency-sensitive deployments, generate several short candidates first and execute only the most promising or diagnostically valuable examples.

Deployment Guidance

Code explanations are most appropriate for classification, extraction, routing, rule-based decisions, and formatting tasks with clear input–output boundaries. They can complement model-behavior regression tests: after a model or prompt upgrade, execute an explanation program against a fixed suite to detect behavioral drift without relying entirely on manual answer review.

Recommended release gates include:

• Held-out Consistency rather than training-set fit alone;
• Program length, hard-coded constants, and conditional complexity;
• Sandbox success rate, median latency, and timeout rate;
• Label-level and semantic-level agreement with the target model;
• Synchronized error rates on counterfactual and adversarial inputs;
• Failure taxonomy covering logic errors, missing rules, boundary handling, and out-of-distribution cases.

If a program underperforms CoT on held-out data or relies mainly on lookup tables and long branches, it should not be presented as a reliable explanation. It can still serve as a lightweight surrogate if its limits are documented explicitly.

Research Significance

The paper’s main contribution is to move LLM explanations from unverifiable natural-language assertions toward executable, regression-testable artifacts. This is particularly useful for model-behavior test harnesses: after changing a model or prompt, teams can run an explanation program to expose drift rather than manually inspect every response difference.

What the experiments demonstrate is that code explanations can reproduce model behavior compactly in some settings and improve through execution feedback. They do not show that those programs are equivalent to the LLM’s internal reasoning, nor do they show that low consistency on linguistically complex tasks can be eliminated simply through larger models, more data, or stronger optimization. Verification has improved; causal depth remains an open question.

---

Source: ArXiv Machine Learning (cs.LG) (2026-11-13)

— END —

Tags: #LLM Interpretability Follow #Executable Explanations Follow #Consistency Metric Follow #Model Behavior Verification Follow #Code Generation Follow