Continuation Feedback Enables Reusable Craftax Policies Without Inference-Time LLM Calls
Summary:The paper introduces a continuation-guided framework that turns coding agents into offline synthesizers of executable policies. In Craftax, a partially observable and procedurally generated game environment, an agent synthesizes a rule program and refines it through simulator feedback; the resulting program then runs without any inference-time LLM call. The central mechanism is a feedback cascade that replaces a single sparse episode return with hierarchical goals and belief updates, giving loca
Core result
This work moves the LLM out of the online control loop and into an offline program-synthesis stage. In Craftax, a coding agent generates and revises an executable rule program using simulator feedback. At deployment, a fixed interpreter executes the resulting program without making any inference-time LLM calls.
The contribution is more than replacing a neural policy with source code. It separates capability acquisition from capability reuse. The agent first develops a competent task policy through a continuation-guided feedback cascade, then optimizes a goal compiler that maps a structured task g to a new program p_g. That distinction matters: moving model calls offline only reduces cost if the learned structure can be reused across objectives rather than requiring a fresh, expensive search for every task.
How the feedback cascade addresses long-horizon credit assignment
Craftax is partially observable and procedurally generated. An episode return alone is a sparse optimization signal. A local code edit may improve one intermediate segment while changing downstream behavior, ultimately reducing final success.
The proposed cascade introduces progressively denser signals:
- Episode returns retain the global outcome and determine whether the complete task was solved.
- Hierarchical goals expose intermediate milestones that can be scored separately.
- Belief updates provide additional dense signals derived from state changes in the simulator.
The conceptual role of continuation scoring is to evaluate an edit not only by its immediate prefix behavior but also by the progress and state transitions it enables later. The abstract does not reveal the scoring equation, continuation horizon, normalization, or acceptance rule, however, so the method should not be reduced to a conventional reward-shaping heuristic without consulting the paper.
The controller is then factorized into a fixed environment interpreter I, containing models, memory, and skills, and an editable rule program p_0. This creates two distinct stages:
- Competence formation: a coding agent proposes, tests, and revises programs around
p_0, using the feedback cascade to obtain a competent policyp*. - Competence reuse: a goal compiler
C_θis optimized to translate a new structured taskginto a new programp_g, which the same interpreter can execute.
The resulting flow is:
g → C_θ → p_g → I → Craftax
Here, “zero-shot” means that no task-specific tuning is performed for the longer evaluation tasks. It does not mean that the compiler is untrained. The paper denotes the optimized compiler as C_θ, but the available description does not specify whether it is a neural module, a parameterized code template, or a hybrid representation. That missing implementation detail materially affects reproducibility and deployment.
Reported performance and evaluation limits
The paper reports the following central Craftax comparison:
| Feedback method | Final-stage arrival | Final-boss defeat | |---|---:|---:| | Episode-level refinement only | Below 1% | Not reported | | Feedback cascade | 50% | 28% |
The gap indicates that increasing the information density of intermediate feedback may be more important than further tuning a single terminal reward in long-horizon program optimization. A 50% final-stage arrival rate and a 28% boss-defeat rate show meaningful long-horizon competence rather than merely successful execution of short fragments.
For the compiler experiment, the paper reports five independent searches trained only on short tasks, followed by evaluation on longer tasks. The resulting compilers reportedly transfer zero-shot without task-specific tuning or runtime LLM calls. The abstract does not provide a transfer success rate for each search, worst-task performance, or variance, so the reported transfer remains qualitative.
Important limitations remain:
- The evaluation covers one simulator and does not establish transfer to robotics, industrial workflows, or other real systems.
- The coding model, candidate-program count, search iterations, simulator time, hardware, and total token cost are not disclosed.
- No seed count, confidence interval, or significance test is reported. The gap from below 1% to 50% is striking, but it still requires independent reproduction.
- The baseline is described only as episode-level refinement. There is no equal-compute comparison with inference-time LLM agents, conventional planners, or other program-search methods.
- Dense belief-update scoring appears to rely on simulator state or privileged observations. Whether equivalent signals exist in real tasks remains unknown.
Procedural generation offers evidence of within-environment generalization, but it does not automatically establish sim-to-real transfer. A goal compiler trained against one interpreter interface may also learn Craftax-specific state structures and regularities rather than task-general compilation knowledge.
Engineering trade-offs
1. Lower online cost, but not lower total cost
Policy execution no longer requires token generation, model-service round trips, or model-side inference jitter. With the interpreter, environment, and program versions fixed, the same input is easier to reproduce, although stochastic components in the environment or underlying models can still introduce nondeterminism. The paper provides no latency, throughput, or memory measurements, so no concrete speedup should be inferred.
Instead, cost moves into synthesis. The coding agent must generate candidates, run the simulator, inspect feedback, and revise code repeatedly. Production evaluation must include offline LLM tokens, simulator time, and search iterations rather than highlighting only the absence of runtime LLM calls.
2. Dense feedback depends on verifiable intermediate state
Hierarchical goals and belief updates are useful because the simulator can determine whether a program advanced a subgoal or updated internal state correctly. Real environments usually lack privileged signals and may require process logs, rule-based validators, learned reward models, or human feedback. If an intermediate proxy diverges from the final objective, the program may optimize the proxy rather than the task.
3. Executable code improves auditability but expands the attack surface
A rule program can be inspected, statically analyzed, and unit-tested more easily than an opaque neural policy. Code generated by an LLM can nevertheless contain faulty logic, resource leaks, infinite loops, or unauthorized operations. Program generation and execution must therefore be isolated behind API allowlists, network restrictions, CPU and memory limits, and differential tests.
4. The fixed-interpreter design creates version coupling
Placing general capabilities in I and task-specific behavior in p_g reduces the search space, but changes to memory structures, skill interfaces, or state semantics can silently invalidate old programs. Production systems need separate versions for the interpreter, program schema, and task definition, together with replayable golden traces for regression testing.
5. “Zero-shot transfer” still carries search variance
Five independent compiler searches suggest the result is not based on one lucky run, but they also imply that the optimization process may vary substantially. A complete evaluation should report the mean, worst search, and failure taxonomy for every run, then test unseen procedural seeds, longer horizons, and different goal structures.
Deployment guidance for developers
- Establish a reproducible baseline: fix simulator, interpreter, model, and seed versions, then compare episode-only feedback, the full cascade, and task-specific recompilation under equal budgets.
- Enforce safety at the execution boundary: let the agent submit patches or source code rather than granting direct execution privileges. Apply API allowlists, network isolation, resource limits, and differential testing.
- Separate three interfaces: define structured goals, rule programs, and environment interpreters independently, and retain replayable traces whenever any component changes.
- Measure lifecycle cost as well as success: record offline LLM tokens, candidate programs, simulator runtime, compile failures, runtime p50/p95 latency, memory use, and worst-case long-task performance.
- Redesign feedback for real environments: when simulator state is unavailable, use verifiable intermediate artifacts, process reward models, or rule-based scoring, and explicitly test for reward exploitation.
- Reproduce the public implementation: the project exposes its code at
github.com/vicgalle/craftax-program-policies, enabling fixed-environment replication and the missing variance and ablation experiments.
Bottom line
The central value of this work is a shift from letting an LLM act directly toward using it as an offline synthesizer of reusable programs. Continuation-guided feedback addresses credit assignment in long-horizon program optimization, while the interpreter/compiler split offers a plausible route to amortizing synthesis cost across tasks.
The 50% final-stage arrival rate and reported short-to-long transfer are promising, but the evidence still comes from one simulator. The decisive engineering question is whether the goal compiler learns transferable task structure rather than accidental regularities in Craftax's interfaces and reward function.
— END —Source: Hugging Face Daily Papers (2026-11-14)
Tags: #Coding Agents #Program Policies #Continuation Feedback #Craftax #Goal Compiler
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments