A 16.7M-Parameter Diffusion Model Makes Pokémon Red Playable Inside a Learned World
By Mr.Xu Community Post
Published:
Summary:Developer Priyanshu Sah has released a playable diffusion world model and a hierarchical game-agent harness for Pokémon Red. The 16.7M-parameter model predicts 72×80 grayscale frames from the previous four frames and a button action, periodically resynchronizing with the real emulator every ten steps. A System 2 planner reads emulator RAM and chooses goals, while ten 1.58M-parameter System 1 policies handle navigation. The navigation policies were trained entirely inside the learned world model
How a compact diffusion model turns a game into an interactive environment
The project avoids training one monolithic model for the entire game. It separates the system into three layers: a 16.7M-parameter diffusion world model that generates the next frame, a System 2 planner that reads game state and selects goals, and ten System 1 policies, each with 1.58M parameters, that perform navigation.
The world model uses a DIAMOND-style EDM U-Net. It receives the previous four 72×80 grayscale frames and a button action, then predicts the next frame. Button presses are discrete conditioning signals, while frames are encoded and injected into the denoising network. The model therefore learns a conditional state-transition distribution rather than directly estimating action values.
During closed-loop execution, the system reads four real emulator frames every ten generated steps. Between those synchronization points, it continues from its own predictions. Periodic realignment is operationally useful because diffusion models accumulate state drift over long trajectories, but it also means the agent is not demonstrating fully autonomous long-horizon consistency.
The System 1 / System 2 split
The slower System 2 planner reads emulator RAM and maintains higher-level objectives such as leveling, healing, travel, and gym progression. It can consult the Pokédex or Bulbapedia, operate through either rules or a local LLM, and verify decisions against memory before execution.
The fast System 1 layer consists of ten small navigation policies. The researcher trained them entirely through imagination rollouts in the diffusion world model, without collecting real-game interaction data, and later transferred them into the emulator. This lowers real-environment interaction cost and permits extensive exploration in a generative environment where failed attempts are relatively cheap.
Six local LLMs were tested as planners, and five achieved 10/10 success in obtaining the first gym badge. The best planner required a median of 7,144.5 button presses, compared with 8,779.5 for the hand-written rules. Button count is useful but incomplete: it may omit path length, waiting time, recovery cost, and post-task state quality.
Engineering cost and model scale
Training reportedly consumed 50 GPU-hours on one RTX 4090. The 72×80 representation reduces each frame to 5,760 pixels, making repeated diffusion sampling tractable for a compact U-Net.
The tradeoff is inference cost. Diffusion models normally require multiple denoising passes per prediction and are therefore more expensive than one-step autoregressive or latent dynamics models. The post does not disclose denoising steps, generated frames per second, or end-to-end latency, so real-time playability cannot be inferred from the training budget alone. Evaluators should request sampling-step count, peak VRAM, CPU emulator overhead, and sustained frame rate.
Recovering a watchable image from grayscale output
Two additional networks handle presentation. A 0.72M-parameter upsampler expands frames from 72×80 to 144×160, while a 1.91M-parameter per-pixel colorizer restores color. The author reports 99.27% color accuracy on held-out frames, with about 25 minutes of training for each network.
Pixel-level color accuracy on held-out frames is not equivalent to perceptual quality. Game assets may contain repeated colors, and an upsampler can also magnify temporal inconsistencies. The town reconstructed in Blender from map data is visually useful, but it is a separate rendering step rather than evidence that the world model itself generated a complete 3D environment.
Measured results and the major limitation
The full agent obtained the first gym badge in 30 out of 30 real-emulator runs. Battles remain scripted because the learned world frequently terminates them early. This negative result matters: the system demonstrates reliable navigation without demonstrating faithful modeling of combat dynamics.
The author also proposes using emulator RAM as a ground truth for measuring world-model drift. That is a stronger approach than judging video quality alone. A frame may look plausible while HP, inventory, or map coordinates have already diverged. A rigorous evaluation should therefore track image quality, RAM-state error, task success, synchronization frequency, and action consistency between imagination and the real simulator.
Reproduction and evaluation guidance
The author states that code, weights, and the full showcase are available. A reproducible evaluation should:
- Measure next-frame quality over single steps and short trajectories rather than relying only on generated videos.
- Record sampling steps, latency, memory use, and achieved frame rate to determine whether the system is genuinely interactive.
- Compare rule-based and LLM planners under fixed seeds, including complete trajectories and failure categories.
- Quantify distribution shift when transferring System 1 policies from imagination into the real emulator.
- Evaluate drift separately across visual frames, RAM state, and action successors, and report combat as an unresolved task.
This is a compact and technically legible personal project. It shows that a diffusion world model and a hierarchical agent can be combined into a playable system on consumer hardware. It remains a promising prototype rather than a peer-reviewed general-world-model result, particularly because combat fidelity and fully autonomous long-horizon rollout are still unsolved.
— END —Source: Reddit r/MachineLearning (2026-11-08)
Tags: #Diffusion World Model #Game Agent #System 1/System 2 #Model-Based Control #Pokémon Red
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments