Video-Coder Open-Sources an Agentic, Code-Driven Video Generation Framework

Summary:Video-Coder reframes video generation as an agentic workflow: the model writes and executes code to construct a dynamic 3D scene, then passes its structural and motion cues to a downstream video generator. The project open-sources the Video-Coder-SFT-6k and Video-Coder-RL-2k datasets, model artifacts, code, and the Video-Coder Harness, which unifies code execution, 3D rendering, and execution feedback for training and inference. The model is trained with supervised fine-tuning followed by harne

---

Core Idea: Construct an Executable Spatial Plan First

Video-Coder does not directly predict continuous video frames from a text prompt. Instead, it turns generation into a verifiable code-generation and execution process: the model writes scene code, constructs a dynamic 3D environment containing objects, materials, cameras, and motion, and produces a Draft Video whose structural and motion cues are passed to a downstream video generator.

The approach addresses a structural weakness in conventional video generation. Diffusion and autoregressive models often fit distributions directly in latent or pixel space, making them prone to topology changes, spatial-relation drift, discontinuous trajectories, and inconsistencies between camera motion and scene layout. Program code provides a discrete, executable, and inspectable intermediate representation that can be evaluated before expensive video rendering begins.

The central contribution is therefore not a single video backbone, but a code-driven paradigm organized around a loop: write a scene, execute it, inspect feedback, revise the code, and generate the final video.

Video-Coder Harness: Turning Code Execution into a Reinforcement-Learning Environment

The study introduces the Video-Coder Harness, which integrates three components into one interactive environment:

1. Code execution: runs the generated scene program and captures syntax errors, missing dependencies, and runtime exceptions.
2. 3D scene rendering: converts the program into a Draft Video for inspection of object placement, occlusion, camera movement, and dynamic behavior.
3. Execution feedback: returns errors, rendered outputs, and environment states so the model can revise its program in the next iteration.

The Harness creates an optimizable closed loop rather than relying only on static prompt-video pairs. Training trajectories contain actions, observations, and recovery behavior, allowing the agent to learn how to handle non-executable code, repair geometric layouts, and progressively optimize video quality.

This design also introduces failure modes absent from direct diffusion generation. Graphics-library version differences, random initialization, floating-point variation, missing assets, and renderer nondeterminism can all alter the reward signal. If the training environment differs from production, the agent may learn to exploit features of a particular renderer rather than acquire robust scene-planning ability.

Dual Rewards and Reward Annealing

Video-Coder first undergoes supervised fine-tuning on Video-Coder-SFT-6k and is then trained through harness-native agentic reinforcement learning. Two feedback channels are used:

• Execution rewards measure whether the code runs successfully and whether environment feedback matches the intended program behavior.
• Video rewards evaluate the Draft Video or final output in terms of visual quality, motion, and task completion.

Execution rewards alone can encourage code that runs but produces an empty or visually trivial scene. Video rewards alone are more expensive, exhibit higher variance, and may reward malformed code that happens to produce an appealing frame. The training procedure uses reward annealing: execution feedback receives greater weight early so that programs remain executable, while the later stages increasingly optimize video quality.

This is a curriculum-style compromise, but its effectiveness depends heavily on the video reward model. If the evaluator prefers compositions, materials, and motions common in the training set, the agent may overfit to benchmark aesthetics rather than learn a broadly transferable scene-planning capability.

Datasets and Open Resources

The team created two datasets:

• Video-Coder-SFT-6k: approximately 6,000 diverse prompts and reference videos for establishing code-generation and scene-construction ability.
• Video-Coder-RL-2k: approximately 2,000 examples for environment-based reinforcement learning.

The project also releases data, model artifacts, and code, allowing developers to reproduce training, substitute downstream video generators, and study agent behavior in graphical execution environments. This release scope is more valuable than an inference demo alone because code-driven video generation depends not only on weights but also on an executable environment, interaction traces, and reward implementations.

How to Interpret the Benchmark Gains

The paper reports that Video-Coder improves Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0. PhyGenBench emphasizes physical and spatial consistency, while VBench-2.0 covers a broader range of video-generation capabilities. These results support the proposition that structured scene planning can improve downstream generation.

The gains should not be interpreted as equivalent improvements in perceived video quality. Benchmark movement may reflect better prompt adherence, composition, motion control, or adaptation to the evaluation distribution. Important questions remain:

• Do gains transfer to downstream generators not seen during training?
• Does the system use extra inference budget or a stronger reward model?
• How are code failures and rendering anomalies scored?
• How does the approach compare with direct generation, 3D ControlNet, and procedural-video baselines?
• Does performance remain stable across prompt lengths, shot counts, and scene complexity?

Engineering Trade-offs: Better Structure at the Cost of Latency and Complexity

Code-driven generation improves editability, verifiability, and reuse, but it lengthens the pipeline. A single output requires code generation, sandbox execution, 3D rendering, feedback-driven revision, and final diffusion-based rendering. Any stage can become a bottleneck. Complex scenes can also increase geometry counts, render time, and GPU memory consumption.

Compared with direct video generation, the approach may deliver more stable composition and motion at the cost of higher latency, more complicated dependency management, and propagation of code or rendering failures into the final output. Production deployments should impose iteration limits, deterministic random seeds, rendering-resource quotas, and fallback paths. Teams should track code execution success separately from final video quality rather than relying only on an end-to-end score.

Practical Guidance for Developers

Researchers should reproduce the Harness loop first and evaluate the Draft Video separately from the final video. Optimizing only the final score can hide failures in code execution. Training artifacts should retain programs, rendered outputs, feedback logs, and environment versions to expose reward hacking and error propagation.

Application developers can use Video-Coder for advertising storyboards, game prototyping, embodied simulation, and film previsualsization, where spatial and temporal constraints matter. The generated code can also serve as an editable intermediate representation: camera or object trajectories can be changed and re-rendered without regenerating the entire video.

The limitation is equally important. Code does not replace high-quality assets, stable physics simulation, or a strong video generator. It is better understood as an explicit intermediate layer for spatial planning and rendering control. Generalization to open worlds, complex interactions, and long multi-shot sequences still requires independent evaluation and production-scale testing.

---

Source: Hugging Face Daily Papers (2026-11-06)

— END —

Tags: #Video-Coder Follow #Video Generation Follow #Coding Agents Follow #Reinforcement Learning Follow #3D Scenes Follow