Video-Coder Agent Scripts 3D Scenes Before Video Gen

A multi-institution research team has released Video-Coder, an open-source agent that writes and executes Blender Python code to construct 3D draft scenes before passing control to a video generator. The system, developed by researchers at the University of Central Florida, University of Würzburg, Microsoft Research, Tsinghua University, UNC-Chapel Hill, and UW-Madison, reports benchmark gains of approximately 7 points on PhyGenBench and 24 points on VBench-2.0 when applied to the MiniMax-H3 backbone.

The approach represents a shift from asking diffusion models to infer physics and structure from pixels alone. Instead, Video-Coder autonomously scripts dynamic 3D scenes that render explicit motion and geometry signals, which then condition the downstream generator with structured inputs rather than relying on latent representations.

How Video-Coder Works

Video-Coder operates in two stages. First, the agent writes Blender Python code to assemble a 3D scene with objects, camera motion, lighting, and dynamics. Second, that scene is rendered as a draft video, which serves as a conditioning signal for the video generator. The generator sees explicit structure rather than attempting to hallucinate physics from scratch.

The team describes the system as the first code-driven video generation agent, framing the draft video as a programmable scaffold that encodes physical relationships before pixel synthesis begins. By separating scene construction from rendering, the pipeline gives the generator a clearer prior on what should move, where, and how.

Training Pipeline and Datasets

Training hangs on two new datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, built by wiring Gemini 3.8 Flash into a custom Video-Coder Harness. The harness runs code, renders 3D scenes, and returns execution feedback in one loop. Supervised fine-tuning teaches the base scene-programming primitives; a second stage of agentic reinforcement learning then optimizes two rewards simultaneously:

• Execution reward: whether the generated program runs without errors
• Video reward: the quality of the finished video output

An annealing schedule gradually reduces the weight of the execution reward over time, allowing the model to rely more on the video reward in later training stages. This shift encourages the agent to prioritize visual fidelity once it has mastered basic code correctness.

Benchmark Results

Applied to MiniMax-H3, Video-Coder delivered approximately 7-point gains on PhyGenBench and 24-point gains on VBench-2.0. PhyGenBench measures physical plausibility in generated video, while VBench-2.0 evaluates broader visual quality metrics including consistency, motion smoothness, and semantic alignment.

The paper quotes averaged benchmark deltas on a single backbone. Per-prompt failure modes and head-to-head comparisons against other generators are not reported in the abstract. The team has open-sourced the data, models, and code on Hugging Face, making the system reproducible for further research.

Release Date and Availability

Officially released on November 9, 2026, Video-Coder joins a growing wave of agentic pipeline releases in generative AI. The open-source release includes the full training harness, datasets, and inference code, enabling researchers to extend the approach to other video generators or 3D rendering engines beyond Blender.

What This Means

Video-Coder demonstrates that injecting an executable intermediate representation between prompt and pixel can improve both physical plausibility and visual quality. By teaching an agent to script scenes rather than relying on end-to-end diffusion alone, the system offers a composable architecture where improvements in code generation, 3D rendering, and video synthesis can compound. The open-source release lowers the barrier for labs exploring hybrid symbolic-neural video pipelines, and the benchmark gains suggest that code-driven conditioning is a viable lever for scaling video generation quality.