Ctrl-World
Stanford University, United States, China · October 2025
Stanford University and Tsinghua University
World model · Open weights, MIT
- Parameters
- not published
- No total is published.
- GPU memory
- not published
- weights at bf16, computed
- Inference latency
- not published
- per action step
- Weights
- MIT
- MIT for the repository and the Hugging Face checkpoint.
What it is
Ctrl-World is an action-conditioned world model from Stanford and Tsinghua, built to evaluate and improve generalist robot policies in imagination. Initialised from the 1.5B Stable Video Diffusion model, it predicts three camera views jointly, conditions every frame on 7-dimensional Cartesian actions and retrieves pose-matched memory frames, which keeps rollouts consistent for more than 20 seconds. Trained on DROID (95k trajectories, 564 scenes), it lets π0.5 run inside it, and policy rankings in imagination track the real ones, although the model tends to underestimate real success. Fine-tuning π0.5-DROID on 25 to 50 selected imagined trajectories per task raised its success on new instructions from 38.7 to 83.4 percent on average.
Architecture
- Backbone
- Stable Video Diffusion (1.5B) as initialisation, extended to three jointly predicted camera views, with CLIP image and text encoders.
- Action head
- No action output. A 3-layer MLP embeds 7-dimensional Cartesian actions (15 per second, downsampled to 5 steps) and frame-wise cross-attention conditions each predicted frame on them. Actions come from the policy under test, such as π0.5.
- Parameters
- No total is published. Initialised from the 1.5B Stable Video Diffusion model; the README lists the Ctrl-World checkpoint at about 8 GB and the CLIP encoders at about 600M.
- Pretraining data
- DROID: about 95k trajectories from 564 scenes on a Franka arm, three camera views.
- Embodiments
- Franka (DROID setup)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
No memory figure. The README gives about 10 s per interaction step (one 1 second action chunk) on an A100 and about 5 s on an H100. Training used 2 x 8 H100 GPUs for about 2 to 3 days.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
No results in this category are published for this model.
Fine tuned tasks
What the model reaches after being adapted to a new task or a new robot, usually from a small number of demonstrations. This is the number that matters if you bring your own data.
- DROID Franka setup (Ctrl-World paper protocol) π0.5-DROID after post-training on imagined trajectories, average over instruction-following tasks83.4%average success rateAuthors' own evaluation of the improved policy. Before post-training 38.7 percent. The number measures π0.5 after training on Ctrl-World rollouts, not the world model itself; trial counts are not stated.source
- DROID Franka setup (Ctrl-World paper protocol) π0.5-DROID after post-training on imagined trajectories, novel objects75%
On real hardware
π0.5-DROID fine-tuned on Ctrl-World rollouts: spatial understanding 28.75 to 87.5 percent, shape understanding 43.74 to 91.25, towel folding with direction 57.5 to 80, novel objects 25 to 75, average 38.7 to 83.4. Trial counts are not stated. For evaluation, the paper compares results in the world model with real ones per task, for example π0.5 drawer success 0.80 real and 0.30 in the world model.
Fine tuning it yourself
The repository covers replaying DROID trajectories, keyboard control, interaction with π0.5 inside the world model and training on DROID. Policy post-training in the paper: 2k steps on 4 H100 GPUs with 25 to 50 hand-picked imagined trajectories per task, chosen from 400 generated ones.
Where it helps, where it does not
Strengths
- Lets a VLA run closed loop inside a learned model, so policies can be compared without robot time.
- Clear policy gain from imagined data: π0.5-DROID went from 38.7 to 83.4 percent on new instructions.
- Better video fidelity than IRASim and WPE on DROID in the multi-view setting: PSNR 23.56 and FVD 97.4 against 20.21 and 165.4 for IRASim.
- Open code and weights under MIT.
Limits
- Underestimates real success; π0.5 drawer success was 0.80 on the real robot and 0.30 in the world model.
- Slow: about 5 s per interaction step on an H100.
- Imagined trajectories were filtered by hand: 25 to 50 kept out of 400 per task.
- Trained on DROID (Franka) only.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2510.10125
- https://arxiv.org/html/2510.10125v3
- https://github.com/Robert-gyj/Ctrl-World
- https://huggingface.co/yjguo/Ctrl-World
- https://ctrl-world.github.io/
Entry last checked 2026-11-12.
Try a policy on a real arm
A real SO-100 runs on /live and can be driven from the browser when it is online. Five of the models in this arena can be fine tuned on a dataset you record with your own arm.