World Models for Robotics: Video Prediction as Robot Control

A policy maps what the robot sees now onto what the joints should do next. A world model does something different: it maps what the robot sees now, plus a proposed action, onto what the robot would see next. That second thing is harder, more expensive, and for a while it has been the most heavily funded idea in robot learning. This page goes through what has actually been published, what the numbers say, and how far any of it is from an arm on your desk.

Three routes lead from a video predictor to a moving arm. You can plan inside the model, you can generate training data with it and hand that data to an ordinary policy, or you can turn the world model itself into the policy. All three have working implementations with open weights as of August 2026. All three cost far more compute than fine-tuning a vision-language-action model on 50 recorded episodes, and only the second one is something you can currently feed into a training run here.

Four different things get called a world model

The term covers at least four architectures that fail in different ways and cost different amounts. Sorting them out first saves a lot of confusion when reading benchmark tables.

The first family is old and well understood. Ha and Schmidhuber's World Models (2018) trained a controller entirely inside a hallucinated dream and transferred it back to the real environment. DreamerV3 pushed the same idea to over 150 tasks with a single configuration and became the first algorithm to collect diamonds in Minecraft without human data or curricula. Neither result involved a physical manipulator, and that gap is the whole story of the last few years.

The lineage, dated

Route 1: plan inside the model

The purest version is model predictive control. You have a goal image. You sample candidate action sequences, roll each one forward through the world model, score how close the imagined result is to the goal, keep the best, execute one step, repeat. No policy is trained at all. The world model plus a search loop is the controller.

Meta's V-JEPA 2 paper is the cleanest published measurement of this, because it ran both a latent-space predictor and a pixel-space one on the same robot with the same planner. The action-conditioned model V-JEPA 2-AC is a roughly 300M parameter block-causal transformer post-trained on under 62 hours of unlabeled Droid video, on top of an encoder pretrained on over 1 million hours of internet video. It was deployed zero-shot on two Franka arms in two different labs.

The compute comparison in the same paper is the part worth memorising. Both planners ran the cross-entropy method on a single RTX 4090. The Cosmos baseline, an action-conditioned video generation model based on latent diffusion (the open Cosmos-Predict1 7B weights fine-tuned on Droid), used 80 samples, 10 refinement steps and a planning horizon of 1, and needed 4 minutes to compute a single action. The paper notes that at that rate a full pick-and-place trajectory takes over an hour. V-JEPA 2-AC used 800 samples, ten times more, and needed 16 seconds per action. In that head to head, run in lab 2, Cosmos scored 80 percent on reaching against 100 percent, and 0 percent on grasping a cup against 60 percent.

V-JEPA 2's own limitations section is equally useful. The model infers its action coordinate frame implicitly from a monocular camera, and the authors state they manually tried several camera positions before settling on one that worked across experiments. Autoregressive rollouts accumulate error, so long horizons degrade. Goals have to be given as images, not language. Anyone who has fought a policy that only works in one setup will recognise all three failure modes.

Route 2: dream the data, then train an ordinary policy

This is the route with the best cost-to-benefit ratio today, and the only one you can currently connect to this platform. Instead of running the world model on the robot, you run it offline to manufacture episodes, recover actions for those episodes, and then train a conventional policy on the mixture. NVIDIA's DreamGen paper and the data collection question it tries to answer are the reference points here.

DreamGen is four stages. Fine-tune a video world model on video of your specific robot. Prompt it with a single frame plus language for behaviors and environments you never recorded. Recover pseudo-actions for each generated video using either a latent action model or an inverse dynamics model. Train the policy on the resulting video-action pairs, which the paper calls neural trajectories.

1. 1
   Install the Cosmos-Predict2 environment

   GR00T-Dreams builds on cosmos-predict2, so the setup is that repo's setup, plus a handful of extra packages the DreamGen scripts need.

   bash

   git clone https://github.com/nvidia-cosmos/cosmos-predict2
   cd cosmos-predict2
   uv sync --extra cu126
   source .venv/bin/activate
   
   # extra deps the GR00T-Dreams scripts need
   pip install openai tyro numpydantic albumentations tianshou \
       git+https://github.com/facebookresearch/pytorch3d.git

2. 2
   Post-train the video model on your embodiment

   The published example uses the GR1 humanoid dataset. The experiment name encodes model size, embodiment and resolution; the 2B config trains on 93-frame clips at 432x768 with batch size 1 per GPU.

   bash

   huggingface-cli download nvidia/GR1-100 --repo-type dataset \
       --local-dir datasets/benchmark_train/hf_gr1/
   
   python -m scripts.get_t5_embeddings_from_groot_dataset \
       --dataset_path datasets/benchmark_train/gr1
   
   EXP=predict2_video2world_training_2b_groot_gr1_480
   torchrun --nproc_per_node=8 --master_port=12341 -m scripts.train \
       --config=cosmos_predict2/configs/base/config.py -- experiment=${EXP}

3. 3
   Generate the dreams

   One image plus one instruction per dream. Guardrails are on by default and will silently drop generations, which is why the DreamGen benchmark instructions turn them off.

   bash

   python -m examples.video2world_gr00t \
       --model_size 14B \
       --gr00t_variant gr1 \
       --batch_input_json dream_gen_benchmark/gr1_object/batch_input.json \
       --disable_guardrail

4. 4
   Recover actions and write LeRobot format

   This is the step that turns video back into something a policy trainer can read. GR00T-Dreams ships preprocessing scripts for four embodiments: franka, gr1, so100 and robocasa.

   bash

   python IDM_dump/convert_directory.py \
       --input_dir "${COSMOS_PREDICT2_OUTPUT_DIR}" \
       --output_dir "results/dream_gen_benchmark/cosmos_predict2_14b_gr1_object_step3"
   
   # if your embodiment is not one of the four provided, train your own IDM
   PYTHONPATH=. torchrun scripts/idm_training.py \
       --dataset-path demo_data/robot_sim.PickNPlace/ \
       --embodiment_tag gr1

5. 5
   Fine-tune the policy on the mixture

   The repo's own recommendation is to raise batch size to the maximum that fits and train for 20k steps. At that point you are back to an ordinary VLA fine-tune on an ordinary dataset.

The results are worth stating carefully, because they are strong and narrow at the same time. A humanoid performed 22 new behaviors having been teleoperated on a single pick-and-place task in one environment. GR00T N1 trained on that pick-and-place data alone scored 0 percent on most novel behavior and environment tests. With neural trajectories added, new behaviors in seen environments went from 11.2 to 43.2 percent, and completely unseen environments reached 28.5 percent. Training only on neural trajectories, with no real data at all, gave 20.6 percent average success across 24 RoboCasa tasks. That is non-trivial rather than good, and the paper says so.

Route 3: make the world model the policy

The newest and, on benchmark numbers, the strongest route. Cosmos Policy takes Cosmos-Predict2-2B-Video2World and post-trains it once, on robot demonstrations, with no architectural changes at all. Actions, future proprioception, future camera images and a value estimate are all encoded as additional latent frames inside the same latent diffusion sequence. The video model's own learning objective does the work; nothing bolts an action head onto the side. That is a real departure from the action chunking head design used by GR00T N1.7 and Pi0.5.

Read the last three rows together and the honest picture appears. A forward pass of 0.61 seconds, or 0.95 seconds at the 10 denoising steps the ALOHA deployment actually used, cannot close a loop at 25 Hz. It works because the policy emits two seconds of motion and commits to all of it before looking again. That is inference latency being hidden by open-loop blindness, the same trade every chunked policy makes, just at a larger scale. And the planning gain of 12.5 points required eight H100s running in parallel for a single arm.

Cosmos 3, published in June 2026, folds this into a single omnimodal architecture where language, image, video, audio and action share one mixture-of-transformers backbone. The released policy checkpoint, Cosmos3-Nano-Policy-DROID, is a 16B model post-trained on DROID's 76k trajectories and 350 hours. It predicts 32 future absolute joint positions at 15 Hz, samples with 4 diffusion steps, and skips video-latent decoding entirely at inference to save time. Serving it takes two NVIDIA RTX Pro 6000 GPUs. It ranked first on RoboArena at the time of submission. It also weighs about 35 times what SmolVLA does.

What the video-prediction numbers actually mean

Papers in this area report PSNR, SSIM, FVD, DreamSim and IoU. It is worth knowing what a good score does and does not buy you, because a model can look convincing and still be useless for control.

That IoU number is the one to sit with. NVIDIA built a controlled benchmark in Isaac Sim with eight rigid-body scenarios, free fall, slopes, stacks, dominoes, seesaw, gyroscope, and rendered 800 clips at 1080p. The best configuration tracked objects at 0.598 average IoU over 33 frames. The paper's own conclusion is that larger models did not help and that all the world foundation models equally struggle with physics adherence. A model that produces beautiful, high-FVD video of a robot arm may still put the cube in the wrong place, and that is exactly the error a planner will happily optimise into.

The hardware bill, in real numbers

NVIDIA publishes generation times per GPU for Cosmos-Predict2, which makes this the least speculative part of the whole subject. These are seconds to produce one clip at 480p and 16 fps.

At 720p and 16 fps the picture gets worse. The same 2B Video2World model takes 228.8 seconds per clip on an H100 SXM and 378.5 seconds on an H100 PCIe; the 14B takes 856.9 and 1425.4 seconds on the same two cards. NATTEN sparse attention cuts that by 2.0x to 2.6x, but only at 720p and only on Hopper and Blackwell datacentre cards. If you want thousands of neural trajectories, do the multiplication first: DreamGen reports that generating its 240k-sample RoboCasa dataset took 54 hours on 1500 NVIDIA L40 GPUs.

Doing it yourself against doing it here

Where this touches an SO-100 today

Three concrete contact points exist right now, and they are smaller than the press releases suggest but they are real.

• GR00T N1.7 already carries a Cosmos component. Its VLM backbone is Cosmos-Reason2-2B, from the same model family as the world models above. You are not training a world model when you fine-tune GR00T N1.7 on an SO-100, but the reasoning half of its stack came out of that research programme.
• GR00T-Dreams ships an so100 embodiment. The inverse dynamics preprocessing and fine-tuning scripts list franka, gr1, so100 and robocasa. If you generate dreams for an SO-100, the pseudo-action extraction path is already written and it outputs LeRobot format.
• The arena tracks the world-model policies. Cosmos Policy, LAPA, Moto-GPT, villa-X, Seer, GR-2 and UniVLA all have entries with their benchmark numbers linked to source.

What does not exist is a path where you press a button here and get a world model. That is deliberate rather than an oversight: the smallest useful video world model needs more VRAM than the card that trains ACT and SmolVLA, the generation step alone would cost more than an entire training run, and the result would still need a normal policy trained on top of it. When that arithmetic changes, the honest thing is to say so then.

Honest limits, stated plainly

If you are deciding whether to spend the next month on this, here is the summary that the benchmark tables do not put on the front page.

• Genie 3 is the most impressive interactive world model published, 720p at 24 fps with several minutes of consistency, and it is a limited research preview with no weights. You cannot build on it.
• Every strong robot result required fine-tuning the world model on the target embodiment first. There is no published zero-shot path from a general video model to a specific arm.
• The planning results that beat policies used 8 GPUs for one robot. The results that used one GPU took minutes per action.
• Route 2 is the only one whose output plugs into an ordinary training pipeline, and its headline number, 43.2 percent on new behaviors, is a large relative gain over 11.2 percent rather than a working system.
• Adding a network round trip on top of any of this makes it worse. Inference has to sit next to the servos for fast tasks; the control loop here is 20 to 485 ms per action step depending on the model, and public-internet latency turns a working policy into a hesitant one. Remote inference is viable for slow pick and place, not for fast reactive motion.

The pragmatic sequence has not changed. Record 50 clean episodes, train a policy on them, find out what actually fails, and only then ask whether a world model addresses that specific failure. If the answer is environment generalization, route 2 has published evidence. If the answer is anything about speed or precision, it does not. Compare against what VLAs already do before assuming the newer idea is the better one.