UCB

UniSim

UC Berkeley, United States 路 October 2023

UC Berkeley, Google DeepMind and MIT

World model 路 Closed weights 路 Also written Universal Simulator, Learning Interactive Real-World Simulators

Parameters
5.6 B
5.6B parameters; training took 512 TPU-v3 chips for 20 days.
GPU memory
10 GB
weights at bf16, computed. Fits a 16 GB card
Inference latency
not published
per action step
Weights
closed
no public checkpoint

What it is

UniSim is an early learned simulator of real-world interaction from UC Berkeley, Google DeepMind and MIT, and an ICLR 2024 Outstanding Paper. A 5.6B video diffusion model, trained on a mix of robot data, human activity video, navigation scans, simulation and internet images and video, renders the visual outcome of high-level instructions such as opening a drawer or of low-level controls. Policies trained purely inside it, a vision language planner and a reinforcement learning policy for Language Table, were deployed on the real robot zero-shot, although the real-robot results are shown only qualitatively. No code or weights are public.

Architecture

Backbone
Video U-Net diffusion model conditioned on past frames and on actions, either language or low-level controls, through classifier-free guidance.
Action head
No action output; actions condition the generated video.
Parameters
5.6B parameters; training took 512 TPU-v3 chips for 20 days. Smaller 500M and 1.6B versions appear in the scaling study.
Pretraining data
A curated mix: Habitat HM3D and Language Table simulation, real robot data (Bridge, RT-1, Language Table), human activity video (Ego4D, Something-Something V2, EPIC-KITCHENS), Matterport panorama scans, LAION-400M and ALIGN image and text pairs, and internet video.
Embodiments
Language Table robot (zero-shot deployment of policies trained in UniSim)

What hardware it needs

This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.

Computed from 5.6 B parameters. Weights only, so treat it as a floor: activations, image encoder intermediates and the CUDA context come on top, in practice 30 to 50 percent more for inference.
PrecisionWeightsNote
fp3221 GBTraining master weights, rarely used for inference
bf16 or fp1610 GBThe usual way these checkpoints are served
int85.2 GBQuantised, expect some loss of precision on fine motions
int42.6 GBAggressive quantisation, verify success rate before trusting it

Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 16 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.

Not published. The model was trained on 512 TPU-v3 chips.

The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.

Reported results

Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.

Simulation

No results in this category are published for this model.

Real world

No results in this category are published for this model.

Fine tuned tasks

No results in this category are published for this model.

On real hardware

Zero-shot deployment on the real robot of a planner and an RL policy trained only in UniSim, shown qualitatively. In simulated evaluation over 48 Language Table tasks the RL policy trained in UniSim reached 0.81 success against 0.58 for behaviour cloning, and the planner trained with hindsight relabelling in UniSim reduced the distance to goal 3 to 4 times more than the baseline.

Fine tuning it yourself

Not possible; no code or weights are public.

Where it helps, where it does not

Strengths

  • Early evidence that a learned video simulator can train policies that transfer to a real robot without real interaction.
  • Combines robot, human, navigation and internet data so that each covers a different aspect of interaction.
  • Clear scaling evidence: FVD improves from 277.85 at 500M to 211.30 at 5.6B parameters.

Limits

  • Closed: no weights or code.
  • Real robot results are qualitative only.
  • From 2023; newer open world models such as Cosmos-Predict2.5 and Genie Envisioner exist.

Sources

Everything on this page was taken from these documents. Where they disagree with what you read here, they win.

Entry last checked 2026-11-12.

Try a policy on a real arm

A real SO-100 runs on /live and can be driven from the browser when it is online. Five of the models in this arena can be fine tuned on a dataset you record with your own arm.