IRASim

ByteDance, China 路 June 2024

ByteDance Research and Hong Kong University of Science and Technology

World model 路 Open weights, Apache-2.0 路 Also written Interactive Real-Robot Action Simulator

Parameters
not published
The paper scales the model from 33M to 679M parameters; the README does not state the size of the released checkpoints.
GPU memory
not published
weights at bf16, computed
Inference latency
not published
per action step
Weights
Apache-2.0
Apache-2.0 for the repository; the checkpoints are linked from the README.

What it is

IRASim is an action-conditioned video world model for manipulation from ByteDance Research and HKUST. A diffusion transformer generates the next 15 frames from one past frame and a trajectory of 15 actions, with a frame-level action conditioning module in every block so that each generated frame matches its action. Policy success rates measured inside IRASim tracked the ground-truth simulator with a Pearson correlation of 0.99 on a LIBERO task, and planning with it raised a Push-T diffusion policy from 0.637 to 0.961 IoU. Code, checkpoints for RT-1, Bridge and Language-Table and the training data are public under Apache-2.0.

Architecture

Backbone
Diffusion transformer over latents of the SDXL VAE, with frame-level action conditioning in each transformer block. Up to 288 by 512 pixels.
Action head
No action output; action trajectories condition the generated frames, and planning chooses among candidate policy trajectories.
Parameters
The paper scales the model from 33M to 679M parameters; the README does not state the size of the released checkpoints.
Pretraining data
Trained per dataset on RT-1, Bridge, Language-Table and RoboNet video with actions; the preprocessed data is also on Hugging Face (fangqi/IRASim).
Embodiments
Google robot (RT-1 data), WidowX (Bridge), Language-Table, Franka (LIBERO simulation), Push-T simulation

What hardware it needs

This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.

Not published.

The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.

Reported results

Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.

Simulation

Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.

  • LIBERO policy evaluation, 4 diffusion policy checkpoints on one task, 50 runs each
    0.99 correlation
    Pearson correlation with the ground-truth simulatorAuthors' own evaluation. Success rates in MuJoCo against IRASim: 0.18 and 0.28, 0.50 and 0.48, 0.80 and 0.74, 1.00 and 0.96.source
  • Push-T diffusion policy with IRASim planning, 50 candidate trajectories, 100 trials
    0.961 IoU
    IoUAuthors' own evaluation. The same diffusion policy without planning reaches 0.637, GPC-RANK+OPT 0.882.source

Real world

No results in this category are published for this model.

Fine tuned tasks

No results in this category are published for this model.

On real hardware

No real robot success rates are reported. Video quality on held-out RT-1 and Bridge data: PSNR 26.048 and 25.275, FVD 25.58 and 20.91 for the main model.

Fine tuning it yourself

Training scripts for single and multi-GPU runs are in the repository, with configs per dataset. A Language-Table demo lets you drive the virtual arm with a keyboard.

Where it helps, where it does not

Strengths

  • Policy evaluation inside the model matched the real simulator closely (Pearson 0.99 in the LIBERO test).
  • Test-time planning with the model lifted a Push-T policy from 0.637 to 0.961 IoU.
  • Fully open: code, checkpoints and data under Apache-2.0.

Limits

  • The correlation evidence rests on four checkpoints of one LIBERO task.
  • Trained per dataset, not as one general robot world model.
  • Short horizon per prediction: 15 future frames, longer rollouts are autoregressive.

Sources

Everything on this page was taken from these documents. Where they disagree with what you read here, they win.

Entry last checked 2026-11-12.

Try a policy on a real arm

A real SO-100 runs on /live and can be driven from the browser when it is online. Five of the models in this arena can be fine tuned on a dataset you record with your own arm.