IRASim
ByteDance, China 路 June 2024
ByteDance Research and Hong Kong University of Science and Technology
World model 路 Open weights, Apache-2.0 路 Also written Interactive Real-Robot Action Simulator
- Parameters
- not published
- The paper scales the model from 33M to 679M parameters; the README does not state the size of the released checkpoints.
- GPU memory
- not published
- weights at bf16, computed
- Inference latency
- not published
- per action step
- Weights
- Apache-2.0
- Apache-2.0 for the repository; the checkpoints are linked from the README.
What it is
IRASim is an action-conditioned video world model for manipulation from ByteDance Research and HKUST. A diffusion transformer generates the next 15 frames from one past frame and a trajectory of 15 actions, with a frame-level action conditioning module in every block so that each generated frame matches its action. Policy success rates measured inside IRASim tracked the ground-truth simulator with a Pearson correlation of 0.99 on a LIBERO task, and planning with it raised a Push-T diffusion policy from 0.637 to 0.961 IoU. Code, checkpoints for RT-1, Bridge and Language-Table and the training data are public under Apache-2.0.
Architecture
- Backbone
- Diffusion transformer over latents of the SDXL VAE, with frame-level action conditioning in each transformer block. Up to 288 by 512 pixels.
- Action head
- No action output; action trajectories condition the generated frames, and planning chooses among candidate policy trajectories.
- Parameters
- The paper scales the model from 33M to 679M parameters; the README does not state the size of the released checkpoints.
- Pretraining data
- Trained per dataset on RT-1, Bridge, Language-Table and RoboNet video with actions; the preprocessed data is also on Hugging Face (fangqi/IRASim).
- Embodiments
- Google robot (RT-1 data), WidowX (Bridge), Language-Table, Franka (LIBERO simulation), Push-T simulation
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
Not published.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- LIBERO policy evaluation, 4 diffusion policy checkpoints on one task, 50 runs each0.99 correlationPearson correlation with the ground-truth simulatorAuthors' own evaluation. Success rates in MuJoCo against IRASim: 0.18 and 0.28, 0.50 and 0.48, 0.80 and 0.74, 1.00 and 0.96.source
- Push-T diffusion policy with IRASim planning, 50 candidate trajectories, 100 trials0.961 IoUIoUAuthors' own evaluation. The same diffusion policy without planning reaches 0.637, GPC-RANK+OPT 0.882.source
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
No real robot success rates are reported. Video quality on held-out RT-1 and Bridge data: PSNR 26.048 and 25.275, FVD 25.58 and 20.91 for the main model.
Fine tuning it yourself
Training scripts for single and multi-GPU runs are in the repository, with configs per dataset. A Language-Table demo lets you drive the virtual arm with a keyboard.
Where it helps, where it does not
Strengths
- Policy evaluation inside the model matched the real simulator closely (Pearson 0.99 in the LIBERO test).
- Test-time planning with the model lifted a Push-T policy from 0.637 to 0.961 IoU.
- Fully open: code, checkpoints and data under Apache-2.0.
Limits
- The correlation evidence rests on four checkpoints of one LIBERO task.
- Trained per dataset, not as one general robot world model.
- Short horizon per prediction: 15 future frames, longer rollouts are autoregressive.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2406.14540
- https://arxiv.org/html/2406.14540v2
- https://github.com/bytedance/IRASim
- https://gen-irasim.github.io/
- https://github.com/bytedance/IRASim#dataset
Entry last checked 2026-11-12.
Try a policy on a real arm
A real SO-100 runs on /live and can be driven from the browser when it is online. Five of the models in this arena can be fine tuned on a dataset you record with your own arm.