MoDE
Karlsruhe Institute of Technology, Germany, United States 路 December 2024
Karlsruhe Institute of Technology (Intuitive Robots Lab) and MIT CSAIL
Robot foundation model 路 Open weights, MIT 路 Also written Mixture of Denoising Experts, MoDE Diffusion Policy, MoDE_Pretrained
- Parameters
- not published
- The paper's figures disagree: Table 3 lists 685M parameters for the pretrained model, while Table 8 lists 740M total and 436M...
- GPU memory
- not published
- weights at bf16, computed
- Inference latency
- 12.2 ms
- an NVIDIA A6000, average per action
- Weights
- MIT
- MIT for the repository and the model card.
What it is
MoDE is a diffusion transformer policy from KIT and MIT that replaces dense feed-forward layers with a mixture of denoising experts routed by the noise level, so each denoising step uses only part of the network and the chosen experts can be cached. Pretrained on 196K trajectories from six Open X-Embodiment sources, it reaches an average length of 4.01 on CALVIN ABC to D and 95 percent on LIBERO-90, and it averages 26.3 percent on SimplerEnv, ahead of OpenVLA (23.7) and Octo (17.75) in the authors' table. Language enters through a frozen CLIP text encoder and images through FiLM-conditioned ResNets, with no vision language model. Inference took 12.2 ms per action on an NVIDIA A6000 in the paper.
Architecture
- Backbone
- No VLM backbone. Frozen CLIP ViT-B/32 text encoder and FiLM-conditioned ResNet-50 image encoders (ResNet-18 in some benchmark models) feeding a mixture-of-experts transformer with 12 layers, 4 experts and width 1024 in the pretrained model.
- Action head
- Diffusion transformer with noise-conditioned expert routing and noise-conditioned self-attention; DDIM sampling with 5 steps according to the model card, action window of 10, 7-dimensional delta end-effector actions.
- Action chunk
- 10 steps per forward pass
- Parameters
- The paper's figures disagree: Table 3 lists 685M parameters for the pretrained model, while Table 8 lists 740M total and 436M active. The benchmark models without pretraining are listed at 460M.
- Pretraining data
- 196K trajectories from six sources: BC-Z, LIBERO-10, Bridge, CMU Play-Fusion, Google Fractal and DOBB-E.
- Embodiments
- Franka (CALVIN and LIBERO simulation), Google robot and WidowX (SimplerEnv)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Inference latency, reported | 12.2 ms on an NVIDIA A6000, average per action source |
No memory figure. Pretraining took 6 NVIDIA A6000 GPUs for 3 days (300K steps).
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- LIBERO LIBERO-90, pretrained model95%success rateAuthors' own evaluation, reported as 0.95 in the abstract. Without pretraining 91.0 percent.source
- SimplerEnv average over the authors' SIMPLER task set26.3%
- CALVIN ABCD to D, pretrained model4.39 tasks (out of 5)average length (tasks completed in sequence)Authors' own evaluation, plus or minus 0.04. Without pretraining 4.30.source
- CALVIN ABC to D, pretrained model4.01 tasks (out of 5)average length (tasks completed in sequence)Authors' own evaluation, plus or minus 0.04. Without pretraining 3.39.source
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
No real robot experiments are reported.
Fine tuning it yourself
Start from mbreuss/MoDE_Pretrained, pretrained for 300k steps on the Open X-Embodiment subset, and fine-tune on your task; the model takes a static and a gripper RGB camera plus a language instruction. Fine-tuned CALVIN and LIBERO checkpoints are published by the same author.
Where it helps, where it does not
Strengths
- Efficient: 12.2 ms per action, and expert caching cuts inference cost by about 90 percent, according to the authors.
- Strong simulation results for its size: 4.01 on CALVIN ABC to D and 95 percent on LIBERO-90.
- Open weights under MIT, with pretrained and fine-tuned checkpoints.
Limits
- Simulation only; no real robot evaluation.
- Parameter counts are inconsistent across the paper's tables.
- Small pretraining mixture: 196K trajectories against 800k for Octo.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2412.12953
- https://arxiv.org/html/2412.12953v1
- https://github.com/intuitive-robots/MoDE_Diffusion_Policy
- https://huggingface.co/mbreuss/MoDE_Pretrained
Entry last checked 2026-11-12.
Try a policy on a real arm
A real SO-100 runs on /live and can be driven from the browser when it is online. Five of the models in this arena can be fine tuned on a dataset you record with your own arm.