RDT-1B
Tsinghua University, China · October 2024
Tsinghua University (thu-ml)
- Parameters
- 1.2 B
- The paper states 1.2B parameters for the diffusion transformer and calls it the largest diffusion-based manipulation...
- GPU memory
- 2.2 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- downloadable checkpoint
What it is
RDT-1B is a 1.2B parameter diffusion transformer for bimanual manipulation from Tsinghua University. It deliberately avoids a VLM backbone: a frozen SigLIP vision encoder and a frozen T5-XXL text encoder feed a diffusion policy that emits a chunk of 64 future actions from up to three camera views plus proprioception. Pre-training used 46 robot datasets with more than 1M episodes mapped into a shared, physically interpretable action space, followed by fine-tuning on over 6K self-collected Mobile ALOHA episodes. Everything is MIT licensed, which makes it one of the few genuinely permissive bimanual foundation models. Its declared weak point is cross-embodiment transfer, since the model card states the checkpoint does not generalize to unseen robot platforms without fine-tuning.
Architecture
- Backbone
- No VLM backbone. Frozen SigLIP-SO400M-patch14-384 vision encoder plus frozen T5-v1.1-XXL text encoder feeding a purpose-built diffusion transformer.
- Action head
- Diffusion transformer trained with a DDPM objective, sampled at inference with DPM-Solver++
- Action chunk
- 64 steps per forward pass
- Parameters
- The paper states 1.2B parameters for the diffusion transformer and calls it the largest diffusion-based manipulation foundation model at the time. The Hugging Face card labels the model 1B. The frozen SigLIP-SO400M vision encoder and the frozen T5-v1.1-XXL text encoder are additional and are not included in that count, which matters in practice because T5-XXL alone is the reason the repository warns about GPU memory.
- Pretraining data
- 46 robot datasets with more than 1M episodes, including RT-1, RH20T, DROID, BridgeData V2, RoboSet and an Open X-Embodiment subset, all mapped into a Physically Interpretable Unified Action Space. Fine-tuned afterwards on a self-collected multi-task bimanual dataset of more than 6K episodes recorded on a Mobile ALOHA dual-arm robot.
- Embodiments
- Mobile ALOHA (dual-arm, 14-DoF state in the reference config), single-arm and dual-arm manipulators, joint or end-effector control, wheeled locomotion (a robot dog was evaluated in the paper)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 4.5 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 2.2 GB | The usual way these checkpoints are served |
| int8 | 1.1 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.6 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
No explicit VRAM figure is published. The paper reports running inference on the robot's onboard RTX 4090 24GB GPU, which is the only hard hardware datapoint. For fine-tuning, the repository warns that on an RTX 4090 or lower the GPU memory may be too low to load the t5-v1_1-xxl encoder and recommends precomputing language embeddings offline instead of loading the encoder during training.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Real robot (Mobile ALOHA, RDT paper protocol) Pour Water-L-1/3, instruction following (correct amount poured)100%success rateAuthors' own evaluation. OpenVLA reaches 50 percent, Octo 0 percent in the same row.source
- Real robot (Mobile ALOHA, RDT paper protocol) Wash Cup, unseen object generalization87.5%success rateAuthors' own evaluation. Best baseline in the same row is ACT at 37.5 percent; OpenVLA and Octo score 0 percent.source
- Real robot (aggregate across the paper's task suite) average over all evaluated tasks versus best baseline56%relative improvement in success rateHeadline claim from the authors, stated as a 56 percent improvement in success rates over state-of-the-art baselines. It is an aggregate over a heterogeneous task set, not a single-benchmark number.source
- Real robot (quadruped, RDT paper protocol) Robot dog, walk straight48%success rateAuthors' own evaluation. Best baseline is Octo at 4 percent. Absolute performance below 50 percent shows the limit of the approach on locomotion.source
Fine tuned tasks
What the model reaches after being adapted to a new task or a new robot, usually from a small number of demonstrations. This is the number that matters if you bring your own data.
- Real robot (Mobile ALOHA, RDT paper protocol) Handover, 5-shot few-shot learning100%success rateAuthors' own evaluation. ACT reaches 88 percent on the same task, so this row is the least discriminative of the set.source
On real hardware
Evaluated on a real Mobile ALOHA dual-arm platform and on a quadruped. Reported real-robot numbers include 87.5 percent on Wash Cup with an unseen cup, 100 percent on a 5-shot Handover task, and 48 percent on a robot dog walk-straight task. Inference ran onboard on an RTX 4090.
Fine tuning it yourself
Fine-tuning converts episodes into the paper's unified 128-dimensional action vector, with a per-embodiment mask selecting the active dimensions. The repository documents full fine-tuning and strongly recommends precomputing T5-XXL language embeddings so the encoder never has to sit in GPU memory during training. Pre-training itself consumed 48 H100 80GB GPUs for about a month across 1M iterations, so reproduction of pre-training is out of reach for almost everyone and only fine-tuning is a realistic path.
Where it helps, where it does not
Strengths
- MIT license on code, weights and data, which is unusually permissive for a bimanual foundation model
- Predicts a 64-step action chunk and runs onboard on a single RTX 4090, so it does not require a server-class GPU at deployment
- Explicitly designed for heterogeneous robots through a unified action space, so the same checkpoint format covers single-arm, dual-arm, joint and end-effector control
- Reported few-shot adaptation from 1 to 5 demonstrations on new skills
Limits
- The model card states it cannot generalize to unseen robot platforms because of the embodiment gap and requires fine-tuning on target-robot data
- Depends on a frozen T5-v1.1-XXL text encoder, which is the practical VRAM bottleneck during training and forces an offline embedding precomputation step
- No published VRAM or millisecond-latency figures, only a 6 Hz chunk rate on an RTX 4090
- Real-robot evaluations use 8 trials per task in most rows, so individual percentages carry wide confidence intervals
- Pre-training required 48 H100 80GB GPUs for roughly one month and is not reproducible on a normal budget
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2410.07864
- https://arxiv.org/html/2410.07864v2
- https://huggingface.co/robotics-diffusion-transformer/rdt-1b
- https://github.com/thu-ml/RoboticsDiffusionTransformer
- https://rdt-robotics.github.io/rdt-robotics/
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.