RDT-1B: the 1.2B Diffusion Foundation Model for Bimanual Robots

What RDT-1B actually is

RDT-1B stands for Robotics Diffusion Transformer. It is a vision-language-action model from the TSAIL group at Tsinghua University, released in October 2024 and presented as a poster at ICLR 2025. The goal is narrower than the phrase foundation model suggests. The paper puts it directly: multi-robot data is there to improve generalisation in bimanual manipulation, not to build a cross-embodiment model. Multi-robot pre-training is the means, a dual-arm policy is the end.

The design choice worth reading about is not the parameter count. It is how the model handles the fact that a Franka arm, an ALOHA rig and a wheeled manipulator all describe motion differently. RDT does not learn a head per robot, and it does not discretise actions into text tokens the way RT-2 does. It writes every robot into one fixed 128-slot vector where each slot has a physical meaning, then trains a diffusion model on that vector.

RDT-170M is the paper's ablation model, released two weeks later as a lower-VRAM option. Expect one discrepancy across sources: the paper calls it RDT (small) at 166 M parameters, the model card calls it RDT-170M. Same model. Both sizes have entries in the model arena: RDT-1B and RDT-170M.

The architecture: three modifications to a Diffusion Transformer

The backbone is a DiT with cross-attention, the same family as the image diffusion transformers, adapted for a time series of joint commands rather than a pixel grid. The paper lists exactly three key modifications to the standard block, each with a stated reason and an ablation behind it.

• QKNorm and RMSNorm together. Robot physical quantities arrive in an unstable numerical range, so query-key normalisation goes into every attention layer. LayerNorm becomes RMSNorm because its centering step causes token shift and attention shift, which the authors argue destroys the symmetry of a time series. Figure 4 shows the loss curve going unstable without these.
• MLP decoder instead of a linear head. The projection from latent space back to physical action space is a nonlinear MLP, because robot dynamics are nonlinear and a linear readout loses the dexterity that depends on them.
• Alternating condition injection. Image tokens vastly outnumber language tokens, so injecting both into one cross-attention layer lets the images overshadow the instruction. The blocks alternate instead: one layer attends to language, the next to images.

# configs/base.yaml (excerpt, thu-ml/RoboticsDiffusionTransformer)
common:
  img_history_size: 2
  action_chunk_size: 64
  num_cameras: 3
  state_dim: 128

model:
  lang_adaptor: mlp2x_gelu
  img_adaptor: mlp2x_gelu
  state_adaptor: mlp3x_gelu
  lang_token_dim: 4096      # T5-XXL
  img_token_dim: 1152       # SigLIP so400m
  state_token_dim: 128
  rdt:
    # 1B: num_head 32 hidden_size 2048
    hidden_size: 2048
    depth: 28
    num_heads: 32
    cond_pos_embed_type: multimodal
  noise_scheduler:
    type: ddpm
    num_train_timesteps: 1000
    num_inference_timesteps: 5
    beta_schedule: squaredcos_cap_v2  # Critical choice
    prediction_type: sample
    clip_sample: False

The low-dimensional inputs, proprioception, the noisy action chunk, the control frequency and the diffusion timestep, are each encoded by MLPs with Fourier features and concatenated along the sequence axis, giving a token sequence of length 1 + Ta + 1 + 1 that the paper calls in-context conditioning. Language and images are cross-attention conditions, outside that sequence.

Training uses a DDPM scheduler with 1000 timesteps and a squared-cosine beta schedule. At sampling time the paper reports DPM-Solver++ cutting the denoising loop from 100 steps to 5, and the shipped config sets num_inference_timesteps to 5 to match. On the robot's onboard RTX 4090 24 GB that gave 6 Hz of action chunks and 381 Hz of individual actions, since each chunk holds 64. During training each input modality is independently masked with probability 0.1, which is what buys tolerance for a missing camera.

The 128-dimensional unified action space

This is the part worth stealing even if you never run the model. Instead of normalising every dataset into an anonymous [-1, 1] box, RDT defines one 128-slot vector where slot 30 always means right end effector x in metres and slot 10 always means right gripper opening. A robot fills the slots it has and pads the rest. The mapping is in configs/state_vec.py and in Table 4 of the paper.

The consequences are practical. An arm with fewer than ten joints fills only the leading slots of its block: a six degree of freedom arm fills the first six. A single-arm follower fills the right arm block, never the left, because that is how the single-arm pre-training data was mapped. Units are SI units, which the README notes keeps most values inside [-1, 1] anyway. The one quantity that gets normalised is gripper width, min-max scaled to [0, 1], because gripper travel is arbitrary across hardware.

# data/hdf5_vla_dataset.py, the part you actually write (around L180-194).
# See configs/state_vec.py for the full name -> index mapping.
from configs.state_vec import STATE_VEC_IDX_MAPPING
import numpy as np

def fill_in_state(values):
    # values: (T, n_joints) joint angles in radians, gripper last
    n_arm = values.shape[1] - 1
    uni_vec = np.zeros((values.shape[0], 128), dtype=np.float32)

    idx = [STATE_VEC_IDX_MAPPING[f"arm_joint_{i}_pos"] for i in range(n_arm)]
    uni_vec[:, idx] = values[:, :n_arm]

    # gripper width, min-max normalised to [0, 1] - the one exception
    uni_vec[:, STATE_VEC_IDX_MAPPING["gripper_open"]] = values[:, -1]
    return uni_vec

If your action space carries end effector rotation, the repo requires the 6D representation from Zhou et al. rather than Euler angles or quaternions, and ships docs/test_6drot.py to convert. The README notes the mapping is not reversible: different Euler angles can be equivalent and land on the same 6D vector.

What went into pre-training, and why you will not repeat it

The pre-training collection is 46 datasets, more than 1 million trajectories and 21 TB on disk. Each dataset starts at a sampling weight of the square root of its size, so the RT-1 dataset with its 130K trajectories does not swamp a 1K-trajectory bimanual set, and the authors then raised the weights of datasets whose intermediate loss converged slowly. Familiar names include Open X-Embodiment, DROID, BridgeData V2, RH20T and RoboSet.

One appendix detail is worth flagging because papers usually bury it: the reported fine-tune did not start from the 1M-step pre-trained checkpoint. Scheduling forced the authors to branch from the 500K-step one. The published RDT-1B weights are nonetheless the 1M-step checkpoint, while RDT-170M is a 500K-step checkpoint.

Their fine-tuning dataset is instructive: 300+ tasks, 100+ rigid and non-rigid objects, 15+ rooms with different lighting, and GPT-4-Turbo generating 100 expanded instructions plus one simplified one per task, so the language side sees many phrasings of the same job. That last trick is nearly free and worth copying when you build your own LeRobot dataset, as argued in how to collect high-quality VLA training data.

What it takes to run RDT-1B

The checkpoint is small by 2026 standards. The problem is the frozen encoder riding along with it: T5-XXL is by a wide margin the largest component in the pipeline, and the README says outright that on an RTX 4090 or lower it may not fit. That is the most common reason a first RDT run dies.

Fine-tuning RDT-1B on your own data

This is the real path. Nobody outside a large lab pre-trains this model. What follows is the upstream fine-tuning flow as documented in the repo README at the state of the main branch on 23 August 2026.

1. 1
   Install the environment

   Python 3.10, CUDA 12.1 wheels, flash-attn last because it compiles against the installed torch.

   bash

   git clone git@github.com:thu-ml/RoboticsDiffusionTransformer.git
   cd RoboticsDiffusionTransformer
   
   conda create -n rdt python=3.10.0
   conda activate rdt
   
   pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121
   pip install packaging==24.0
   pip install flash-attn --no-build-isolation
   pip install -r requirements.txt

2. 2
   Link the two frozen encoders

   The repo expects them under a google/ directory at the root. Symlinks keep the weights out of your git tree.

   bash

   mkdir -p google
   ln -s /path/to/t5-v1_1-xxl google/t5-v1_1-xxl
   ln -s /path/to/siglip-so400m-patch14-384 google/siglip-so400m-patch14-384

3. 3
   Register your dataset

   Three JSON files carry the registration, with an agilex placeholder in all three. Replace it with your dataset name and give the control frequency in Hz. The sampling weight is irrelevant while you have one dataset.

   bash

   cd data && mkdir -p datasets
   ln -s /path/to/my_cool_dataset datasets/my_cool_dataset
   
   # then edit, replacing the agilex placeholder:
   #   configs/dataset_control_freq.json    "agilex": 25  -> {"my_cool_dataset": 30}
   #   configs/finetune_datasets.json       ["agilex"]    -> ["my_cool_dataset"]
   #   configs/finetune_sample_weights.json {"agilex":100}-> {"my_cool_dataset": 100}

4. 4
   Write the loader

   Edit HDF5_DIR and DATASET_NAME at lines 21 and 22 of data/hdf5_vla_dataset.py, then implement parse_hdf5_file() and parse_hdf5_file_state_only(). Despite the name you need not use HDF5; only the class contract matters. Lines 180 to 194 hold the 128-slot mapping.

   bash

   $EDITOR data/hdf5_vla_dataset.py     # L21 HDF5_DIR, L22 DATASET_NAME
                                        # L148 path to precomputed lang embeddings
                                        # L180-194 fill the unified action vector

5. 5
   Compute dataset statistics

   One command from the repo root. It writes the per-dimension statistics the trainer needs.

   bash

   python -m data.compute_dataset_stat_hdf5

6. 6
   Precompute language embeddings if you are under 24 GB

   Set TASK_NAME and INSTRUCTION in scripts/encode_lang.py, and point OFFLOAD_DIR at a real directory if T5-XXL still does not fit. Then add --precomp_lang_embed to the training script.

   bash

   # scripts/encode_lang.py
   #   MODEL_PATH  = "google/t5-v1_1-xxl"
   #   CONFIG_PATH = "configs/base.yaml"
   #   SAVE_DIR    = "outs/"
   #   OFFLOAD_DIR = None      # set an existing path if VRAM < 24GB
   
   python -m scripts.encode_lang
   # batch version for a whole dataset: scripts/encode_lang_batch.py

7. 7
   Launch the fine-tune

   finetune.sh is the shipped example and the flags below are its values on main. Point pretrained_model_name_or_path at the Hugging Face id and it pulls the 1M-step checkpoint.

   bash

   deepspeed --hostfile=hostfile.txt main.py \
       --deepspeed="./configs/zero2.json" \
       --pretrained_model_name_or_path="robotics-diffusion-transformer/rdt-1b" \
       --pretrained_text_encoder_name_or_path="google/t5-v1_1-xxl" \
       --pretrained_vision_encoder_name_or_path="google/siglip-so400m-patch14-384" \
       --output_dir="./checkpoints/rdt-finetune-1b" \
       --train_batch_size=32 \
       --sample_batch_size=64 \
       --max_train_steps=200000 \
       --checkpointing_period=1000 \
       --sample_period=500 \
       --checkpoints_total_limit=40 \
       --lr_scheduler="constant" \
       --learning_rate=1e-4 \
       --mixed_precision="bf16" \
       --dataloader_num_workers=8 \
       --image_aug \
       --dataset_type="finetune" \
       --state_noise_snr=40 \
       --load_from_hdf5 \
       --report_to=wandb

Monitor two numbers: a long-window moving average of loss and overall_avg_sample_mse. The README reports that empirically the lower this sample MSE the better the model performs, and that fine-tuning is over when it converges. The paper's appendix carries the reasoning, a positive correlation between that MSE and deployment performance on the robot, with the caveat that an unusually low value can mean overfitting. The repo FAQ asks for at least 150,000 steps regardless of batch size; the example script sets max_train_steps to 200,000.

Deploying the fine-tuned checkpoint

Inference is wrapped in a class called RoboticDiffusionTransformerModel in scripts/agilex_model.py, and you call its step() method. Two functions must be rewritten for your hardware: _format_joint_to_state() at line 164, which packs joint readings into the 128-slot vector, and _unformat_action_to_joint() at line 196, which unpacks the output into servo commands. The control frequency defaults in the same file and is passed on the command line.

python -m scripts.agilex_inference \
    --use_actions_interpolation \
    --pretrained_model_name_or_path=checkpoints/rdt-finetune-1b/checkpoint-<STEP> \
    --lang_embeddings_path=outs/lang_embeddings/your_instr.pt \
    --ctrl_freq=25

Two ways to get a policy onto an arm

# the three things that must be true before you launch
python -c "import torch, flash_attn; print(torch.__version__)"
ls google/t5-v1_1-xxl google/siglip-so400m-patch14-384
python -m data.compute_dataset_stat_hdf5

Where RDT-1B sits against the policies you can actually train here

The comparison below mixes two kinds of number. The five AY-Robots rows are platform figures for the per-action-step latency of a served policy; the RDT row is the paper's own throughput measurement on the robot's onboard RTX 4090. Different metrics, not a ranking.

Two structural similarities are worth naming. RDT's chunk of 64 actions is the same idea as action chunking in ACT, unsurprising since the paper devotes an appendix to the technique and cites ACT for it. And RDT's diffusion head addresses the same multi-modality problem that flow matching addresses in Pi0.5, by a different route, compared in our Pi-Zero flow matching article.

What the reported numbers say

The 56 percent improvement in success rate that gets quoted everywhere comes from the paper's introduction, not its abstract, and is measured on a Cobot Mobile ALOHA with two 7 degree of freedom arms against ACT, OpenVLA and Octo. The ManiSkill simulation results, added to the repo in December 2024, are easier to read because the protocol is fixed: 250 trials per task, 10 seeds with 25 trials each, RDT fine-tuned from the released checkpoint for 300K iterations.

Read the PlugCharger column before the mean. 1.2 percent is a failure, and PegInsertionSide at 13.2 percent is not a working policy either: contact-rich insertion is unsolved for all four methods, and the mean is carried by the easy tasks. The ablation is more useful to practitioners: removing pre-training dropped unseen-object success from 50 percent to 0, while the 166 M small variant dropped it to 37.5 percent. Pre-training scale mattered more than model size.

The honest limits

Three things about RDT-1B are routinely overstated when the paper gets cited, and one thing about running any policy over a network applies here too.

• It is not a zero-shot cross-embodiment model. The model card says it plainly: because of the embodiment gap, RDT cannot yet generalise to robot platforms absent from the pre-training data, and the recommendation is to collect a small target-robot dataset and fine-tune. The unified action space makes multi-robot training possible; it does not make a new arm free.
• The bimanual results are one robot. All real-robot evaluation is on a Cobot Mobile ALOHA, and the appendix states the mobile base was used only to move between rooms, never during training or inference. These are static bimanual tasks.
• The pre-training scale is the contribution. The ablation shows the from-scratch variant collapsing to 0 percent on unseen objects. If you fine-tune on a handful of episodes and it works, that is the 21 TB doing the work.
• Latency is local or it is nothing. The paper's 6 Hz chunk rate is measured on a GPU bolted to the robot. A public-internet round trip between the cameras and the policy adds tens to hundreds of milliseconds per decision, turning a working policy into a hesitant one. Remote inference suits slow pick-and-place, not fast reactive motion. That is true of our inference pods too, and we would rather say it than sell around it.

One practical limit has nothing to do with the model: RDT does not read LeRobot datasets natively. Checked on 23 August 2026, the lerobot policies directory holds act, diffusion, pi0, pi05, pi0_fast, smolvla, groot, xvla and about a dozen more, and no rdt. If your episodes came from the LeRobot toolchain, you write that conversion yourself. Recording is where the platform helps: see record your first dataset and teleoperation.

RDT2, and what it means for RDT-1B

The same group released RDT2 in September 2025, with the paper following in February 2026. It is a different bet: instead of scaling robot teleoperation data, RDT2 scales UMI-style handheld gripper data, more than 10,000 hours of human manipulation video in over 100 indoor scenes, collected with redesigned higher-strength UMI hardware. Two variants shipped. RDT2-VQ is a VLA adapted from Qwen2.5-VL-7B-Instruct that uses residual vector quantisation as its action tokenizer; RDT2-FM is an improved RDT used as a flow-matching action expert, with much lower inference latency.

For a broader map of how these families relate, our overview of vision-language-action models covers the lineage, and the policies page covers what is trainable here today. If you are choosing hardware rather than a model, start at the SO-100 page or the SO-100 complete guide.