RDT2-FM
Tsinghua University, China · September 2025
Tsinghua University (thu-ml)
- Parameters
- not published
- No total parameter count is published for RDT2-FM.
- GPU memory
- 16 GB
- reported by the authors
- Inference latency
- not published
- per action step
- Weights
- Apache-2.0
- downloadable checkpoint
What it is
RDT2-FM is the continuous-action sibling of RDT2-VQ. It reuses the same Qwen2.5-VL-7B-Instruct vision-language backbone and replaces the residual-VQ token head with a flow-matching action expert built on an improved RDT architecture, emitting the same (24, 20) relative action chunk. The practical argument for it over RDT2-VQ is cost: the model card puts fine-tuning of the FM head alone at roughly 16GB VRAM, against 32GB for LoRA and 80GB for a full fine-tune on the VQ variant. Both variants are Apache-2.0 and both depend on the UMI action convention, so calibration and wrist-camera geometry, not the model, are usually what decide success on a new arm.
Architecture
- Backbone
- RDT2-VQ vision-language backbone (Qwen2.5-VL-7B-Instruct)
- Action head
- Flow-matching action expert combining an improved RDT architecture with a flow-matching objective, producing continuous actions without iterative denoising
- Action chunk
- 24 steps per forward pass
- Parameters
- No total parameter count is published for RDT2-FM. It builds on the RDT2-VQ vision-language backbone (Qwen2.5-VL-7B-Instruct, 8B total per the RDT2-VQ card) and adds a flow-matching action expert whose size is not disclosed, so no honest total can be given.
- Pretraining data
- Same corpus as RDT2-VQ: more than 10,000 hours of UMI-gripper manipulation demonstrations across more than 100 indoor scenes.
- Embodiments
- Bimanual UR5e, Franka Research 3 (FR3)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Inference memory, reported | 16 GB |
| Fine tuning memory, reported | 16 GB |
RTX 4090 class card is sufficient for both inference and for fine-tuning the flow-matching head alone, which is a substantially lower bar than the 80GB needed for a full RDT2-VQ fine-tune.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
Same platform coverage as RDT2-VQ, namely zero-shot bimanual UR5e and Franka FR3 after calibration. No separate numeric real-robot table is published for the FM variant.
Fine tuning it yourself
The cheap path is to freeze the vision-language backbone and fine-tune only the flow-matching head, which the model card puts at roughly 16GB VRAM. Output is a float32 tensor of shape (T=24, D=20) of relative action deltas, 10 dimensions per arm (3 position, 6D rotation, 1 gripper width). Action statistics and normalization must match the released ones or behaviour degrades.
Where it helps, where it does not
Strengths
- Fine-tuning only the flow-matching head fits in about 16GB VRAM, the lowest published fine-tuning bar of any model in this comparison
- Apache-2.0 weights
- Flow matching avoids iterative denoising, which the authors give as the reason for low-latency control
- Shares the zero-shot cross-embodiment behaviour of RDT2-VQ
Limits
- No published total parameter count
- No numeric benchmark table of its own; the RDT2 paper reports results as bar figures
- The 'low-latency' claim is qualitative, with no millisecond number and no named GPU
- Same UMI-convention dependency as RDT2-VQ: calibration quality and camera placement decide whether it transfers
- The model card notes that dataset and action-statistics shifts degrade behaviour
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://huggingface.co/robotics-diffusion-transformer/RDT2-FM
- https://github.com/thu-ml/RDT2
- https://arxiv.org/abs/2602.03310
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.