RDT2-VQ
Tsinghua University, China · September 2025
Tsinghua University (thu-ml)
- Parameters
- 8.3 B
- The Hugging Face card lists 8B parameters.
- GPU memory
- 16 GB
- reported by the authors
- Inference latency
- not published
- per action step
- Weights
- Apache-2.0
- downloadable checkpoint
What it is
RDT2-VQ is the Tsinghua group's successor to RDT-1B and a complete architectural break from it. Instead of a bespoke diffusion transformer it fine-tunes Qwen2.5-VL-7B-Instruct to autoregressively emit discrete action tokens produced by residual vector quantization, predicting a 24-step relative action chunk at 30 fps (0.8 seconds) in a 20-dimensional bimanual end-effector space. The training data is not robot teleoperation but more than 10,000 hours of UMI-gripper human demonstrations from 100-plus indoor scenes, which is what enables the headline claim of zero-shot deployment on unseen arms such as bimanual UR5e and Franka FR3. It is Apache-2.0 and, rarely for this class, publishes concrete VRAM tiers (16GB inference, 32GB LoRA, 80GB full fine-tune). The catch is that everything is expressed in the UMI convention, so camera placement, calibration and gripper geometry decide whether the zero-shot claim holds on your hardware.
Architecture
- Backbone
- Qwen2.5-VL-7B-Instruct
- Action head
- Autoregressive prediction of discrete action tokens produced by Residual Vector Quantization (RVQ)
- Action chunk
- 24 steps per forward pass
- Parameters
- The Hugging Face card lists 8B parameters. The paper describes the system as built on a 7B parameter VLM, so the 8B figure includes the action-token machinery on top of the Qwen2.5-VL-7B-Instruct backbone. Tensor type is BF16.
- Pretraining data
- More than 10,000 hours of human manipulation demonstrations collected with an enhanced, embodiment-agnostic Universal Manipulation Interface (UMI) gripper across more than 100 indoor scenes and households. The paper describes this as one of the largest open-source robotic datasets.
- Embodiments
- Bimanual UR5e, Franka Research 3 (FR3), UMI gripper data collection rig
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 31 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 15 GB | The usual way these checkpoints are served |
| int8 | 7.7 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 3.9 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 24 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Inference memory, reported | 16 GB |
| Fine tuning memory, reported | 32 GB |
RTX 4090 class card (16GB VRAM and 32GB system RAM) is the stated inference floor. LoRA fine-tuning needs an A100 40GB class card, full fine-tuning an A100 80GB, H100 or B200. Tested on Ubuntu 24.04.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
Demonstrated zero-shot on Bimanual UR5e and Franka Research 3 setups without embodiment-specific fine-tuning, on unseen objects, scenes and instructions. The paper also reports dexterous, long-horizon and dynamic downstream tasks including table tennis. Numeric per-task success rates appear only as bar figures in the paper, so this research could not verify exact values.
Fine tuning it yourself
Two supported paths: LoRA fine-tuning on a 32GB-plus card, or full fine-tuning on 80GB-plus. Data must respect the model's action normalization statistics and RVQ codebook, and the model card warns that a mismatch in normalization or RVQ code version breaks behaviour. The action space is relative, so gripper geometry and camera extrinsics must be calibrated to the UMI convention.
Where it helps, where it does not
Strengths
- Apache-2.0 weights on an 8B model, unusually permissive at this scale
- Explicit, published VRAM tiers for inference, LoRA fine-tuning and full fine-tuning, which almost no competing VLA publishes
- Zero-shot cross-embodiment deployment is the design goal and is demonstrated on two unseen arm platforms
- Trained on 10,000-plus hours of UMI data, which decouples data collection from any specific robot
Limits
- Bound to the UMI action convention: the model card states that open-world robustness depends on calibration quality, camera placement and gripper specifics
- Input is binocular wrist-camera RGB, so a robot without the expected wrist camera geometry needs re-rigging, not just fine-tuning
- No readable numeric success-rate tables in the paper, only bar figures, which makes head-to-head comparison against pi-0.5 or GR00T unverifiable from the primary source
- No published millisecond latency for the base model; the speed claim applies to a distilled UltraFast variant
- The model card explicitly requires human supervision and hardware interlocks for safety-critical use
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://huggingface.co/robotics-diffusion-transformer/RDT2-VQ
- https://github.com/thu-ml/RDT2
- https://arxiv.org/abs/2602.03310
- https://arxiv.org/html/2602.03310v1
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.