RT-2
Google DeepMind, US · July 2023
- Parameters
- 55 B
- The headline model RT-2-PaLI-X-55B has 55B parameters.
- GPU memory
- 102 GB
- weights at bf16, computed
- Inference latency
- not published
- per action step
- Weights
- closed
- no public checkpoint
What it is
RT-2 is the model that defined the term vision-language-action. It co-fine-tunes a large VLM (PaLI-X at 5B and 55B, or PaLM-E at 12B) on robot trajectories and internet VQA data by writing actions as text tokens in the same vocabulary. The payoff is semantic: about 3x on emergent skills such as symbol understanding and reasoning, and roughly 2x generalization to unseen objects and environments. The cost is deployment, since the 55B model only reaches 1 to 3 Hz served from a multi-TPU cloud service, and nothing was ever released. Treat it as a reference point in the literature, not as something you can run.
Architecture
- Backbone
- Two instantiations: PaLI-X (5B and 55B) and PaLM-E (12B), co-fine-tuned on robot trajectories and internet vision-language data.
- Action head
- Actions expressed as text tokens. Each action dimension is emitted autoregressively as a string in the same vocabulary as natural language, which is what allows co-fine-tuning with VQA data.
- Parameters
- The headline model RT-2-PaLI-X-55B has 55B parameters. The paper also evaluates RT-2-PaLI-X-5B, RT-2-PaLM-E-12B and a PaLI-3B variant used for the simulated Language-Table experiments. No weights of any size were released.
- Pretraining data
- Web-scale vision-language pretraining inherited from PaLI-X and PaLM-E, co-fine-tuned with the RT-1 robot demonstration dataset (over 130k episodes on the Everyday Robots mobile manipulator).
- Embodiments
- Everyday Robots mobile manipulator (single arm plus mobile base)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 205 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 102 GB | The usual way these checkpoints are served |
| int8 | 51 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 26 GB | Aggressive quantisation, verify success rate before trusting it |
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- Language-Table (simulation) held-out simulated tasks90%success rateRT-2-PaLI-3B variant, reported as 90 +/- 10. Same table lists RT-1 at 74 +/- 13 and LAVA at 77 +/- 4.source
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- RT-2 emergent skills evaluation symbol understanding, reasoning and human recognition3 x
- RT-2 generalization evaluation unseen objects, backgrounds and environments2 ximprovement over baselinesStated by the authors as approximately 2x, no exact per-category table on the project page.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
Evaluated on the Everyday Robots mobile manipulator across roughly 6k real evaluation trials. The demonstrated wins are semantic rather than dexterous: interpreting commands absent from robot data, picking by relative size or proximity, and chain-of-thought selection of an improvised tool.
Fine tuning it yourself
Not possible outside Google. No weights, no code and no serving stack were published.
Where it helps, where it does not
Strengths
- First model to show that web-scale VLM knowledge transfers directly into robot action, roughly 3x on emergent skills
- Chain-of-thought prompting enables multi-stage semantic reasoning before acting
- Actions as text tokens is a simple recipe that most later VLAs build on
Limits
- Weights, code and checkpoints were never released, so it cannot be reproduced or deployed
- Requires a multi-TPU cloud service, which rules out offline or latency-critical use
- 1 to 3 Hz for the 55B model is too slow for reactive control
- Single embodiment only, and the paper reports no new dexterous skills beyond what the robot data already contained
- No parameter-efficient or consumer-GPU path exists at 55B
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2307.15818
- https://robotics-transformer2.github.io/
- https://ar5iv.labs.arxiv.org/html/2307.15818
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.