RoboCat
Google DeepMind, UK · June 2023
- Parameters
- 1.2 B
- The paper specifies a 1.18B parameter decoder-only transformer (24 layers, embedding size 2048, feedforward hidden size 8196).
- GPU memory
- 2.2 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- closed
- no public checkpoint
What it is
RoboCat is a 1.18B parameter visual goal-conditioned decision transformer built on the Gato architecture with a frozen VQ-GAN encoder. Its distinguishing idea is the self-improvement loop, where the trained agent generates data for its own next generation, which raised success on unseen tasks from 36 percent to roughly 74 percent. It ran on 36 real arms across Panda, Sawyer and KUKA embodiments and adapted to new ones from 100 to 1000 demonstrations. It is conditioned on goal images rather than language, which makes it awkward for instruction-following use, and nothing was ever released. DeepMind's robotics work has since moved to the Gemini Robotics family.
Architecture
- Backbone
- Gato architecture. A 1.18B parameter decoder-only transformer with 24 layers, embedding size 2048 and a post-attention feedforward hidden size of 8196, on top of a pretrained and frozen VQ-GAN image encoder.
- Action head
- Visual goal-conditioned decision transformer. Actions are emitted as discrete tokens conditioned on a goal image rather than on a language instruction.
- Parameters
- The paper specifies a 1.18B parameter decoder-only transformer (24 layers, embedding size 2048, feedforward hidden size 8196). The VQ-GAN encoder is pretrained and frozen and is not counted separately in the report.
- Pretraining data
- Millions of trajectories from simulated and real robot arms, including self-generated data. The mixture spans 2 simulated and 3 real-world embodiments, 5 simulated and 11 real task families, and 123 simulated and 134 real objects.
- Embodiments
- Franka Panda (15 arms), Rethink Sawyer (17 arms), KUKA arm with three-fingered gripper (4 arms), 2 additional simulated embodiments
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 4.4 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 2.2 GB | The usual way these checkpoints are served |
| int8 | 1.1 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.5 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
No results in this category are published for this model.
Fine tuned tasks
What the model reaches after being adapted to a new task or a new robot, usually from a small number of demonstrations. This is the number that matters if you bring your own data.
- RoboCat fine-tuning with 1000 demonstrations KUKA lifting86%success ratesource
- RoboCat fine-tuning with 1000 demonstrations insertion-bowl84%success ratesource
- RoboCat self-improvement loop previously unseen tasks, final self-improved generation74%success rateDeepMind states the final generation more than doubled the success rate on the same tasks.source
- RoboCat fine-tuning with 1000 demonstrations removal-bowl72%success ratesource
- RoboCat self-improvement loop previously unseen tasks, first generation after 500 demonstrations36%success ratesource
On real hardware
Controlled 36 real robots in total (15 Panda, 17 Sawyer, 4 KUKA). It adapted to a KUKA arm with a three-fingered gripper and roughly twice the controllable inputs of the training arms, using 100 to 1000 demonstrations. Training task success spans a wide band, from 17 percent to 97 percent depending on the task.
Fine tuning it yourself
The published adaptation recipe uses 100 to 1000 demonstrations of the target task, after which the fine-tuned agent is used to generate additional data for the next training generation. None of this is reproducible externally since no weights or code exist.
Where it helps, where it does not
Strengths
- Demonstrated a working self-improvement loop where the model generates its own training data
- Adapts to a new embodiment with 100 to 1000 demonstrations, including a three-fingered gripper with twice the DoF
- Scale of real hardware evidence is unusual: 36 physical arms across three embodiments
Limits
- Goal-image conditioned rather than language conditioned, so you must supply a goal image for every task
- Weights, code and data were never released
- Training task success ranges from 17 percent to 97 percent, so per-task variance is very large
- No published control frequency, latency or memory requirements
- Superseded inside DeepMind by the Gemini Robotics line, and no RoboCat successor was released
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2306.11706
- https://deepmind.google/discover/blog/robocat-a-self-improving-robotic-agent/
- https://ar5iv.labs.arxiv.org/html/2306.11706
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.