RT-1: The Robot Transformer That Started It All

RT-1 was the paper that made large-scale imitation learning for robots look like a serious engineering programme rather than a demo. It arrived on arXiv on 13 December 2022 as arXiv 2212.06817, carried 51 authors from Robotics at Google, Everyday Robots and Google Research, and was published at Robotics: Science and Systems 2023. Its claim was narrow and testable: a 35 million parameter transformer, trained by imitation learning on about 130,000 human teleoperation demonstrations, can perform over 700 language instructions on a real robot at 97 percent success and still run its control loop at 3 Hz.

What RT-1 actually is

Strip the marketing and RT-1 is a sequence model that maps six camera frames plus one instruction string to eleven integers. Those integers index bins. Each bin decodes back to a float that the robot's controller consumes as a delta command. There is no planner, no state estimator, no task graph. The whole policy is one forward pass, repeated three times a second until the model emits a terminate token or the episode hits a step limit.

The mode dimension is the part people forget. It is a three-way switch: control the arm, control the base, or terminate. RT-1 is a mobile manipulation policy, not an arm policy, and the base dimensions are part of the same token sequence as the wrist rotation. That matters if you try to port the idea to a fixed-base arm like the SO-100, where three of the eleven dimensions have nothing to predict.

The 130,000 demonstrations, and how they were collected

Thirteen robots, seventeen months, one purpose-built room. The paper calls the collection sites robot classrooms: partial kitchen counters modelled on two real office kitchens, which were kept as held-out evaluation environments. Each robot drove itself to its station, told the human operator which instruction to demonstrate and how to randomise the scene, then recorded the episode. Demonstrations were captured with two VR remotes in direct line of sight, with the remote's 6-DoF displacement mapped onto the policy's own action space so that the transition dynamics stayed consistent between demonstration and rollout.

That distribution is worth staring at. Two skills account for 499 of the 744 instructions. Placing upright and knocking over have eight each. RT-1 is not evenly multi-task; it is a picking-and-moving policy with a long tail. If you have ever built your own LeRobot dataset you will recognise the shape: the task you started with dominates, and the variations you added later are thin. The public dataset directory has the same skew.

The dataset itself is public. It ships as fractal20220817_data in TensorFlow Datasets: 87,212 train episodes, 111.38 GiB, images at 256 x 320, with the instruction carried both as a string and as a precomputed 512-dimensional sentence embedding. It later became one of the larger single components of Open X-Embodiment, whose own dataset figure names xArm and Google Robot as the two embodiments contributing the most trajectories.

Action tokenisation: 256 bins, and nothing clever

This is the single idea from RT-1 that spread furthest, and it is almost embarrassingly simple. Clip the action to the spec bounds, normalise to [0, 1], multiply by 255, cast to int. The released tokeniser is 157 lines. Here is the entire encode path, verbatim from the archived repository:

a = tf.clip_by_value(a, spec.minimum, spec.maximum)
# Normalize the action [batch, actions_size]
token = (a - spec.minimum) / (spec.maximum - spec.minimum)
# Bucket and discretize the action to vocab_size, [batch, actions_size]
token = tf.cast(token * (self._vocab_size - 1), tf.int32)

a = action_tokens[..., token_index:token_index + 1]
a = tf.cast(a, tf.float32)
a = a / (self._vocab_size - 1)
a = (a * (spec.maximum - spec.minimum)) + spec.minimum

Integer-typed dimensions such as the terminate flag are assumed to be tokens already and pass through with an argmax. Everything continuous gets the bucket treatment. RT1ActionTokenizer itself has no default bin count: vocab_size is a required constructor argument. The 256 comes one level up, from TransformerNetwork, which defaults to vocab_size=256 and passes that straight into the tokeniser. The tokeniser docstring's worked example builds eight tokens per action (terminate, three world-vector, three rotation-delta, one gripper) rather than the eleven the paper describes, because the spec in that example has no mobile-base dimensions.

The architecture, stage by stage

Two details in that table did real work. First, the FiLM layers are identity-initialised: the dense layers producing the affine transform start at zero, so the inserted FiLM layer initially does nothing and the ImageNet weights survive intact. Insert FiLM naively into a pretrained network and you scramble the intermediate activations you paid for. Second, action tokens are emitted in a single pass, not autoregressively. The ablation shows why: autoregressive actions cost more than 2x inference time (36 ms against 15 ms) for no gain worth having.

The released library exposes the whole thing as a Keras network, though its constructor defaults are the library defaults rather than the paper configuration. Read them for the shape of the model, not for the numbers that produced the published results.

TransformerNetwork(
    vocab_size=256,
    token_embedding_size=512,
    num_layers=1,
    layer_size=4096,
    num_heads=8,
    feed_forward_size=512,
    dropout_rate=0.1,
    time_sequence_length=1,
    use_token_learner=True,
)

The numbers RT-1 reported

Every baseline in the comparison was retrained on RT-1's own dataset, so this table compares architectures, not data. Gato was shrunk from 1.2 B parameters to 37 M because the full model needed 1.9 s per action on the robot, which is not a control loop. BC-Z is the ResNet feedforward policy from the BC-Z paper, also present in SayCan.

The ablation table is more informative than the headline. It tells you which design choices carried weight and which were decoration. One caution on the size row: the paper gives two different figures for the same ablation. The ablation description says the model was cut from 35 M to 21 M parameters, and the discussion says that cut removed 40 percent of the parameters, which only fits 35 to 21. The same sentence then prints 31 M to 25 M, which is a 19 percent cut. The two self-consistent numbers are used here.

The finding that outlived the architecture

Section 6.5 of the paper is the part still worth quoting in 2026. The team built reduced datasets two ways: cap the number of examples per task (same breadth, less depth) or delete the tasks with the least data (same depth, less breadth). Then they measured what broke.

Removing 25 percent of the tasks while keeping 97 percent of the frames damaged generalisation about as much as throwing away half the dataset. The paper's own summary: data diversity has a larger impact than data size. That is the single most transferable sentence in the paper, and it is the reason the platform's minimum-episode numbers are floors rather than targets. Fifty episodes of one scene will fit; fifty episodes spread over lighting, object placement and start pose will generalise. There is more on this in the guide to collecting high-quality VLA training data and in the recording tutorial.

Running RT-1 today, in 2026

The original repository, google-research/robotics_transformer, is archived and read-only; its last commit landed on 31 January 2024. It contains the tokenisers, the FiLM-EfficientNet backbone, the transformer and three SavedModel checkpoints. It contains no training script and no dataset loader, and its requirements file sets floors from the TensorFlow 1.13 era (tensorflow>=1.13.0, tensorflow-serving-api>=1.13.0, tf-agents>=0.3.0) and installs tensor2robot from git. Do not start there.

The working path is SimplerEnv, the real-to-sim benchmark from the SIMPLER paper. It ships a maintained RT-1 policy wrapper, and Google's public bucket still hosts three RT-1 checkpoints at different training stages plus the RT-1-X one. This runs today on a single NVIDIA GPU.

1. 1
   Create the environment

   SimplerEnv needs CUDA 11.8 or newer and below 13, an NVIDIA GPU (ray tracing is slow on non-RTX cards), and Python 3.10 or 3.11. TPUs are not supported because SAPIEN needs a GPU.

   bash

   conda create -n simpler_env python=3.10
   conda activate simpler_env
   git clone https://github.com/simpler-env/SimplerEnv --recurse-submodules

2. 2
   Install the simulator, with numpy pinned

   The numpy pin is not optional. Newer numpy breaks inverse kinematics in pinocchio, and the failure looks like a physics bug rather than a dependency bug.

   bash

   pip install numpy==1.24.4
   cd SimplerEnv/ManiSkill2_real2sim && pip install -e .
   cd .. && pip install -e .

3. 3
   Add the TensorFlow stack for RT-1 inference

   RT-1 is a TensorFlow SavedModel. This is the step that pulls in a couple of gigabytes.

   bash

   sudo apt install ffmpeg
   pip install tensorflow==2.15.0
   pip install -r requirements_full_install.txt
   pip install 'tensorflow[and-cuda]==2.15.1'

4. 4
   Download the checkpoints

   Four public checkpoints sit in the same bucket: three of RT-1, named by training step, and one of RT-1-X. rt_1_tf_trained_for_000400120 is the converged RT-1 used for the headline results; each of the three RT-1 directories is 617 MiB across the SavedModel and the raw ckpt files. The 58,240-step and 1,120-step checkpoints exist so you can measure a policy that is partly and barely trained, which is genuinely useful for calibrating your own eval harness.

   bash

   mkdir -p checkpoints
   BUCKET=gs://gdm-robotics-open-x-embodiment/open_x_embodiment_and_rt_x_oss
   
   # converged RT-1, 400,120 steps
   gsutil -m cp -r $BUCKET/rt_1_tf_trained_for_000400120 checkpoints/
   
   # RT-1 at 15% of training, 58,240 steps
   gsutil -m cp -r $BUCKET/rt_1_tf_trained_for_000058240 checkpoints/
   
   # RT-1-X, 2,272,480 steps, 739 MiB zip
   gsutil -m cp $BUCKET/rt_1_x_tf_trained_for_002272480_step.zip .
   unzip rt_1_x_tf_trained_for_002272480_step.zip -d checkpoints/

5. 5
   Run one episode

   The prepackaged visual-matching environments give you a Google Robot scene with real backgrounds composited in. Google Robot environments run at 3 Hz control and roughly 500 Hz simulation, matching RT-1's own control rate.

   python

   import simpler_env
   from simpler_env.utils.env.observation_utils import get_image_from_maniskill2_obs_dict
   
   env = simpler_env.make('google_robot_pick_coke_can')
   obs, reset_info = env.reset()
   instruction = env.get_language_instruction()
   print('Instruction', instruction)

Two paths to a language-conditioned policy on your own arm

What survived, and what did not

Read that table as a fork, and it is the fork that shapes every vision-language-action model built since. The token branch runs through RT-2 into OpenVLA and buys you an off-the-shelf language model as your policy backbone. The continuous branch runs through diffusion and flow matching and buys you smooth, high-rate control. The head-to-head on GR00T N1.7 against Pi0.5 is a comparison between two members of the second branch; neither of them tokenises actions the way RT-1 did.

How RT-1 holds up when someone else evaluates it

The SIMPLER benchmark re-ran RT-1 under controlled simulation in 2024 and reported both the original real-world numbers and the simulated ones. On the visual-matching setup the correlation with real performance is strong: Pearson r of 0.976 on pick-coke-can and 0.915 on the drawer tasks.

That last row is the honest one. On a two-stage task, the converged RT-1 succeeds under one attempt in five. The 97 percent headline was measured on single-skill instructions. Chain two of them and the success rate multiplies down, which is exactly why the SayCan long-horizon results in the paper top out at 67 percent execution success over roughly ten steps.

What this means if you have an SO-100 on your desk

RT-1's scale is not reachable in a hobby setup and does not need to be. 130,000 demonstrations across 13 robots over 17 months is an industrial data-collection programme. What transfers is the method: vary the scene between episodes, keep the instruction annotated, prefer more tasks over more repetitions of one task, and treat inference latency as a design constraint rather than a benchmark line.

The latency point deserves a plain statement, because this platform does not solve it. RT-1 ran its 3 Hz loop with the model on the robot. Cloud inference here has a control loop of 20 to 485 ms per action step depending on the model, and adding public-internet round trips on top turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place. It is not viable for fast reactive motion. If your task needs reactivity, the model has to sit next to the servos, and that constrains which of the five policies you can actually use.

If you want to feel the difference between a 2022 discrete-token policy and a 2026 flow-matching one without buying hardware, the live arm streams a physical SO-100 with no signup, and the three ways to start page lays out what each route costs. The RT-1 arena entry and the RT-1-X entry carry the published benchmark numbers with links back to their sources.

The footnote nobody puts in the abstract

Everyday Robots, the Alphabet subsidiary that built the mobile manipulators RT-1 was trained on, was closed in February 2023 during Alphabet's layoffs, roughly two months after the RT-1 preprint appeared. Parts of the team and technology were folded into Google Research. The robot in every figure of the paper no longer exists as a product. What outlived it is the dataset, the tokenisation scheme, and one sentence about data diversity beating data quantity, which is a reasonable definition of a paper that mattered.