Octo: The Open-Source Generalist Robot Policy, Explained

Octo is a transformer policy trained on 800k robot trajectories from Open X-Embodiment and released whole: weights, pretraining code, finetuning scripts and data loaders, all MIT. It came out of UC Berkeley's RAIL lab with Stanford, CMU and Google DeepMind co-authors, and was presented at Robotics: Science and Systems 2024. Two sizes shipped, 27M and 93M parameters, which makes it a very small vision-language-action model. That smallness is the point: Octo is the cleanest worked example of a design the big VLAs inherited and then buried under scale, one shared transformer taking arbitrary tokens in and a tiny separate head turning one output embedding into a chunk of continuous actions. It is also not one of the five models AY-Robots trains, and the tabs below say what that means in practice.

What Octo actually is

A policy maps observations to actions. Most published robot policies fix the observation space at training time: one camera, this resolution, this action dimension. Change any of it and you retrain from scratch. Octo's contribution is that it does not. You can add or remove a camera, swap end-effector deltas for joint positions, or move from a single arm to a bimanual setup, and keep the pretrained transformer.

The README prints those speeds as it/sec without defining the unit. Read them as model calls: Octo emits 4 actions per call, so 13 it/sec is about 77 ms per call, or about 19 ms per commanded action if you execute the whole chunk. That is the order of ACT and far quicker than any 3B VLA. Run receding horizon control instead, one action per call, and you throw that factor of four away. See action chunking for why that choice changes how a policy feels on hardware.

The architecture: everything becomes a token

No fusion module, no cross-attention adapter, no per-embodiment head registry. Every input becomes tokens, the tokens get learnable position embeddings, and one transformer attends over all of it.

Tokenizers

• Workspace camera at 256x256: shallow convolutional stack, 16x16 patches, 256 tokens per frame.
• Wrist camera at 128x128: same stack, same patch size, 64 tokens per frame.
• Language: T5 tokenizer plus a frozen t5-base (111M) encoder, 16 tokens. The pretraining config freezes it with frozen_keys=("*hf_model*",), so those 111M parameters never see a gradient.
• Goal image: an optional task token block, run through the same image tokenizer as an observation. It is what lets Octo be conditioned on a target state instead of a sentence.
• History: 2 frames. The paper reports significantly diminishing gains beyond the first extra frame.

Note what is missing. The released checkpoints carry no proprioception tokenizer, and that is a result rather than an oversight: the authors report that policies trained with proprioceptive observations were generally worse, which they attribute to causal confusion between the state and the actions it correlates with. If you want joint angles anyway, you add the tokenizer at finetuning time, which is what the repo's ALOHA example does.

Readout tokens and block-wise attention

Octo inserts learned readout tokens into the sequence. A readout token attends to the observation and task tokens before it and is not attended to by any of them, a passive probe; the paper's analogy is the BERT [CLS] token. The action head consumes that embedding, not the observation embeddings.

The diffusion action head

The head is small and separate: a 3-layer MLP with hidden dimension 256, residual connections and layer normalization, trained with the standard DDPM objective on a cosine noise schedule with 20 denoising steps. The transformer runs once per timestep for the readout embedding, then the denoising loop runs entirely inside that MLP, so denoising cost is small and roughly independent of backbone size.

# octo/model/components/action_heads.py, class DiffusionActionHead
readout_key: str
use_map: bool = False          # False = mean-pool, True = attention pooling
action_horizon: int = 1        # the pretraining config sets this to 4
action_dim: int = 7            # 6-DoF end-effector delta + gripper
max_action: float = 5.0
loss_type: str = "mse"

# diffusion-specific config with sane defaults
time_dim: int = 32
num_blocks: int = 3            # the "3-layer MLP" from the paper
dropout_rate: float = 0.0      # turned off in 1.5, it fought layer norm
hidden_dim: int = 256
use_layer_norm: bool = True
diffusion_steps: int = 20      # cosine beta schedule
n_diffusion_samples: int = 1

Both obvious alternatives were tried, and the failures are in the appendix. A simple L2 loss produced hedging policies that moved very slowly and failed to rotate the gripper in the WidowX evaluations. Discretizing each dimension into 256 bins with cross-entropy, the RT-1 recipe, gave more decisive policies that lacked precision and often missed the grasp.

Read it as a ranking: the discretized head cost 65 points, cutting the mix to one robot cost 40, the visual backbone cost 13. Rates are averaged over 40 trials across four tasks, so single-digit gaps are noise; the ordering is not.

What it was trained on

The mix is 25 datasets picked out of Open X-Embodiment, the 22-robot, 527-skill collection from 21 institutions. Curation was subtractive: drop everything with no image stream, everything not using delta end-effector control, then whatever is too repetitive, too low-resolution or too niche. In the repo the survivors are the constant oxe_magic_soup, where a 26th entry, uiuc_d3field, sits commented out because its raw data is broken. Preprocessed, the mix is about 1.2 TB.

Those shares are mostly relative dataset size, with two manual thumbs on the scale: datasets judged more diverse get double weight, a few very repetitive ones get down-weighted so they cannot dominate. Fractal and Kuka carry explicit sub-1.0 weights in the repo, 0.541 and 0.834, and still land at 17 percent of every batch because they are that much bigger. Two of the sources have their own write-ups: BridgeData V2 and BC-Z.

Gripper actions across all 25 sources were aligned to one absolute convention, plus one open, zero closed. A relative encoding was tried, plus one or zero only on the timestep the gripper changes and 0.5 otherwise. It raised grasp success slightly, most frames being do-not-change commands, but the policy then retried less after a failed grasp, which the authors judged worse overall.

What the authors say actually moved the needle

Appendix E lists what worked and what did not, and it is more useful than the headline results. Findings that transfer to any imitation learning pipeline, not just this one:

• 16x16 image patches beat 32x32, particularly on grasping and other fine-grained tasks, at 4x the token count.
• Action chunking produced more coherent movement. Temporal ensembling on top of receding horizon control added nothing in their evaluations.
• Shuffling mattered. A 20k shuffle buffer with trajectory-level interleaving hurt zero-shot performance badly; they shuffle frames before decoding images to fit a 500k buffer, and subsample at most 100 steps per trajectory.
• The frozen t5-base gave the best language-conditioned policies. A t5-large (386M) encoder did not help, nor did finetuning its last two layers, which they blame on how little free-form language the datasets contain.

Adapting Octo to a new observation and action space

This is what the paper is about, and the repo ships a runnable example: examples/02_finetune_new_observation_action.py refits a checkpoint pretrained on single-arm 7-dimensional end-effector deltas to a simulated bimanual ALOHA cube-handover setup: 14-dimensional joint actions, one camera, proprioception. Nothing about that target matches the pretraining. Five moves:

1. 1
   Install the pinned stack

   Requirements pin jax==0.4.20, flax==0.7.5, tensorflow==2.15.0, numpy==1.24.3 and Python 3.10. Do not float these.

   bash

   conda create -n octo python=3.10
   conda activate octo
   git clone https://github.com/octo-models/octo.git
   cd octo
   pip install -e .
   pip install -r requirements.txt
   pip install --upgrade "jax[cuda11_pip]==0.4.20" \
     -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

2. 2
   Prove the path on the debug dataset first

   The repo ships a tiny dataset under tests/debug_dataset, which is what --debug points at. Verify the install before converting anything of your own.

   bash

   python scripts/finetune.py \
     --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 \
     --debug

3. 3
   Load the checkpoint and read its spec

   get_pretty_spec() prints the observation and task keys the checkpoint expects. Read it before writing a data transform, not after.

   python

   from octo.model.octo_model import OctoModel
   
   model = OctoModel.load_pretrained("hf://rail-berkeley/octo-base-1.5")
   print(model.get_pretty_spec())

4. 4
   Delete a tokenizer, add a tokenizer, replace the head

   The pretrained config is a plain dict. Here the 7-dim diffusion head becomes a 14-dim L1 head with a 50-step horizon, the ALOHA recipe.

   python

   config = pretrained_model.config
   del config["model"]["observation_tokenizers"]["wrist"]
   
   config["model"]["observation_tokenizers"]["proprio"] = ModuleSpec.create(
       LowdimObsTokenizer,
       n_bins=256, bin_type="normal", low=-2.0, high=2.0,
       obs_keys=["proprio"],
   )
   config["model"]["heads"]["action"] = ModuleSpec.create(
       L1ActionHead,
       action_horizon=50,
       action_dim=14,
       readout_key="readout_action",
   )

5. 5
   Rebuild from the edited config, then merge the old weights in

   from_config initialises fresh parameters, then merge_params copies in every pretrained tensor whose name and shape still match. New position embeddings and the new head stay random. That is the whole mechanism.

   python

   model = OctoModel.from_config(
       config, example_batch, text_processor,
       dataset_statistics=dataset.dataset_statistics,
   )
   merged_params = merge_params(model.params, pretrained_model.params)
   model = model.replace(params=merged_params)

The example runs 5000 steps at a constant 3e-5 after a 100-step warmup, which is a smoke test, not the recipe. The paper's is longer: about 100 in-domain trajectories, 50k steps, cosine decay with linear warmup, full model updated. The authors state that updating everything beat the recipes freezing subsets, worth knowing because the repo still offers head_only and head_mlp_only.

The finetuning defaults you will actually run

The advanced path is scripts/finetune.py driven by scripts/configs/finetune_config.py. Its config string is two comma-separated choices, freezing mode and conditioning mode, defaulting to full,multimodal. Values below come from the last commit, 31 July 2024.

# full transformer update, images only, no language
python scripts/finetune.py \
  --config=scripts/configs/finetune_config.py:full,image_conditioned \
  --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 \
  --config.save_dir=/path/to/checkpoints

# reproduce the pretraining instead (needs the ~1.2 TB OXE mix)
python scripts/train.py \
  --config scripts/configs/octo_pretrain_config.py:vit_b \
  --name=octo \
  --config.dataset_kwargs.oxe_kwargs.data_mix=oxe_magic_soup

Do it yourself, or use what is already wired up

Same goal in both columns: a policy running on your own arm, trained on your own demonstrations.

# after your dataset is in RLDS and registered
python scripts/finetune.py \
  --config=scripts/configs/finetune_config.py:full,language_conditioned \
  --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 \
  --config.dataset_kwargs.name=my_arm_dataset \
  --config.dataset_kwargs.data_dir=/data/rlds \
  --config.save_dir=/data/ckpt

The numbers, from three evaluations

The paper's own finetuning table

Six target domains, roughly 100 demonstrations each, identical hyperparameters, 20 trials per domain. Three of the six changed the observation or action space relative to pretraining, which is the claim under test. VC-1 is a pretrained visual representation.

Zero-shot, on setups drawn from the pretraining distribution, Octo averaged a 29 percent higher success rate than RT-1-X (35M) and performed similarly to RT-2-X (55B). On the WidowX tasks, goal-image conditioning beat language conditioning by 25 percent, the goal image simply carrying more information. Both baselines have arena entries: RT-1-X and RT-2-X, and the line is covered in the RT-2 write-up.

LIBERO, as reported by the OpenVLA authors

Independent numbers matter more than a paper's own table. Three seeds, 500 trials each. Octo is finetuned on the target suite and OpenVLA with LoRA; Diffusion Policy is trained from scratch, a strong baseline rather than a pretrained one.

A 93M policy landing 1.4 points behind a 7B policy, and beating it on LIBERO-Goal, is the most quoted fact about Octo and it is fair. It is also narrow: LIBERO is simulation, both pretrained policies saw only real-world data, and the OpenVLA authors attribute the tight margins to exactly that domain gap.

SIMPLER, which is less flattering

SIMPLER evaluated policies in the real world and in a matched simulator, and its real-world tables are the least flattering thing published about Octo. Note what was not run: Octo-Small appears only on WidowX + Bridge and RT-2-X only on the Google Robot, so this is not a clean four-way comparison.

On the Google Robot, where Fractal is 17 percent of the mix, Octo-Base trails both RT models everywhere. On the WidowX, where Bridge is another 17 percent, RT-1-X scores 0.000 on all four tasks while both Octo sizes finish some, and Octo-Small beats Octo-Base on three of the four at less than a third of the size. Distance from the dominant training data predicts performance here better than parameter count.

Where Octo sits next to newer VLAs

Octo went to arXiv on 20 May 2024, last commit 31 July 2024. The field then moved in a direction Octo deliberately did not take: newer models start from a pretrained vision-language model and inherit web-scale semantics, where Octo started from scratch on robot data with only a frozen t5-base. That is why Octo will not follow an instruction about an object it never saw in 800k trajectories, and Pi0.5 or a GR00T checkpoint sometimes will.

The same lab shipped the successor. CrossFormer scaled the identical token-in, readout-out idea to 900K trajectories across 20 robot embodiments, including wheeled robots, quadcopters and quadrupeds, and drops even the manual alignment of observation and action spaces that Octo needed. Neither is a production choice today. For the wider map, the VLA overview covers the families and the arena has the numbers with a source link on every one.

Can you run Octo on an SO-100?

Technically yes, and nobody has made it easy. The missing pieces are the ones the repo never had: an RLDS conversion, a standardization transform, an action space decision (the default action_dim is 7, a 6-DoF end-effector delta plus a gripper, so a joint-space head for an SO-100 means declaring the dimension you command and rebuilding the head for it), and a serving loop that talks to your arm at a fixed rate. None of it is research. All of it is a week.

The parts of this workflow that transfer to any policy are worth investing in: clean, consistently framed episodes, a rig that does not move between recording and inference, enough episodes for the objective to fit. Recording your first dataset and training your first policy walk that path with trainers already wired up, and the dataset directory shows what other people's episodes look like. No arm yet? The live arm streams a physical SO-100 you can drive from the browser without signing up.

Frequently asked questions