Action Tokenization: How FAST Turns Trajectories Into Tokens

A servo takes a number. A transformer emits a symbol from a fixed vocabulary. Action tokenization is the glue between those two facts. Get it wrong and the model trains to a low loss, the checkpoint loads, and the arm still drifts, because its symbols cannot reconstruct the motion you recorded.

Two schemes matter: per-dimension per-timestep binning, used by RT-1, RT-2 and OpenVLA, and FAST, the frequency-space scheme behind Pi0-FAST. It ends on the part most write-ups skip: none of the five policies you can fine-tune here has a tokenizer knob, and that is the right default.

The gap tokenization has to close

An episode from an SO-100 is a table of floats: six joints, one row per control tick, stored by the LeRobot dataset format. Record at 30 Hz, ask the policy for one second of motion, and one training example is a 30 by 6 block.

import numpy as np

# one second of SO-100 motion at 30 Hz, six joints, radians
chunk = np.load("episode_004_actions.npy")
print(chunk.shape, chunk.dtype)     # (30, 6) float32
print(chunk.min(), chunk.max())     # -2.61 1.94

# a transformer cannot emit a float32. It emits an index into a vocabulary.
# tokenization decides how those 180 numbers become a sequence of indices.

A diffusion or flow matching head sidesteps this: it regresses the block and never needs a vocabulary. An autoregressive model cannot. Its output layer is a softmax over a fixed vocabulary, so reusing its language pre-training means expressing motion in the same currency as words. What follows is driven by what a language model finds easy to predict.

Binning: one symbol per joint per tick

Take each dimension independently, split its range into N uniform bins, emit the bin index. RT-1 (arXiv 2212.06817, December 2022) set the convention everyone copied: 256 bins per dimension, uniform within each variable's bounds, over 7 arm dimensions, 3 base dimensions and one mode dimension switching between arm, base and termination.

import numpy as np

N_BINS = 256

def bin_encode(chunk, lo, hi, n_bins=N_BINS):
    """chunk: (H, D) float. Returns (H*D,) int token ids."""
    clipped = np.clip(chunk, lo, hi)
    idx = ((clipped - lo) / (hi - lo) * (n_bins - 1)).round().astype(int)
    return idx.reshape(-1)          # row-major: t0d0 t0d1 ... t0d5 t1d0 ...

def bin_decode(tokens, lo, hi, H, D, n_bins=N_BINS):
    idx = np.asarray(tokens).reshape(H, D)
    return lo + idx / (n_bins - 1) * (hi - lo)

# quantile bounds, not min/max: outliers otherwise eat the resolution
lo = np.quantile(all_actions, 0.01, axis=0)
hi = np.quantile(all_actions, 0.99, axis=0)

tokens = bin_encode(chunk, lo, hi)
print(len(tokens))     # 180 for a 30 x 6 chunk. One token per number.

Every model made one variation. They look cosmetic and are not: the bin bounds decide how much of the 256-way resolution you actually use.

Where binning breaks, and it is not where you expect

Binning does not fail because 256 bins are too coarse. It fails on token count. One token per dimension per timestep means sequence length is H times D, so doubling the control frequency doubles the tokens per second.

Read that as a statement about your own recording rate. At 5 Hz compression is nearly irrelevant. At 50 Hz it is the difference between a workable sequence and one that dominates the context window, and every token still has to be predicted at inference.

The subtler failure is statistical. At 50 Hz the joint angle at tick t and tick t+1 usually differ by less than one bin width, so the best strategy under a next-token objective is to copy the previous token. The FAST paper measures it: as sampling density rises, error climbs until the model simply copies the first action. That model is not undertrained. It found the optimum of a badly posed objective.

FAST: compress the trajectory, then tokenize the compression

The paper is by Pertsch, Stachowicz, Ichter, Driess, Nair, Vuong, Mees, Finn and Levine, submitted 16 January 2025. The idea comes from JPEG and MP3: a smooth signal is sparse in the frequency domain, so transform first and quantize second. A joint trajectory is about as smooth as signals get.

1. 1
   Normalize the chunk

   Bring every dimension into roughly [-1, 1]. Quantile normalization survives outliers. Not optional, and the step people skip.

   python

   lo = np.quantile(all_actions, 0.01, axis=0)
   hi = np.quantile(all_actions, 0.99, axis=0)
   norm = 2 * (chunk - lo) / (hi - lo) - 1
   norm = np.clip(norm, -1, 1)   # (30, 6)

2. 2
   Discrete cosine transform along time

   One orthonormal DCT per dimension over the time axis. Same shape out, but the energy moves into the low-frequency coefficients: a slow reach puts almost all of it in the first three.

   python

   from scipy.fft import dct
   
   # tokenizer input is [batch, timesteps, action_dim], DCT over axis=1
   coeff = dct(norm[None], axis=1, norm="ortho")   # (1, 30, 6)

3. 3
   Scale and round

   Multiply by the scale hyperparameter, round to integers. The lossy part: larger scale keeps more small coefficients and lengthens the sequence, smaller scale blurs the chunk.

   python

   SCALE = 10
   coeff_q = np.around(coeff * SCALE)
   print((coeff_q == 0).mean())
   # a large fraction of coefficients are now exactly zero

4. 4
   Flatten low frequency first

   Row-major flattening puts the lowest-frequency component of every dimension first, so runs of zeros end up adjacent at the tail.

   python

   flat = coeff_q[0].flatten().astype(int)   # length 180
   # order: freq0-dim0..dim5, freq1-dim0..dim5, ...

5. 5
   Byte pair encoding over the integers

   Integers become characters and BPE is trained over those strings, merging frequent runs. The runs are mostly zeros, so the tail collapses. Lossless.

   python

   # what the shipped tokenizer literally does. min_token shifts the
   # integer range so it can be addressed as unicode code points
   token_str = "".join(map(chr, np.maximum(flat - MIN_TOKEN, 0)))
   tokens = bpe_tokenizer(token_str)["input_ids"]

The whole pipeline is 158 lines of Python in processing_action_tokenizer.py, depending on nothing heavier than scipy and the tokenizers library. Using the released tokenizer takes four lines.

import numpy as np
from transformers import AutoProcessor

# FAST+, fitted on roughly 1M real robot action chunks
tokenizer = AutoProcessor.from_pretrained(
    "physical-intelligence/fast", trust_remote_code=True
)

# expects actions already normalized to [-1, 1]
action_data = np.random.rand(256, 50, 14)     # [batch, horizon, action_dim]
tokens = tokenizer(action_data)               # list[list[int]]
decoded = tokenizer.decode(tokens)            # back to [256, 50, 14]

print(np.mean([len(t) for t in tokens]))      # tokens per chunk

The decoder needs the horizon and action dimension, because the token sequence does not encode its own shape. The tokenizer caches them from the last encode call, which works in a notebook and misbehaves in a serving loop where encode and decode run in different processes. Pass both to decode() explicitly when you deploy.

The defaults, and where paper and artifact disagree

The paper uses rounding scale 10 and BPE vocabulary size 1024 across its single-dataset experiments, and says the parameters are not very sensitive. The released tokenizer says something else. Its config, last modified 16 January 2025:

The vocabulary difference is not a typo. FAST+ covers single-arm, bi-manual and mobile manipulators at many control frequencies, so its coefficient range is wider. The negative min_token says the same: the smallest scaled coefficient during fitting was -354, and everything shifts by that to be addressable as a code point.

# run this before you spend money on a GPU
tokens = tokenizer(norm_chunks)                      # [B, H, D] in [-1, 1]
recon  = tokenizer.decode(tokens, time_horizon=H, action_dim=D)

mae = np.abs(recon - norm_chunks).mean()
p99 = np.percentile([len(t) for t in tokens], 99)

print(f"round-trip MAE {mae:.4f}   p99 tokens/chunk {p99:.0f}")
# MAE should be small relative to your action scale.
# p99 must stay below max_action_tokens (256 in lerobot pi0_fast).

Fitting a tokenizer to your own SO-100 data

FAST+ works across action spaces, but the paper's own framing is that a tokenizer fitted to your data compresses better. lerobot ships a command, and fitting takes minutes rather than GPU hours.

1. 1
   Install the pi extras

   The FAST path sits behind the pi extra. Without it, an import error on scipy or tokenizers.

   bash

   git clone https://github.com/huggingface/lerobot.git
   cd lerobot
   pip install -e ".[pi]"

2. 2
   Fit the tokenizer on your dataset

   --encoded_dims selects which dimension ranges to encode, which matters for bi-manual rigs. For a six-joint SO-100 it is "0:6".

   bash

   lerobot-train-tokenizer \
       --repo_id "yourname/so100-pick-place" \
       --action_horizon 10 \
       --encoded_dims "0:6" \
       --vocab_size 1024 \
       --scale 10.0 \
       --normalization_mode QUANTILES \
       --sample_fraction 0.1 \
       --output_dir "./fast_so100" \
       --push_to_hub \
       --hub_repo_id "yourname/fast-so100"

3. 3
   Read the compression report

   The script samples up to 1000 chunks and logs compression ratio, mean and p99 token length. Its ratio is (horizon x action_dim) over mean tokens.

   text

   Compression Statistics:
     Average compression ratio: 4.31x
     Mean token length: 13.9
     P99 token length: 24
     Min token length: 8
     Max token length: 31

4. 4
   Train the policy against it

   chunk_size and n_action_steps must match the horizon you fitted on, or the decoder reshapes into the wrong geometry.

   bash

   lerobot-train \
       --dataset.repo_id=yourname/so100-pick-place \
       --policy.type=pi0_fast \
       --policy.pretrained_path=lerobot/pi0fast-base \
       --policy.action_tokenizer_name=yourname/fast-so100 \
       --policy.chunk_size=10 \
       --policy.n_action_steps=10 \
       --policy.max_action_tokens=256 \
       --policy.dtype=bfloat16 \
       --policy.gradient_checkpointing=true \
       --batch_size=4 --steps=100000 --policy.device=cuda

If the vocabulary is too small for the coefficient range, fit() raises an assertion rather than degrading quietly, and warns when the alphabet comes within 100 entries of it. Raise the vocabulary rather than lowering the scale: lowering the scale throws away real motion.

What tokens cost you at inference

This is the part lost when FAST gets called a straight win. Compression is a training-time win. At inference the policy emits its 30 to 60 tokens one forward pass at a time, and nothing removes that.

750 ms is fine for a one-second chunk executed open loop, and not fine if you wanted to replan. lerobot softens the cost with KV-caching and greedy decoding at temperature 0.0, but sequential forward passes still equal token count. Same inference latency budget as in action chunking: a longer chunk amortizes latency but stales the plan.

Where the five trainable policies actually sit

Here is the honest part. If you are about to fine-tune something on an SO-100, tokenization is not your decision. Every policy on this platform predicts continuous actions at inference: no bin count and no FAST scale, because none of the five predicts tokens.

That row deserves its footnote. The Pi0.5 paper (arXiv 2504.16054, 22 April 2025) states that during pre-training all tasks including robot actions are represented with discrete tokens, which leads to simple, scalable and efficient training, and that post-training adds an action expert for finer granularity and more compute-efficient inference, because discrete representations require expensive autoregressive decoding. FAST is in the ancestry of the Pi0.5 checkpoint you fine-tune, and out of the loop when you run it.

That is a design decision, not a missing feature. The tokenizer is only a lever when you build an autoregressive VLA from a language backbone, and for a six-joint arm at 30 Hz the defaults are already the useful settings. To compare the families on published benchmarks, the arena lists Pi0-FAST, Pi0 and Pi0.5.

Two routes to the same fine-tuned policy

Failure modes that come from the tokenizer

Most policy failures are data problems, not representation problems. A handful of symptoms point at the tokenizer, and they are worth recognising because they look like model failures.

• The arm holds its starting pose and drifts. Binning collapse: consecutive values fall in the same bin, so the model learned to copy.
• Motion is right in shape but visibly quantized. Bin resolution eaten by outliers, or a DCT scale so low the mid-frequency coefficients rounded to zero.
• Fast segments fail while slow segments work. Fast motion means more non-zero coefficients, more tokens, and hits the max_action_tokens ceiling first. Check the p99.
• The policy stops mid-motion and resumes at the next chunk. A decode failure that silently returned a zero chunk. See policy freezes mid-motion.
• Reconstruction is fine locally and terrible in the serving loop. The decoder is using a cached shape from a different call.

How to decide

Autoregressive VLA from a language backbone, data above about 20 Hz: use FAST, not binning. The paper's 20 Hz and 50 Hz results show binning making no progress. On 5 Hz data binning is defensible. With a working continuous action head, tokenization is a training-speed argument against an inference-speed penalty, which fits pre-training and not the last mile. Fine-tuning one of the five here: spend the attention on recording better episodes instead.

One caveat applies to every route. The control loop here runs at 20 to 485 ms per action step, and inference has to sit next to the servos for anything fast. Public-internet round trips on top of a 485 ms Pi0.5 step turn a working policy into a hesitant one. Remote inference suits slow pick-and-place, not fast reactive motion. Background in our overview of VLA models and the Pi0 flow matching write-up.