HKU

Moto (Moto-GPT)

HKU, China · December 2024

The University of Hong Kong and ARC Lab, Tencent PCG, with the Chinese University of Hong Kong and UC Berkeley

Action policyOpen weights, Apache-2.0Also written Moto-GPT, Latent Motion Token, Moto
Parameters
98 M
The paper states 98M parameters for the GPT backbone, in the sentence "Despite having only 98M parameters for the GPT backbone".
GPU memory
0.2 GB
weights at bf16, computed. Fits an 8 GB card
Inference latency
not published
per action step
Weights
Apache-2.0
downloadable checkpoint

What it is

Moto attacks the data problem from the video side. A Latent Motion Tokenizer compresses each frame transition into 8 discrete tokens from a 128-entry codebook, learned entirely without action labels, and Moto-GPT then pre-trains by autoregressing over those tokens on 109k Open X-Embodiment videos. A co-fine-tuning stage attaches a small MLP action head and teaches the model to emit robot actions alongside motion tokens, which is how the video prior reaches real control. The result is a 98M-parameter backbone that matches RT-2-X on SIMPLER overall (61.4 versus 60.7) while being roughly 500 times smaller in backbone parameters. The failure mode is articulated objects: 43.1 percent on Open/Close Drawer, and only three real-world tasks with a 60 percent average.

Architecture

Backbone
Two-stage stack. A Latent Motion Tokenizer (frozen ViT encoder, learnable ViT decoder, M-Former with 8 learnable queries, VQ codebook of 128 codes) turns each frame transition into 8 discrete motion tokens without any action labels. Moto-GPT is then a GPT-style transformer with 12 layers, hidden size 768 and 12 heads, about 98M parameters, autoregressing over those motion tokens.
Action head
Two-layer MLP action head with hidden size 384, added during co-fine-tuning. The model predicts the next latent motion token and the continuous robot action jointly, which is the mechanism that transfers motion priors learned from action-free video into real control.
Parameters
The paper states 98M parameters for the GPT backbone, in the sentence "Despite having only 98M parameters for the GPT backbone". The Latent Motion Tokenizer (ViT encoder and decoder, M-Former) and the two-layer action head are additional and are not included in that figure, so the deployed system is larger than 98M.
Pretraining data
Motion-token autoregression on 109k trajectory videos from Open X-Embodiment, including RT-1 and Bridge, with no action labels used during pre-training. For the human-video experiments the authors add 105k filtered videos from Something-Something v2. The CALVIN experiments pre-train on 18k CALVIN trajectory videos with language annotations, which is 35 percent of the available data.
Embodiments
CALVIN simulation benchmark, SIMPLER, Google Everyday Robot (EDR) embodiment, A real robot arm in the authors' three-task suite

What hardware it needs

This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.

Computed from 98 M parameters. Weights only, so treat it as a floor: activations, image encoder intermediates and the CUDA context come on top, in practice 30 to 50 percent more for inference.
PrecisionWeightsNote
fp320.4 GBTraining master weights, rarely used for inference
bf16 or fp160.2 GBThe usual way these checkpoints are served
int80.1 GBQuantised, expect some loss of precision on fine motions
int40.0 GBAggressive quantisation, verify success rate before trusting it

Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.

The paper reports eight 40 GB GPUs for pre-training and four 40 GB GPUs for fine-tuning, with Moto-GPT trained for 10 epochs on the Open X-Embodiment data. The GPU model is not named beyond the 40 GB memory class. No inference VRAM figure and no latency figure is published.

The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.

Reported results

Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.

Simulation

Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.

  • SimplerEnv Google EDR, Pick Coke Can
    74%
    success rateAuthors' own evaluation. RT-2-X reaches 78.7 on the same task.source
  • SimplerEnv Google EDR embodiment, three tasks, overall
    61.4%
    success rateAuthors' own evaluation. Baselines in the same table: RT-2-X 60.7, RT-1-X 53.4, OpenVLA fine-tuned 34.9, Octo-Base 16.9. Moto uses static RGB only.source
  • SimplerEnv Google EDR, Open/Close Drawer
    43.1%
    success rateAuthors' own evaluation. Weakest of the three SIMPLER tasks for Moto, though above RT-2-X at 25.0. RT-1-X leads here at 59.7.source
  • CALVIN ABC to D, chains of 5 language instructions
    3.1 tasks
    average successful sequence lengthAuthors' own evaluation. Per-chain success: 0.897, 0.729, 0.601, 0.484, 0.386. Baselines in the same table: GR-1 3.06, SuSIE 2.69.source
  • CALVIN ABC to D, ablation without latent motion tokens
    2.14 tasks
    average successful sequence lengthAuthors' own ablation. The latent motion token pre-training is worth 0.96 tasks, which is the paper's core claim.source

Real world

Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.

  • Real-world (authors' own three-task suite) Real robot arm, average over three manipulation tasks
    60%
    success rateAuthors' own evaluation. The same setup without motion tokens reaches 23.33 percent.source

Fine tuned tasks

No results in this category are published for this model.

On real hardware

The authors report a real robot suite of three manipulation tasks, where Moto-GPT reaches a 60 percent average success rate against 23.33 percent for the same model trained without latent motion tokens. No per-task breakdown, no trial counts and no robot model are extracted from the sources here.

Fine tuning it yourself

Three artifacts are released on Hugging Face at TencentARC/Moto: the Latent Motion Tokenizer, the pre-trained Moto-GPT and the fine-tuned Moto-GPT. The intended path is co-fine-tuning, where the model predicts the next latent motion token and the robot action jointly, so the action-free motion prior is not destroyed by action supervision. The repository documents preprocessing for Open X-Embodiment and CALVIN data. The authors used four 40 GB GPUs for fine-tuning and eight for pre-training, with 10 epochs of pre-training on OXE.

Where it helps, where it does not

Strengths

  • Learns manipulation priors from action-free video, so the pre-training corpus is not limited to teleoperated robot data. The authors also pre-train on 105k human videos from Something-Something v2.
  • At 98M backbone parameters it matches RT-2-X (built on PaLI-X 55B) on SIMPLER overall (61.4 versus 60.7) and beats fine-tuned OpenVLA (Prismatic-7B) at 34.9, using static RGB only.
  • The ablations quantify the contribution cleanly: 2.14 to 3.10 average length on CALVIN and 23.33 to 60 percent on the real robot, both attributable to the latent motion tokens.
  • Apache-2.0 licensed code and checkpoints, with the tokenizer, the pre-trained Moto-GPT and the fine-tuned Moto-GPT all released on Hugging Face. ICCV 2025 Oral.

Limits

  • Weak where geometry and articulation matter: 43.1 percent on SIMPLER Open/Close Drawer, well below RT-1-X at 59.7 percent.
  • The 98M figure covers only the GPT backbone. The Latent Motion Tokenizer with its ViT encoder and decoder is not counted, so the deployed footprint is larger than the headline number.
  • No inference latency, no control frequency and no VRAM figure are published, so real-time feasibility has to be measured by the integrator.
  • The latent motion tokenizer uses only 128 codes and 8 tokens per frame transition, which is a hard information bottleneck on how much motion detail can be represented.
  • Real-world evidence is three tasks on the authors' own setup with a 60 percent average, so this is a research result rather than a deployment-ready policy.
  • Pre-training required eight 40 GB GPUs, which is above what a single-workstation lab typically has.

Sources

Everything on this page was taken from these documents. Where they disagree with what you read here, they win.

Entry last checked 2026-08-11.

Try a policy on a real arm

A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.