Moto (Moto-GPT)
HKU, China · December 2024
The University of Hong Kong and ARC Lab, Tencent PCG, with the Chinese University of Hong Kong and UC Berkeley
- Parameters
- 98 M
- The paper states 98M parameters for the GPT backbone, in the sentence "Despite having only 98M parameters for the GPT backbone".
- GPU memory
- 0.2 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- Apache-2.0
- downloadable checkpoint
What it is
Moto attacks the data problem from the video side. A Latent Motion Tokenizer compresses each frame transition into 8 discrete tokens from a 128-entry codebook, learned entirely without action labels, and Moto-GPT then pre-trains by autoregressing over those tokens on 109k Open X-Embodiment videos. A co-fine-tuning stage attaches a small MLP action head and teaches the model to emit robot actions alongside motion tokens, which is how the video prior reaches real control. The result is a 98M-parameter backbone that matches RT-2-X on SIMPLER overall (61.4 versus 60.7) while being roughly 500 times smaller in backbone parameters. The failure mode is articulated objects: 43.1 percent on Open/Close Drawer, and only three real-world tasks with a 60 percent average.
Architecture
- Backbone
- Two-stage stack. A Latent Motion Tokenizer (frozen ViT encoder, learnable ViT decoder, M-Former with 8 learnable queries, VQ codebook of 128 codes) turns each frame transition into 8 discrete motion tokens without any action labels. Moto-GPT is then a GPT-style transformer with 12 layers, hidden size 768 and 12 heads, about 98M parameters, autoregressing over those motion tokens.
- Action head
- Two-layer MLP action head with hidden size 384, added during co-fine-tuning. The model predicts the next latent motion token and the continuous robot action jointly, which is the mechanism that transfers motion priors learned from action-free video into real control.
- Parameters
- The paper states 98M parameters for the GPT backbone, in the sentence "Despite having only 98M parameters for the GPT backbone". The Latent Motion Tokenizer (ViT encoder and decoder, M-Former) and the two-layer action head are additional and are not included in that figure, so the deployed system is larger than 98M.
- Pretraining data
- Motion-token autoregression on 109k trajectory videos from Open X-Embodiment, including RT-1 and Bridge, with no action labels used during pre-training. For the human-video experiments the authors add 105k filtered videos from Something-Something v2. The CALVIN experiments pre-train on 18k CALVIN trajectory videos with language annotations, which is 35 percent of the available data.
- Embodiments
- CALVIN simulation benchmark, SIMPLER, Google Everyday Robot (EDR) embodiment, A real robot arm in the authors' three-task suite
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.4 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.2 GB | The usual way these checkpoints are served |
| int8 | 0.1 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.0 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
The paper reports eight 40 GB GPUs for pre-training and four 40 GB GPUs for fine-tuning, with Moto-GPT trained for 10 epochs on the Open X-Embodiment data. The GPU model is not named beyond the 40 GB memory class. No inference VRAM figure and no latency figure is published.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- SimplerEnv Google EDR, Pick Coke Can74%
- SimplerEnv Google EDR embodiment, three tasks, overall61.4%success rateAuthors' own evaluation. Baselines in the same table: RT-2-X 60.7, RT-1-X 53.4, OpenVLA fine-tuned 34.9, Octo-Base 16.9. Moto uses static RGB only.source
- SimplerEnv Google EDR, Open/Close Drawer43.1%success rateAuthors' own evaluation. Weakest of the three SIMPLER tasks for Moto, though above RT-2-X at 25.0. RT-1-X leads here at 59.7.source
- CALVIN ABC to D, chains of 5 language instructions3.1 tasksaverage successful sequence lengthAuthors' own evaluation. Per-chain success: 0.897, 0.729, 0.601, 0.484, 0.386. Baselines in the same table: GR-1 3.06, SuSIE 2.69.source
- CALVIN ABC to D, ablation without latent motion tokens2.14 tasksaverage successful sequence lengthAuthors' own ablation. The latent motion token pre-training is worth 0.96 tasks, which is the paper's core claim.source
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Real-world (authors' own three-task suite) Real robot arm, average over three manipulation tasks60%success rateAuthors' own evaluation. The same setup without motion tokens reaches 23.33 percent.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
The authors report a real robot suite of three manipulation tasks, where Moto-GPT reaches a 60 percent average success rate against 23.33 percent for the same model trained without latent motion tokens. No per-task breakdown, no trial counts and no robot model are extracted from the sources here.
Fine tuning it yourself
Three artifacts are released on Hugging Face at TencentARC/Moto: the Latent Motion Tokenizer, the pre-trained Moto-GPT and the fine-tuned Moto-GPT. The intended path is co-fine-tuning, where the model predicts the next latent motion token and the robot action jointly, so the action-free motion prior is not destroyed by action supervision. The repository documents preprocessing for Open X-Embodiment and CALVIN data. The authors used four 40 GB GPUs for fine-tuning and eight for pre-training, with 10 epochs of pre-training on OXE.
Where it helps, where it does not
Strengths
- Learns manipulation priors from action-free video, so the pre-training corpus is not limited to teleoperated robot data. The authors also pre-train on 105k human videos from Something-Something v2.
- At 98M backbone parameters it matches RT-2-X (built on PaLI-X 55B) on SIMPLER overall (61.4 versus 60.7) and beats fine-tuned OpenVLA (Prismatic-7B) at 34.9, using static RGB only.
- The ablations quantify the contribution cleanly: 2.14 to 3.10 average length on CALVIN and 23.33 to 60 percent on the real robot, both attributable to the latent motion tokens.
- Apache-2.0 licensed code and checkpoints, with the tokenizer, the pre-trained Moto-GPT and the fine-tuned Moto-GPT all released on Hugging Face. ICCV 2025 Oral.
Limits
- Weak where geometry and articulation matter: 43.1 percent on SIMPLER Open/Close Drawer, well below RT-1-X at 59.7 percent.
- The 98M figure covers only the GPT backbone. The Latent Motion Tokenizer with its ViT encoder and decoder is not counted, so the deployed footprint is larger than the headline number.
- No inference latency, no control frequency and no VRAM figure are published, so real-time feasibility has to be measured by the integrator.
- The latent motion tokenizer uses only 128 codes and 8 tokens per frame transition, which is a hard information bottleneck on how much motion detail can be represented.
- Real-world evidence is three tasks on the authors' own setup with a 60 percent average, so this is a research result rather than a deployment-ready policy.
- Pre-training required eight 40 GB GPUs, which is above what a single-workstation lab typically has.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2412.04445
- https://arxiv.org/html/2412.04445v4
- https://github.com/TencentARC/Moto
- https://huggingface.co/TencentARC/Moto
- https://github.com/TencentARC/Moto/blob/main/LICENSE.txt
- https://chenyi99.github.io/moto/
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.