HPT-Base
MIT, United States · September 2024
MIT CSAIL and Meta FAIR
- Parameters
- 13 M
- 12.6M is the trunk parameter count from the paper's model-size table (depth 16, width 256, 8 heads).
- GPU memory
- 0.0 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- downloadable checkpoint
What it is
HPT-Base is the default released trunk of the Heterogeneous Pre-trained Transformer family, 12.6M parameters at depth 16 and width 256. The idea is that the reusable part of a robot policy is a shared trunk operating on a short token sequence, while the robot-specific parts are a small input stem and a small action head, which is what lets the same trunk be pre-trained across 52 heterogeneous datasets and 200k trajectories from real robots, six simulators and human video. On the real Sweep Leftover task, transferring this trunk raises success from 43.3 percent (from scratch) to 70.0 percent, ahead of R3M and VC-1 encoders. The weakness is evidence coverage: only one real task is reported as a numeric table, and everything else in the paper sits inside figures.
Architecture
- Backbone
- Shared transformer trunk of depth 16, width 256 and 8 attention heads. Embodiment-specific stems align proprioception and vision into a short token sequence, using a frozen pretrained ResNet-18 image encoder. The authors state they also tried MAE ViT-Base, DINOv2 and CLIP ViT-Base as encoders and chose ResNet for simplicity and common usage.
- Action head
- Task and embodiment specific action head on top of the shared trunk. Only the trunk transfers.
- Parameters
- 12.6M is the trunk parameter count from the paper's model-size table (depth 16, width 256, 8 heads). The frozen ResNet-18 encoder, the stems and the head are additional. A separate variant with language conditioning, HPT-Base (With Language), is released at 50.6M parameters.
- Pretraining data
- Default setting: 27 robot teleoperation datasets including a subset of Open X-Embodiment, capped at 1000 trajectories per dataset for 16k trajectories in total. Scaled setting: 52 datasets and 200k trajectories, covering 42 Open X-Embodiment datasets including DROID, simulation sources (Drake, MuJoCo, Isaac Sim, PyBullet, Sapien, Flex), deployed robots (FrodoBot) and human video (EPIC-Kitchens, PoCo).
- Embodiments
- Cross-embodiment by design, with a stem per embodiment, Transfer evaluated on 4 simulation benchmarks and on real hardware across 2 embodiments, 45 trials per approach
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.0 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.0 GB | The usual way these checkpoints are served |
| int8 | 0.0 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.0 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
The paper states neither GPU type nor GPU count for pre-training. It reports batch size 256 for 80k iterations (around 0.65B tokens in the latent space, around 5B tokens in the vision and proprioception token spaces) for the default setting and batch size 2048 for the scaled experiments. At 12.6M trunk parameters this variant is small enough for a single consumer GPU.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Real-world (authors' own setup) Sweep Leftover, fine-tuned from the pre-trained trunk70%success rateAuthors' own evaluation, reported as 70.0 plus or minus 3.0. Same policy trained from scratch: 43.3 plus or minus 3.8. VC-1 encoder baseline: 53.3 plus or minus 2.6. R3M: 50.0 plus or minus 3.0.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
On the real-world Sweep Leftover task the authors report 70.0 plus or minus 3.0 percent for the fine-tuned HPT-Base against 43.3 plus or minus 3.8 percent for the same policy trained from scratch, 26.7 plus or minus 3.3 percent for a from-scratch model without proprioception, 50.0 plus or minus 3.0 percent for R3M, 46.7 plus or minus 3.8 percent for Voltron and 53.3 plus or minus 2.6 percent for VC-1. A further figure covers four tasks across two embodiments with 45 trials per approach, but publishes those numbers only graphically.
Fine tuning it yourself
Download the trunk from huggingface.co/liruiw/hpt-base, write an embodiment-specific stem that tokenizes your robot's proprioception and camera views, and an action head for your action space, then train stem and head on your demonstrations while transferring the trunk. The vision encoder is a frozen pretrained ResNet-18 with one camera view in the paper's setup. The repository ships a quickstart notebook, and the authors maintain a LeRobot fork at github.com/liruiw/lerobot.
Where it helps, where it does not
Strengths
- Turns the pre-training benefit into a measured number: 70.0 percent versus 43.3 percent from scratch on the real Sweep Leftover task, which is a 26.7 point gap for a 12.6M trunk.
- Beats representation-learning baselines that use much larger encoders, specifically VC-1 at 53.3 percent and R3M at 50.0 percent on the same task.
- Only 12.6M trunk parameters, so it fits comfortably on a single consumer GPU and does not need a server for inference.
- MIT licensed, on Hugging Face, with a quickstart notebook and a LeRobot fork for integration.
- A language-conditioned variant (50.6M) is released for tasks that need instruction following.
Limits
- The published transfer evidence is thin: one real-world task with a numeric table (Sweep Leftover). The other three real tasks and all four simulation benchmarks appear only in figures, so their per-task numbers cannot be quoted.
- The paper's headline claim of over 20 percent improvement on unseen tasks is not backed by a single explicit numeric comparison in the text beyond the Sweep Leftover table.
- No language input in the default model. Instruction following requires the separate hpt-base-lang checkpoint.
- The 12.6M figure is trunk only and understates the deployed footprint, which also includes the frozen ResNet-18 encoder, the stem and the head.
- No latency, no VRAM, no control-frequency and no GPU-count figures are published anywhere.
- The pre-training mixture caps each dataset at 1000 trajectories in the default setting, so a target robot that is underrepresented in Open X-Embodiment gets little relevant prior.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2409.20537
- https://arxiv.org/html/2409.20537v1
- https://github.com/liruiw/HPT
- https://huggingface.co/liruiw/hpt-base
- https://liruiw.github.io/hpt/
- https://proceedings.neurips.cc//paper_files/paper/2024/hash/e0f393e7980a24fd12fa6f15adfa25fb-Abstract-Conference.html
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.