HPT-Small
MIT, United States · September 2024
MIT CSAIL and Meta FAIR
- Parameters
- 3 M
- 3.1M is the trunk parameter count from the paper's model-size table (depth 16, width 128, 8 heads).
- GPU memory
- 0.0 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- downloadable checkpoint
What it is
HPT-Small is the smallest released trunk of the Heterogeneous Pre-trained Transformer family, 3.1M parameters at depth 16 and width 128. HPT's premise is that the expensive, shareable part of a policy is a task and embodiment agnostic trunk, while the parts that must differ per robot are a small input stem and a small action head. Pre-training runs across up to 52 heterogeneous datasets and 200k trajectories mixing real teleoperation, six simulators, deployed robots and human video. For this specific size the paper publishes no transfer result, so it should be treated as a cheap probe rather than as a validated policy.
Architecture
- Backbone
- Shared transformer trunk of depth 16, width 128 and 8 attention heads. Embodiment-specific stems align proprioception and vision into a short token sequence, using a frozen pretrained ResNet-18 image encoder for the vision features. No language input in this variant.
- Action head
- Task and embodiment specific action head on top of the shared trunk. The trunk is what transfers; the stem and the head are trained per embodiment and per task.
- Parameters
- 3.1M is the trunk parameter count from the paper's model-size table (depth 16, width 128, 8 heads). The frozen ResNet-18 vision encoder, the embodiment-specific stems and the action head are additional and are not included. The same table also lists HPT-Huge at 1.1B trunk parameters (depth 80, width 1024, 16 heads), which is not among the released checkpoints.
- Pretraining data
- Default setting: 27 robot teleoperation datasets including a subset of Open X-Embodiment, capped at 1000 trajectories per dataset for 16k trajectories in total. Scaled setting: 52 datasets and 200k trajectories, covering 42 Open X-Embodiment datasets including DROID, simulation sources (Drake, MuJoCo, Isaac Sim, PyBullet, Sapien, Flex), deployed robots (FrodoBot) and human video (EPIC-Kitchens, PoCo).
- Embodiments
- Cross-embodiment by design. Pre-training spans 52 datasets from real teleoperation, six simulators, deployed robots and human video, each attached through its own stem, Transfer evaluated on 4 simulation benchmarks and on real hardware across 2 embodiments
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.0 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.0 GB | The usual way these checkpoints are served |
| int8 | 0.0 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.0 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
The paper states neither GPU type nor GPU count for pre-training. It reports a batch size of 256 for 80k iterations in the default setting, which it describes as around 0.65B tokens in the latent space and around 5B tokens in the vision and proprioception token spaces, and a batch size of 2048 for the scaled experiments. At 3.1M trunk parameters this is the smallest released variant and the cheapest to run.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
No real-world result is published for HPT-Small. The paper's real-world Table 3 (Sweep Leftover) reports only HPT-Base and HPT-XLarge.
Fine tuning it yourself
Download the trunk from huggingface.co/liruiw/hpt-small, then write an embodiment-specific stem (proprioception and vision tokenizer) and an action head for the target robot and train those on your demonstrations while transferring the trunk. The repository ships a quickstart notebook and the authors provide a LeRobot fork at github.com/liruiw/lerobot for integration.
Where it helps, where it does not
Strengths
- Smallest released HPT trunk at 3.1M parameters, so it is the cheapest option for testing whether heterogeneous pre-training helps on a given robot before committing to a larger trunk.
- MIT licensed, released on Hugging Face, with a quickstart notebook and a LeRobot integration fork.
- Trunk transfer only requires writing a stem and a head for the target robot, so a new embodiment does not need a new pre-training run.
Limits
- The paper reports no benchmark result specifically for HPT-Small. All published transfer numbers are for HPT-Base and HPT-XLarge, so this size is unproven.
- The 3.1M figure is trunk only. It does not include the frozen ResNet-18 encoder, the stems or the head, so it is not a deployable model size.
- No language conditioning in this variant. A separate HPT-Base with language (50.6M) is released for that.
- No latency, no VRAM and no control-frequency figures are published anywhere in the paper or repository.
- The paper's simulation transfer results appear only inside figures, not as tables, so per-task simulation numbers cannot be quoted.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2409.20537
- https://arxiv.org/html/2409.20537v1
- https://github.com/liruiw/HPT
- https://huggingface.co/liruiw/hpt-small
- https://liruiw.github.io/hpt/
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.