HPT-Large
MIT, United States · September 2024
MIT CSAIL and Meta FAIR
- Parameters
- 51 M
- 50.5M is the trunk parameter count from the paper's model-size table (depth 16, width 512, 8 heads).
- GPU memory
- 0.1 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- downloadable checkpoint
What it is
HPT-Large is the 50.5M-parameter trunk in the Heterogeneous Pre-trained Transformer size ladder, depth 16 and width 512. It shares the architecture and the pre-training mixture of the rest of the family: a shared trunk pre-trained across up to 52 heterogeneous datasets and 200k trajectories, with per-embodiment stems and heads carrying the robot-specific parts. The paper publishes no transfer number for this size specifically, so it is an interpolation point between the measured HPT-Base result (70.0 percent on real Sweep Leftover) and the measured HPT-XLarge result (76.7 percent). Choose it when the base trunk underfits and the XLarge trunk does not fit the deployment budget.
Architecture
- Backbone
- Shared transformer trunk of depth 16, width 512 and 8 attention heads. Embodiment-specific stems align proprioception and vision into a short token sequence, using a frozen pretrained ResNet-18 image encoder. No language input in this variant.
- Action head
- Task and embodiment specific action head on top of the shared trunk. Only the trunk transfers.
- Parameters
- 50.5M is the trunk parameter count from the paper's model-size table (depth 16, width 512, 8 heads). The frozen ResNet-18 encoder, the stems and the head are additional. Note that HPT-Base (With Language) is a different model at 50.6M parameters despite the nearly identical count.
- Pretraining data
- Default setting: 27 robot teleoperation datasets including a subset of Open X-Embodiment, capped at 1000 trajectories per dataset for 16k trajectories in total. Scaled setting: 52 datasets and 200k trajectories, covering 42 Open X-Embodiment datasets including DROID, simulation sources (Drake, MuJoCo, Isaac Sim, PyBullet, Sapien, Flex), deployed robots (FrodoBot) and human video (EPIC-Kitchens, PoCo).
- Embodiments
- Cross-embodiment by design, with a stem per embodiment, Transfer evaluated on 4 simulation benchmarks and on real hardware across 2 embodiments
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.2 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.1 GB | The usual way these checkpoints are served |
| int8 | 0.0 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.0 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
The paper states neither GPU type nor GPU count. It reports batch size 256 for 80k iterations in the default setting and batch size 2048 for the scaled experiments. At 50.5M trunk parameters this variant still fits a single consumer GPU.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
No real-world result is published for HPT-Large. The paper's real-world Table 3 (Sweep Leftover) reports only HPT-Base and HPT-XLarge.
Fine tuning it yourself
Download the trunk from huggingface.co/liruiw/hpt-large, then write an embodiment-specific stem and an action head for your robot and train those on your demonstrations while transferring the trunk. Quickstart notebook in the repository, LeRobot integration at github.com/liruiw/lerobot.
Where it helps, where it does not
Strengths
- Sits between HPT-Base and HPT-XLarge, so it is the option when the 12.6M trunk underfits but the 226.8M trunk is too heavy for the target hardware.
- The paper's scaling study across 52 datasets is what motivates the size ladder, and Figure 10 evaluates transfer from HPT-B through HPT-XL across four simulation benchmarks.
- MIT licensed and released on Hugging Face with the rest of the family.
Limits
- No benchmark result is published specifically for HPT-Large. The paper's numeric tables cover only HPT-Base and HPT-XLarge, so this size sits between two measured points without a measurement of its own.
- The 50.5M figure is trunk only and does not include the frozen ResNet-18 encoder, the stems or the head.
- No language conditioning. Use the separate hpt-base-lang checkpoint if instructions are needed.
- No latency, no VRAM, no control-frequency and no GPU-count figures are published.
- Easily confused with HPT-Base (With Language) at 50.6M, which is a different model with a different capability.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2409.20537
- https://arxiv.org/html/2409.20537v1
- https://github.com/liruiw/HPT
- https://huggingface.co/liruiw/hpt-large
- https://liruiw.github.io/hpt/
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.