HPT-XLarge
MIT, United States · September 2024
MIT CSAIL and Meta FAIR
- Parameters
- 227 M
- 226.8M is the trunk parameter count from the paper's model-size table (depth 32, width 768, 16 heads).
- GPU memory
- 0.4 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- downloadable checkpoint
What it is
HPT-XLarge is the largest released trunk of the Heterogeneous Pre-trained Transformer family at 226.8M parameters, depth 32 and width 768. HPT separates a policy into a shareable, embodiment-agnostic trunk and small per-robot stems and heads, which is what makes it possible to pre-train one trunk across 52 heterogeneous datasets and 200k trajectories spanning real teleoperation, six simulators, deployed robots and human video. On the one real task the paper reports numerically, transferring this trunk lifts success from 43.3 percent (from scratch) to 76.7 percent, and it beats R3M, Voltron and VC-1 representation baselines. The catch for anyone evaluating it seriously is that this single table is essentially the whole published numeric evidence, since the simulation results and the other real tasks live inside figures.
Architecture
- Backbone
- Shared transformer trunk of depth 32, width 768 and 16 attention heads, the largest released size. Embodiment-specific stems align proprioception and vision into a short token sequence, using a frozen pretrained ResNet-18 image encoder. No language input in this variant.
- Action head
- Task and embodiment specific action head on top of the shared trunk. Only the trunk transfers between embodiments.
- Parameters
- 226.8M is the trunk parameter count from the paper's model-size table (depth 32, width 768, 16 heads). The frozen ResNet-18 encoder, the stems and the head are additional. The same table lists HPT-Huge at 1.1B trunk parameters (depth 80, width 1024, 16 heads) as part of the scaling study, but HPT-Huge is not among the released checkpoints.
- Pretraining data
- Default setting: 27 robot teleoperation datasets including a subset of Open X-Embodiment, capped at 1000 trajectories per dataset for 16k trajectories in total. Scaled setting: 52 datasets and 200k trajectories, covering 42 Open X-Embodiment datasets including DROID, simulation sources (Drake, MuJoCo, Isaac Sim, PyBullet, Sapien, Flex), deployed robots (FrodoBot) and human video (EPIC-Kitchens, PoCo).
- Embodiments
- Cross-embodiment by design, with a stem per embodiment, Transfer evaluated on 4 simulation benchmarks and on real hardware across 2 embodiments, 45 trials per approach
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.8 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.4 GB | The usual way these checkpoints are served |
| int8 | 0.2 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.1 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
The paper states neither GPU type nor GPU count for pre-training. It reports batch size 256 for 80k iterations (around 0.65B tokens in the latent space, around 5B in the vision and proprioception token spaces) for the default setting and batch size 2048 with 170k trajectories for the scaled experiments. At 226.8M trunk parameters this is the largest released variant.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Real-world (authors' own setup) Sweep Leftover, fine-tuned from the pre-trained trunk76.7%success rateAuthors' own evaluation, reported as 76.7 plus or minus 3.3. Same policy from scratch: 43.3 plus or minus 3.8. VC-1: 53.3 plus or minus 2.6. R3M: 50.0 plus or minus 3.0. Voltron: 46.7 plus or minus 3.8. HPT-Base on the same task: 70.0 plus or minus 3.0.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
On the real-world Sweep Leftover task the authors report 76.7 plus or minus 3.3 percent for the fine-tuned HPT-XLarge, against 70.0 plus or minus 3.0 percent for HPT-Base, 63.3 plus or minus 2.6 percent for a fine-tuned model without proprioception, 53.3 plus or minus 2.6 percent for VC-1, 50.0 plus or minus 3.0 percent for R3M, 46.7 plus or minus 3.8 percent for Voltron, 43.3 plus or minus 3.8 percent from scratch and 26.7 plus or minus 3.3 percent from scratch without proprioception. A further figure evaluates four tasks over two embodiments with 45 trials per approach, but reports those results graphically only.
Fine tuning it yourself
Download the trunk from huggingface.co/liruiw/hpt-xlarge, write an embodiment-specific stem that tokenizes your robot's proprioception and camera views plus an action head for your action space, then train stem and head on your demonstrations while transferring the trunk. The paper's setup uses one camera view with a frozen pretrained ResNet-18. Quickstart notebook in the repository, LeRobot integration at github.com/liruiw/lerobot.
Where it helps, where it does not
Strengths
- Best measured HPT result: 76.7 percent on real Sweep Leftover, 33.4 points above the same policy trained from scratch and 23.4 points above the VC-1 encoder baseline.
- Scaling from HPT-Base to HPT-XLarge buys 6.7 points on the same real task, so the size ladder does pay off, though with diminishing returns per parameter.
- Still only 226.8M trunk parameters, which is an order of magnitude smaller than VLA models built on 7B backbones, so it does not require datacenter hardware.
- MIT licensed, released on Hugging Face, with a quickstart notebook and a LeRobot fork.
Limits
- Only one numeric benchmark table exists in the paper (real Sweep Leftover). The four simulation benchmarks and the remaining three real tasks are reported only inside figures, so no further numbers can be quoted.
- The paper's headline claim of over 20 percent improvement on unseen tasks is stated in the abstract without a corresponding explicit numeric breakdown in the text.
- The 226.8M figure is trunk only and understates the deployed footprint.
- No language conditioning, and no language-conditioned variant is released at this size. Only HPT-Base (With Language) at 50.6M exists.
- No latency, no VRAM, no control-frequency and no GPU-count figures are published anywhere in the paper or repository, so real-time budgets have to be measured, not looked up.
- Every result comes from one lab. There is no cross-lab replication of the transfer claim.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2409.20537
- https://arxiv.org/html/2409.20537v1
- https://github.com/liruiw/HPT
- https://huggingface.co/liruiw/hpt-xlarge
- https://liruiw.github.io/hpt/
- https://proceedings.neurips.cc//paper_files/paper/2024/hash/e0f393e7980a24fd12fa6f15adfa25fb-Abstract-Conference.html
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.