villa-X
Microsoft, United States · July 2025
Microsoft Research
- Parameters
- not published
- No total is stated.
- GPU memory
- not published
- weights at bf16, computed
- Inference latency
- not published
- per action step
- Weights
- closed
- MIT on the code repository and on the released Latent Action Model.
What it is
villa-X inserts an intermediate representation between language and joint commands: a latent action, learned as an abstract description of the motion between two frames, which the policy plans first and only then decodes into 4 robot actions. This lets human egocentric video contribute to pretraining alongside 1.6M robot trajectories, and the ablation shows the mechanism carries the result, since removing latent actions drops the Google Robot average from 77.7 to 36.5 percent. It reports 90.1 percent average across all four LIBERO suites including the hard Long suite at 74.5 percent, which is a stronger claim than three suite averages reported elsewhere. The decisive practical problem is that the actor model is not released. Only the 955M Latent Action Model is downloadable, the release plan still marks the actor as pending, and there has been no commit since the initial release on 1 August 2025, so villa-X cannot currently be run as a policy. Its SIMPLER Google Robot figure also depends on which tasks are counted: 77.7 percent over three tasks in the paper table, 59.6 percent over four on the project page once the Place task, where it scores 5.6 percent, is included.
Architecture
- Backbone
- PaliGemma 3B, pretrained with 224 by 224 images and 128 token text inputs
- Action head
- Two separate experts attached to the backbone, a latent action expert and a robot action expert, each an 18 layer transformer with hidden dimension 1024, 8 attention heads and about 300M parameters, trained from scratch. The model first predicts a latent action plan of length 6, then decodes 4 robot actions.
- Action chunk
- 4 steps per forward pass
- Parameters
- No total is stated. The paper names a PaliGemma 3B backbone and says each of the two experts contains approximately 300M parameters. The separately released Latent Action Model is stated as 955M parameters in the repository README and its Hugging Face file is 0.95 GB.
- Pretraining data
- 1.6M robot trajectories with 223.5M frames from an Open X-Embodiment mixture plus AgiBot (about 20 percent of the mix), together with 3.6M human video clips from Ego4D (21.46 percent), EPIC-KITCHENS (6.95 percent), Something-Something V2 (6.82 percent) and a combined roughly 15 percent from EgoPAT3D, EGTEA Gaze+, HO-Cap, HOI4D, HoloAssist and RH20T.
- Embodiments
- Google Robot (SIMPLER simulation), WidowX / Bridge (SIMPLER simulation), LIBERO (simulation), Realman RM75 with Inspire gripper (real hardware), xArm 7-DoF with XHand 12-DoF dexterous hand (real hardware)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Checkpoint download | 0.95 GB |
No inference or fine-tuning GPU requirement is published. The only compute figures are for pretraining, 128 A100 GPUs for the latent action model and 64 A100 GPUs for the policy.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- LIBERO average over all four suites (Spatial, Object, Goal, Long)90.1%success rateIncludes the Long suite, unlike some other four-value LIBERO averages. Baselines: pi0-FAST 85.5, pi0 81.1. Ablation without latent actions 81.9.source
- SIMPLER Google Robot, visual matching, average over 3 tasks (Pick 98.7, Move 75.0, Drawer 59.3)77.7%success rateThe paper table omits the Place task. Baselines in the same table: RoboVLMs 60.8, GR00T-N1.5 57.9, OpenVLA 32.7, Octo-base 14.6. Ablation without latent actions drops to 36.5.source
- LIBERO Long74.5%
- SIMPLER WidowX / Bridge, visual matching, average over 4 tasks62.5%success rateConsistent between paper and project page. Baselines: GR00T-N1.5 62.0, RoboVLMs 37.5, Octo-base 16.0, OpenVLA 1.0.source
- SIMPLER Google Robot, visual matching, average over 4 tasks (Pick, Move, Drawer, Place)59.6%success rateOfficial project page table. The Place task alone scores only 5.6 percent, which is what pulls the average down. Use this number when comparing against other four task Google Robot averages.source
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
Two setups. A Realman RM75 arm with an Inspire gripper (6-DoF plus 1-DoF gripper) fine-tuned on 375 trajectories across five tasks, with per task success from 30 percent (pick-in) and 50 percent (push, stack) up to 100 percent (pick-out, unstack). An xArm 7-DoF with a 12-DoF XHand dexterous hand fine-tuned on 4000 trajectories across five tasks, reaching 84 percent on seen pick and place versus 68 percent unseen, 75 versus 50 percent on stacking a cube, 60 versus 30 percent on placing a cup upright, 50 versus 40 percent on pouring water and 50 versus 40 percent on flicking a ball. Real robot data was collected at 10 Hz.
Fine tuning it yourself
The published training recipe covers pretraining only: the latent action model on 128 NVIDIA A100 GPUs for about four days at batch size 512 and learning rate 1.5e-4, then policy pretraining on 64 NVIDIA A100 GPUs for four days. Fine-tuning used 375 trajectories for the Realman setup and 4000 for the dexterous hand setup. Since the actor model is unreleased, only the latent action model and its inference code can currently be used.
Where it helps, where it does not
Strengths
- Evaluated on all four LIBERO suites including Long, at 74.5 percent, rather than reporting a three suite average
- Latent action pretraining lets human video contribute, and the ablation quantifies its value at over 40 points on Google Robot
- Real hardware evidence on a 12 DoF dexterous hand, not only on parallel grippers
- MIT licensed with a genuinely large and precisely documented pretraining corpus, 1.6M trajectories and 223.5M frames
- Accepted as an ICLR 2026 poster
Limits
- The actor model is not released. Only the 955M Latent Action Model is downloadable and the repository release plan still marks the actor as pending, with no commit since 1 August 2025.
- The SIMPLER Google Robot headline depends on the task set: 77.7 percent over three tasks in the paper table versus 59.6 percent over four tasks on the project page, where the Place task scores only 5.6 percent.
- No inference VRAM, no latency and no policy control frequency are published. Only training cost is given, 128 A100 GPUs for about four days for the latent action model and 64 A100 GPUs for four days for policy pretraining.
- Dexterous hand performance drops sharply on unseen objects, for example 84 to 68 percent on pick and place and 60 to 30 percent on placing a cup upright.
- Short action chunk of 4 robot actions per plan, which means frequent policy queries at an undocumented cost.
- Real robot fine-tuning used 375 and 4000 trajectories respectively, so the reported numbers assume substantial per setup data collection.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2507.23682
- https://arxiv.org/html/2507.23682v3
- https://microsoft.github.io/villa-x/
- https://github.com/microsoft/villa-x
- https://huggingface.co/microsoft/villa-x
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.