3D-VLA
UMass Amherst, United States · March 2024
UMass Amherst Embodied AGI group with collaborators
- Parameters
- not published
- No parameter count is stated anywhere in the paper, the README or the model cards.
- GPU memory
- not published
- weights at bf16, computed
- Inference latency
- not published
- per action step
- Weights
- closed
- The GitHub repository has no LICENSE file, so the code carries no explicit usage grant.
What it is
3D-VLA is the oldest model in this comparison and the only one that is genuinely a world model rather than a policy: it takes 3D input, imagines the goal state as an image, a depth map or a point cloud, and plans against that imagination instead of mapping pixels straight to actions. The reasoning side works well, with BLEU-4 rising from 10.11 to 26.80 on embodied question answering over the BLIP-2 baseline, and goal generation is the part with released checkpoints. Action planning is where it stops: on CALVIN it completes 44.7 percent of first tasks but 0 percent of five task sequences, and the RLBench results range from 68 percent on putting away a knife down to 24 percent on an unseen cup. For anyone shopping for a deployable policy the deciding facts are that the VLA weights themselves were never published, only the three goal generation diffusion models, the repository has been dormant since October 2024 and carries no LICENSE file, and 3D input requires a depth camera and a reconstruction pipeline that 2D VLAs do not need. Note that a separate 2026 paper (arXiv 2605.29416) also calls itself 3DVLA and is unrelated to this work.
Architecture
- Backbone
- A 3D-based large language model built on BLIP-2 FlanT5-XL, extended with interaction tokens for scene, object and action, plus a set of embodied diffusion models aligned into the LLM for predicting goal images, goal depth and goal point clouds
- Action head
- Interaction tokens generated autoregressively by the 3D LLM, with the aligned diffusion models supplying the imagined goal state that conditions planning. There is no continuous action expert.
- Parameters
- No parameter count is stated anywhere in the paper, the README or the model cards. The backbone is identified only as BLIP-2 FlanT5-XL with unfrozen embeddings and Q-Former weights.
- Pretraining data
- A curated 3D Embodied Instruction Tuning Dataset of about 2M 3D language action data pairs derived from 316K episodes, drawn from 12 robot manipulation datasets plus human object interaction sources including Epic-Kitchens and HOI4D, with RLBench and Hiveformer used for downstream training.
- Embodiments
- RLBench (simulation), CALVIN (simulation)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
No inference or minimum GPU requirement is published. The only compute figures are 6 by 32 V100 GPUs for pretraining and 6 by 64 V100 GPUs for alignment.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- RLBench Put Knife68%success rateBest of the reported RLBench tasks. Other tasks in the same table: Take Umbrella 52, Pick up Cup 40.source
- CALVIN train on scenes A, B, C, D and test on scene D, 1 task completed in a row44.7%
- RLBench Pick up Cup, unseen24%success rateDrops from 40 percent on the seen version, which quantifies the generalisation gap.source
- CALVIN train on scenes A, B, C, D and test on scene D, 5 tasks completed in a row0%success rateIntermediate values in the same row: 2 tasks 16.3, 3 tasks 8.1, 4 tasks 1.6. MCIL is 0 from 4 tasks onward.source
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Embodied QA (authors' 3D embodied instruction benchmark) held-out26.8 score
- Goal RGB image generation held-out17.21 dBPSNRSame table reports CLIP similarity 0.920, SSIM 0.636 and FID 0.177. This is the component whose weights are actually released.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
No real robot evaluation is reported. All action planning results come from the RLBench and CALVIN simulators.
Fine tuning it yourself
Training uses the 3D Embodied Instruction Tuning Dataset with resources from Open X-Embodiment, RLBench and Hiveformer. Pretraining ran for 30 epochs on 6 by 32 V100 GPUs and the alignment stage for 20 epochs on 6 by 64 V100 GPUs. The published training scripts take GPU and node counts as parameters without stating minimum requirements, and the VLA weights are not released, so a run has to start from the pretraining code.
Where it helps, where it does not
Strengths
- Genuine 3D input and an explicit imagined goal state, which makes the plan inspectable rather than opaque
- Large measured gains on embodied reasoning, BLEU-4 26.80 versus 10.11 for the BLIP-2 FlanT5-XL baseline
- The 3D Embodied Instruction Tuning Dataset with about 2M pairs from 316K episodes is a substantial published contribution
- Goal image, goal depth and goal point cloud diffusion checkpoints are published under MIT and are useful on their own
- Published at ICML 2024, so the results went through peer review
Limits
- Action planning collapses over a horizon: 44.7 percent on one CALVIN task in a row, 16.3 percent on two, 8.1 percent on three, 1.6 percent on four and 0 percent on five.
- The 3D-VLA policy weights were never released. Only the three goal generation diffusion models are on Hugging Face.
- The GitHub repository has no LICENSE file, so the code has no explicit usage grant even though the diffusion checkpoints are tagged MIT.
- Requires 3D input, meaning a depth sensor and a point cloud reconstruction pipeline that 2D VLAs do not need.
- No parameter count, no VRAM, no latency and no control frequency are published anywhere. The only compute figures are training on 6 by 32 and 6 by 64 V100 GPUs.
- Repository dormant since October 2024 with no successor. A different, unrelated 2026 paper (arXiv 2605.29416) shares the name 3DVLA, which is a common source of confusion.
- Evaluated only in simulation (RLBench, CALVIN). No real robot results are reported.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2403.09631
- https://arxiv.org/html/2403.09631v1
- https://github.com/UMass-Embodied-AGI/3D-VLA
- https://huggingface.co/anyeZHY/3dvla-diffusion
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.