ChatVLA
Midea Group, China · February 2025
Midea Group with East China Normal University and partners
- Parameters
- 3.4 B
- The Hugging Face safetensors index reports exactly 3,379,205,632 BF16 parameters across the released files, of which a...
- GPU memory
- 6.3 GB
- weights at bf16, computed. Fits a 12 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- MIT on the code repository.
What it is
ChatVLA targets a specific failure of VLA fine-tuning: once a vision language model is trained on robot actions it forgets how to answer questions about images. The fix is a two stage schedule that first masters control and then reintroduces multimodal data, combined with a mixture of experts that duplicates only the FFN layers so control and understanding do not compete for the same weights. It works in the direction claimed, scoring 37.4 on MMMU and 47.2 on MMStar where the VLA baseline ECoT scores 5.4 and 0, while still beating OpenVLA and Octo on the authors' 25 task real robot suite. It does not fully close the gap to the frozen backbone, however: Qwen2-VL-2B alone scores 41.1 on MMMU and 79.7 on TextVQA against ChatVLA's 71.2. For a hardware decision the bigger problem is silence, since no VRAM, latency, control frequency or action chunk figure appears in the paper, the README or the model card, and there are no simulation benchmarks to cross check the robot claims.
Architecture
- Backbone
- Qwen2-VL-2B-Instruct, used off the shelf
- Action head
- Diffusion policy head following the DiffusionVLA design. A dual expert mixture of experts splits only the FFN layers into a control expert and an understanding expert, while the multi-head self-attention layers stay shared.
- Parameters
- The Hugging Face safetensors index reports exactly 3,379,205,632 BF16 parameters across the released files, of which a separate 1.89 GB action_head_params.safetensors holds the action head. The paper itself only describes the backbone as Qwen2-VL-2B and stresses a 3.5 times smaller VLM backbone than ECoT at 7B; it never states a combined total.
- Pretraining data
- Two stage training rather than large scale pretraining. Stage 1 uses robot manipulation data with action trajectories only. Stage 2 co-trains with about 54K image text pairs sampled from the LLaVA-1.5 dataset at a vision text to robot data ratio of roughly 1 to 3.
- Embodiments
- Single arm and bimanual real hardware used by the authors for 25 manipulation tasks
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 13 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 6.3 GB | The usual way these checkpoints are served |
| int8 | 3.1 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 1.6 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 12 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Checkpoint download | 8.69 GB |
No GPU or VRAM figure is published in the paper, the README or the model card. The 8.69 GB bfloat16 checkpoint suggests a modest footprint, but this has not been measured by the authors and should not be treated as a specification.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- OCRBench full729 points
- Real robot (authors' setup) multi-task cross-skill, 107 trials51.4%success rateReported as 55 of 107 trials. OpenVLA 20 of 107 (18.7 percent), Octo 18 of 107 (16.8 percent). Authors' own hardware, not a public benchmark.source
- MMStar full47.2%
- MMMU validation37.4%
Fine tuned tasks
No results in this category are published for this model.
Embodied reasoning
Offline benchmarks that test understanding rather than control. No robot moves in these, so a high score says nothing about grasping.
- TextVQA validation71.2%
On real hardware
Evaluated on 25 real world manipulation tasks on the authors' own hardware. In the multi task cross-skill setting ChatVLA completes 55 of 107 trials against 20 of 107 for OpenVLA and 18 of 107 for Octo. Long horizon results are reported as average success length, where ChatVLA reaches 0.54 on direct prompting, and as perfect completion of a three step sequence when an external planner supplies the steps.
Fine tuning it yourself
Data is H5py in the same layout DexVLA uses, plus LLaVA format vision language data with image paths and conversation pairs. Training runs in two stages: stage 1 on robot data only with the VL expert frozen and the mixture of experts initialised, stage 2 co-training with a default vision language data ratio of 0.33. No GPU requirements are stated in the README.
Where it helps, where it does not
Strengths
- Keeps usable multimodal question answering after action training, which most VLAs lose entirely
- Small backbone at Qwen2-VL-2B, and the whole checkpoint is only 8.69 GB on disk
- Weights are actually published and the code repository is MIT licensed
- Published as an EMNLP 2025 main conference paper, so the understanding claims went through peer review
Limits
- Multimodal understanding is still below the frozen backbone: 37.4 versus 41.1 on MMMU, 71.2 versus 79.7 on TextVQA, 729 versus 809 on OCRBench.
- No VRAM, no latency, no control frequency and no action chunk size are documented anywhere in the paper, README or model card.
- All robot results come from the authors' own real world setup. There is no LIBERO, SIMPLER or CALVIN number to cross check against other VLAs.
- The Hugging Face weight repository has no model card and no license tag, so the licence position of the weights themselves is unclear even though the code is MIT.
- The authors' own successor ChatVLA-2 reports ChatVLA scoring only 4 of 52 on the open-world math matching task and 22 of 156 on toy placement, which puts a firm ceiling on its open world reasoning.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2502.14420
- https://arxiv.org/html/2502.14420v2
- https://chatvla.github.io/
- https://github.com/tutujingyugang1/ChatVLA_public
- https://huggingface.co/zzymeow/ChatVLA
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.