What Is a Vision-Language-Action Model? Inputs, Outputs, Limits

A vision-language-action model is one neural network that reads camera frames, a robot state vector and one sentence of plain English, and writes joint commands. That is the whole idea. Everything after it is an argument about the output representation, the pretraining, and how fast the thing runs next to the servos. The glossary entry at vision-language-action model gives the short version; this is the long one.

The name comes from RT-2 (Google DeepMind, July 2023). The contribution was blunt: take a vision-language model that already knows what a banana looks like and train it to emit robot actions as if they were words. The paper describes VLAs as models that express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. Everything since has kept the three input streams and changed the output head.

What goes in

Three streams, every time, whether the model is 80 million parameters or 3 billion.

That triple is what the LeRobot dataset format stores per frame, which is why it became the interchange layer for this class of model. An episode is one demonstration: a video per camera, a joint-state row and an action row per timestep, and one task string. When you record with the desktop client from the download page, or with lerobot-record locally, that is the file you produce.

What comes out

This is where the designs differ. There are two answers, and one of them is mostly history.

Answer 1: action tokens as text (RT-2, 2023)

RT-2 did not invent this encoding; it says it bases the action encoding on the discretisation proposed for RT-1. The action is the 6-DoF positional and rotational displacement of the end effector, plus gripper extension, plus a discrete command that terminates the episode. Every continuous dimension was discretised into 256 uniform bins, so an action became eight integers, concatenated into a string and predicted like any other sentence.

# RT-2 action target, section 3.2 (Robot-Action Fine-tuning) of arXiv:2307.15818
# terminate  d_pos_x d_pos_y d_pos_z  d_rot_x d_rot_y d_rot_z  gripper
"1 128 91 241 5 101 127"

# 256 uniform bins per continuous dimension.
# PaLI-X: integers up to 1000 each have a unique token, so bins map straight onto them.
# PaLM-E: the 256 least frequently used tokens in the vocabulary get overwritten as the action vocabulary.

It worked, and it was slow: 1 to 3 Hz for the 55B PaLI-X variant, about 5 Hz for the 5B, served from a multi-TPU cloud service over the network. Full breakdown in how RT-2 transfers web knowledge to robot control.

Answer 2: continuous action chunks (Pi0, GR00T N1, SmolVLA)

Every VLA you can train here uses the second answer. Instead of one action per forward pass, the network emits a whole action chunk: the next H joint targets at once, produced by iterative denoising, either diffusion or flow matching, on a small head conditioned on the VLM's output tokens. ACT, the fifth trainable policy here, also emits chunks, but by direct regression rather than denoising and with no VLM in front of it. That difference is the subject of a later section.

The chunk is why a 485 ms model is usable at all. If Pi0.5 predicts 50 actions in one pass and the arm executes them at 30 fps, the model has about 1.6 seconds of runway before it has to think again. It is the first thing to tune when a policy stutters, and the reason inference latency alone is a misleading number.

How it differs from a classical controller

Same task: pick up a red block, drop it in a bin. Here is what the classical ROS 2 stack asks for, and what a VLA trained by imitation learning asks for instead.

The loop, from data to a moving arm

The manual path is three commands plus a lot of waiting. These are the lerobot entry points as documented in August 2026. Note that lerobot-rollout and lerobot-eval are two different tools, not one renamed: lerobot-rollout deploys a policy on a real robot, while lerobot-eval scores a policy inside a simulation environment and wants an --env.type. Reaching for the wrong one is a common first-run stumble.

1. 1
   Record demonstrations

   Drive the arm by leader-follower teleoperation and record. The single_task string is the sentence the model gets conditioned on, so write the one you intend to type at inference time. LeRobot's guidance is at least 50 episodes, 10 per object position.

   bash

   lerobot-record \
       --robot.type=so101_follower \
       --robot.port=/dev/tty.usbmodem585A0076841 \
       --robot.id=my_awesome_follower_arm \
       --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \
       --teleop.type=so101_leader \
       --teleop.port=/dev/tty.usbmodem58760431551 \
       --teleop.id=my_awesome_leader_arm \
       --dataset.repo_id=${HF_USER}/record-test \
       --dataset.num_episodes=5 \
       --dataset.single_task="Grab the black cube"

2. 2
   Fine-tune a base checkpoint

   Not from scratch: this is fine-tuning on a vendor checkpoint that already saw robot data. Hugging Face documents roughly 4 hours on one A100 for 20k steps of SmolVLA.

   bash

   cd lerobot && lerobot-train \
     --policy.path=lerobot/smolvla_base \
     --dataset.repo_id=${HF_USER}/mydataset \
     --batch_size=64 \
     --steps=20000 \
     --output_dir=outputs/train/my_smolvla \
     --job_name=my_smolvla_training \
     --policy.device=cuda

3. 3
   Run the policy on the arm

   lerobot-rollout loads the checkpoint and closes the loop. The --task string must match what you recorded. --strategy.type picks the execution mode (base, sentry, highlight, dagger or episodic). --inference.type defaults to sync, one policy call per control tick; rtc is Real-Time Chunking, which smooths execution across chunk boundaries and is the one the docs point slow VLAs at.

   bash

   lerobot-rollout \
     --strategy.type=base \
     --policy.path=${HF_USER}/my_policy \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM1 \
     --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}}" \
     --task="Put lego brick into the transparent box" \
     --duration=60

What a VLA cannot do

This is the part most explainers skip. The published robustness work is unambiguous, and it matches what happens on a desk with an SO-100 on it.

• It cannot survive a moved camera. LIBERO-Plus (October 2025) perturbed seven dimensions - object layout, camera viewpoint, robot initial state, language, lighting, background texture, sensor noise - and reported success dropping from 95 percent to below 30 percent under modest perturbation, worst for viewpoints and initial states. If your policy only works in one setup, that is expected behaviour: policy only works in one setup.
• It often does not read your instruction. The same paper found models "largely insensitive to language variations" and that they "tend to ignore language instructions completely". The L does less work than the name suggests, especially after a single-task fine-tune where the string never varied.
• It cannot recover reliably. A June 2026 benchmark of Pi0.5, SmolVLA, Wall-X and ACT on the SO-101 found execution instability to be the dominant failure source, with recovery varying substantially across architectures.
• It cannot be fast and remote at once. Inference has to sit next to the servos for anything reactive. The control loop here is 20 to 485 ms per action step, and internet round trips on top turn a working policy into a hesitant one.
• It cannot tell you why. There is no intermediate representation to inspect. The method is ablation: black out one camera, re-run, see what changes.
• It cannot generalise from data you did not collect. The minimums here, 30 episodes for SmolVLA and 50 for the rest, are floors for one tightly controlled task.

Is ACT a VLA? No, and that matters

ACT sits on the same policies page as GR00T and Pi0.5, which makes people assume it belongs to the same family. It does not. ACT, from the ALOHA paper (Zhao et al., 2023), is an Action Chunking Transformer trained as a conditional VAE. Its vision backbone is a ResNet-18; its inputs are camera images, joint positions and a latent style variable z that is set to zero at inference. No language encoder, no web pretraining.

ACT is the cheapest thing that works on a single fixed task, and the baseline every VLA should beat. The paper trains it in about 5 hours on one 11 GB RTX 2080 Ti and measures inference at roughly 0.01 seconds, which is the whole argument for keeping it in the lineup. It also adds temporal ensembling, averaging overlapping chunks to smooth the motion. If it beats your VLA on your data, the data is the problem. There is no ACT base model, which is why the ACT page lists no vendor checkpoint. Compare it to a real VLA on ACT versus SmolVLA before spending an A100 hour.

Getting one running: the two paths

How fast is fast enough

Latency is where the abstraction leaks into hardware, so be precise about which number you are reading. Several measurements get quoted as "VLA latency" and they are not comparable.

The chunk hides most of this. SmolVLA goes further with an asynchronous stack that decouples execution from the next prediction: the paper reports a task finishing in 9.7 seconds instead of 13.75, and 19 pick-and-place cycles in a fixed window against 9 synchronously. In lerobot that is --inference.type=rtc, which the docs point at the models with high inference latency: Pi0, Pi0.5 and SmolVLA.

Where to go from here

• No arm yet: drive a real SO-100 in the browser at live, or see the no-hardware routes at try.
• Arm but no data: record your first dataset, then browse the directory.
• Choosing a model: GR00T N1.7 versus Pi0.5, then a guide under train such as SmolVLA on the SO-100.
• Background: the full SO-100 guide and Pi0 and flow matching.