Distilling a Large Robot Policy Into a Small One: What Survives

Why anyone shrinks a robot policy

A policy that takes half a second to produce an action is not slow the way a slow web page is slow. It is slow the way a car with a delayed steering wheel is slow. The arm keeps moving during the wait, the scene it was looking at is gone by the time the action arrives, and the correction it computes is a correction for the past. That is why inference latency gets its own column on every model page here rather than a footnote.

The five policies you can train on AY-Robots span more than an order of magnitude in that number, and they span it for architectural reasons that distillation research attacks directly.

So there are two ways to get a faster robot. Buy the latency down with engineering that leaves the model alone, which is what action chunking and asynchronous execution do. Or make the model itself smaller. Distillation is the second route, and the interesting question is what you lose on the way.

What distillation means when the output is a trajectory

The original idea is old and simple. Hinton, Vinyals and Dean published Distilling the Knowledge in a Neural Network in March 2015: train a big model, then train a small model to match the big model's soft output distribution rather than the hard labels. The soft distribution carries information the labels do not, which is why the student can beat a small model trained on labels alone. Rusu and colleagues carried that into control later the same year with Policy Distillation, extracting DQN policies into smaller networks and consolidating several task-specific policies into one.

Robot policies broke the recipe in one specific way. A classifier emits a probability vector, and matching probability vectors is a well-posed target. A modern vision-language-action model emits a chunk of continuous joint targets, often through a diffusion or flow-matching head with its own internal iteration count. There is no single soft distribution to copy. So the field split into several different things that all get called distillation.

Which loss, since there is no soft distribution to copy

Hinton's method has one specific ingredient: the temperature-softened output distribution. A robot policy's output is a chunk of joint targets, and there is no temperature knob on a joint angle. So in practice most people fall back to plain regression, and regression has a known pathology in imitation learning. If the teacher sometimes goes left and sometimes right around an obstacle, an L2 student learns the average, which goes straight into the obstacle. That is the same mode-averaging problem that pushed the field toward diffusion and flow-matching heads in the first place, and naive distillation reintroduces it through the back door.

The fixes are architectural rather than clever. Keep a generative head on the student so it can represent a distribution rather than a mean. Distil the teacher's whole chunk instead of only its first action, so multi-step structure survives. Or add supervision that is not an action at all, which is what VLA-AD does with task-phase anchors and multi-frame direction descriptions, and what its authors credit for making the student robust to the teacher's noisy gripper commands.

The numbers people actually published

It is worth putting the reported speedups next to each other, because they measure different things and the spread between them is the whole story. A 44x parameter reduction that buys 3.28x wall-clock speedup is a normal outcome, not a disappointing one. Parameters and milliseconds are only loosely coupled once you leave the same architecture family.

The Gemini Robotics row is the one deployment-scale case with published latency. The report describes the backbone as a distilled version of Gemini Robotics-ER whose query-to-response latency was brought down from seconds to under 160 ms, then paired with a local action decoder on the robot so the end-to-end path from raw observation to action chunk is about 250 ms. Because the chunk contains many actions, the effective control frequency is 50 Hz. Note the shape of that: distillation handled the backbone, chunking handled the residual delay. Neither alone was enough.

What survives distillation, and what goes first

This is the part the abstracts skip. Success rate on a fixed benchmark suite is a poor proxy for what a policy is, because a benchmark suite holds the vocabulary, the objects and the camera fixed. Distillation preserves whatever the distillation data covered and quietly drops the rest.

Survives: the motion

Trajectory shape transfers well. If the teacher approaches from above, decelerates at three centimetres and closes the gripper over 200 ms, a student trained on enough of its chunks reproduces that. This is unsurprising. It is imitation learning with an unusually patient demonstrator that never gets tired and never mis-teleoperates. It is also why distillation is attractive for exactly one situation, which is when you already own a teacher that works and cannot afford to run it.

Goes first: language grounding

The TinyVLA failure analysis is the clearest published evidence. The authors compared TinyVLA-0.4B, TinyVLA-1.3B and a 3B variant across four tasks and counted failures by cause. The 0.4B model failed three times specifically by misinterpreting the instruction, which they attribute to the smaller VLM's limited language comprehension, and that failure class disappeared at 1.3B. Instruction following does not degrade smoothly with size. It works or it does not, and for the architectures they tested the threshold sat somewhere around a billion parameters.

The same effect shows up from the other side in the OpenVLA paper. RT-2-X scores higher on semantic generalization than OpenVLA, which the OpenVLA authors attribute to RT-2-X being co-fine-tuned with internet pretraining data rather than fine-tuned on robot data alone. Fine-tuning on actions costs you web knowledge. Distilling on actions costs more of it, because now the only signal reaching the student is joint angles.

Goes second: the teacher's mistakes get baked in

Does it beat just training the small model directly?

This is the question that decides whether the whole exercise is worth a weekend, and there is a real answer. Busbridge and colleagues published Distillation Scaling Laws in February 2025, revised July 2025 for ICML. Their finding, stated in the abstract: in settings involving many students or an existing teacher, distillation outperforms supervised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable.

That is a language-model result, not a robotics result, so read it as a prior rather than a proof. But it matches what the VLA literature shows when you line the numbers up. The two most successful small VLAs were not distilled at all.

Those figures are Table 2 of the SmolVLA paper (arXiv 2506.01844, June 2025), with the baselines taken from the OpenVLA and TinyVLA papers. A 450M model with no robotics pretraining beats a 7B model that saw 970k robot demonstrations. TinyVLA reaches 14 ms per action prediction on an A6000 against 292 ms for OpenVLA-7B, and its authors got there by starting from a small VLM rather than by shrinking a large one. Neither result required a teacher.

The manual path: distilling a teacher with LeRobot

LeRobot ships no distillation trainer, so the practical recipe is policy distillation in the Rusu sense done by hand: use the teacher as a labeller, write its actions into a LeRobot dataset, train the student on that with the ordinary trainer. Everything below is against lerobot 0.6.2 on main, checked 23 August 2026, which requires Python 3.12 or newer.

1. 1
   Install the pieces you actually need

   The smolvla extra pulls the SmolVLM2 backbone dependencies. The async extra pulls the gRPC stack for the policy server, which you want later for measuring real control rates rather than benchmark numbers.

   bash

   git clone https://github.com/huggingface/lerobot.git
   cd lerobot
   pip install -e ".[smolvla]"
   pip install -e ".[async]"
   python -c "import lerobot; print(lerobot.__version__)"

2. 2
   Measure the teacher before you distil it

   Do this first. If the teacher is only twice too slow, chunking and asynchronous execution close the gap without touching a single weight and you can stop here. The LeRobot async docs put Pi0 at 14 GB of memory at inference time against roughly 2 GB for SmolVLA, which is usually a better predictor of your problem than the parameter count.

   bash

   python -m lerobot.async_inference.policy_server \
     --host=127.0.0.1 \
     --port=8080
   
   # in a second terminal, drive the real arm against it
   python -m lerobot.async_inference.robot_client \
     --server_address=127.0.0.1:8080 \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=follower_so100 \
     --policy_type=smolvla \
     --pretrained_name_or_path=lerobot/smolvla_base \
     --policy_device=cuda \
     --actions_per_chunk=50 \
     --chunk_size_threshold=0.5 \
     --debug_visualize_queue_size=True

3. 3
   Relabel a dataset with the teacher's actions

   This is the distillation step, and it is about thirty lines. Load the teacher, walk your recorded frames, ask for a chunk per frame, write the chunk into a new dataset. Run the teacher on the GPU pod, not on the robot host. Doing this offline on existing recordings is the cheap version; doing it on states the student visits is the DAgger version and it is meaningfully better.

   python

   # distil_relabel.py - lerobot 0.6.2
   import torch
   from lerobot.datasets.lerobot_dataset import LeRobotDataset
   from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
   
   teacher = SmolVLAPolicy.from_pretrained("lerobot/smolvla_base").to("cuda").eval()
   src = LeRobotDataset("<you>/pick_place_so100")
   
   rows = []
   with torch.inference_mode():
       for i in range(len(src)):
           frame = src[i]
           batch = {k: v.unsqueeze(0).to("cuda") for k, v in frame.items()
                    if isinstance(v, torch.Tensor)}
           batch["task"] = [frame["task"]]
           chunk = teacher.predict_action_chunk(batch)   # (1, chunk_size, action_dim)
           rows.append(chunk[0, 0].cpu())                # first action of the chunk
   
   # Write rows back out as the action column of a new LeRobotDataset, keeping the
   # original observations. That file is your student's training set. Keep the full
   # chunk instead of chunk[0] if the student uses the same chunk_size.

4. 4
   Train the student on the teacher's labels

   Nothing exotic here. It is the ordinary trainer pointed at the relabelled dataset. The SmolVLA docs put 20k steps at roughly four hours on a single A100. Keep the student's chunk length equal to the teacher's, or the two disagree about what an action even is.

   bash

   lerobot-train \
     --policy.path=lerobot/smolvla_base \
     --dataset.repo_id=${HF_USER}/pick_place_so100_distilled \
     --batch_size=64 \
     --steps=20000 \
     --policy.chunk_size=50 \
     --policy.n_action_steps=50 \
     --output_dir=outputs/train/student \
     --job_name=distil_student \
     --policy.device=cuda \
     --wandb.enable=true

5. 5
   Evaluate in Hz, not in loss

   The whole point was latency, so the acceptance test is a wall clock. Roll the student out on the real arm and compare completion time against the teacher, not validation loss against the teacher. Real-time chunking is available as an inference-time option in the rollout script and needs no retraining.

   bash

   lerobot-rollout \
     --strategy.type=base \
     --robot.type=so101_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower_arm \
     --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \
     --task="Grasp a lego block and put it in the bin." \
     --inference.type=rtc \
     --inference.rtc.execution_horizon=10 \
     --inference.rtc.max_guidance_weight=10.0 \
     --policy.path=${HF_USER}/student_smolvla

The distillation that already happened: SmolVLA's architecture

There is a fourth flavour that never gets called distillation and arguably should: designing the small model so it reuses a large model's weights without carrying its compute. SmolVLA does exactly this, and the config file is short enough to read in full.

# lerobot 0.6.2, src/lerobot/policies/smolvla/configuration_smolvla.py
vlm_model_name = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
num_vlm_layers = 16           # keep only the FIRST 16 layers of the VLM
self_attn_every_n_layers = 2  # interleave cheap cross-attn with self-attn
attention_mode = "cross_attn"
num_steps = 10                # flow-matching steps at inference
chunk_size = 50
n_action_steps = 50
freeze_vision_encoder = True
train_expert_only = True      # only the ~100M action expert learns
optimizer_lr = 1e-4

The paper explains the reasoning. The vision encoder emits 64 visual tokens per frame using a pixel-shuffle operation on the global image only, with no tiling, because tiling costs inference time. The top half of the VLM is discarded outright rather than distilled, and the ablation says that beats the alternatives: keeping the first N layers scored better than sampling every second layer (75.5 average on LIBERO) and better than swapping in a genuinely smaller 256M VLM (75.8). Of the 450M total, roughly 100M is the action expert, and it is the only part that trains.

This is why the honest recommendation for most readers is to pick a smaller model rather than build one. Comparing ACT against SmolVLA or GR00T N1.7 against SmolVLA gets you the decision distillation would have given you, in an afternoon instead of a month, and with a checkpoint somebody else already validated.

Two ways to get a faster policy on an SO-100

What it costs to answer the question empirically

Running the comparison is cheaper than reading about it. On the spot market a run on the A100 or H100 tier takes 3 to 6 hours at 1.20 to 2.00 USD per hour, so roughly 4 to 12 USD. A run on the RTX 4090 or 24 GB tier takes 2 to 5 hours at 0.30 to 0.60 USD per hour, so roughly 1 to 3 USD. Training the teacher, training the student and comparing them on the arm is a two-digit dollar experiment.

Note what that budget does not include: your relabelling pass. Distillation adds a full inference sweep over the dataset at teacher latency. At 485 ms per action step, a 50-episode dataset recorded at 30 fps with 60-second episodes is roughly 90,000 frames, which is over twelve GPU-hours of pure labelling before the student sees a single gradient. Batch it, but do not forget it exists when you compare against the cost of simply training the small model twice.

What to measure, in order

1. Wall-clock time to complete the task on the real arm, teacher against student. The SmolVLA paper used exactly this: 9.7 seconds asynchronous against 13.75 seconds synchronous, about 30 percent faster, on the same policy.
2. Successful cycles in a fixed window. Same paper, same task: 19 pick-and-place cycles asynchronous against 9 synchronous. This number moves far more than success rate does.
3. Instruction following on held-out phrasings. This is the capability distillation costs you, so test it deliberately rather than hoping.
4. Behaviour on states the teacher never visited. Nudge the object mid-episode. If the student freezes where the teacher recovered, you have a covariate-shift problem rather than a capacity problem, and the freeze has its own failure page.
5. Memory at inference, not parameter count. LeRobot's async docs put Pi0 at 14 GB and SmolVLA at roughly 2 GB, which decides what hardware can sit next to the arm on your SO-100.
6. Whether the loss fell while nothing moved. That combination has its own diagnosis, and distillation makes it more likely, because a student can match a teacher's mean action almost perfectly and still never close the gripper.

One last framing worth carrying around. VITA-VLA (arXiv 2510.09607, October 2025) runs distillation backwards: a small pretrained action model acts as the teacher and a large VLM as the student, with a lightweight alignment stage mapping the VLM's hidden states into the small model's action space. They report 97.3 percent average on LIBERO and 82.0 percent across five real tasks, 17 points above the teacher. Distillation is not a synonym for compression. It is a way of moving one specific competence between two networks, and the direction is a design decision. Related reading here: how VLAs are put together, why flow matching changed the action head, and the SO-100 setup guide if you do not have an arm to test any of this on yet. If you have no hardware at all, the queue-based live arm needs no signup.