How to Train SmolVLA on a 24 GB GPU (lerobot 0.6.1)

Most people who want a vision language action model on a real arm stop at the hardware line. GR00T N1.7 and Pi0.5 are around three billion parameters each and want an A100 80 GB or an H100. If what you own is a gaming PC with an RTX 4090, that is the end of the road. SmolVLA is the exception: 450 M parameters, inside LeRobot, built to fine-tune on one consumer card and serve from a CPU.

The manual route first: install lerobot, pull the lerobot/smolvla_base checkpoint, run the real command, read the run while it happens. Then the platform route, and where it does not help.

What SmolVLA is, in numbers you can check

SmolVLA is a flow matching policy bolted onto a small vision language model. The backbone is SmolVLM2-500M-Video-Instruct; the paper keeps only the first 16 layers of its language model, limits each camera frame to 64 visual tokens with a pixel shuffle instead of image tiling, and interleaves cross attention with a self attention layer every second block. Inference cost was a design constraint, not an afterthought.

The benchmarks are why people bother with a 450 M model. On LIBERO it averages 87.3 percent against 76.5 for OpenVLA at 7 B and 86.0 for a robotics-pretrained Pi0 at 3.3 B; on Meta-World, 57.3 against 47.9. On real SO-100 hardware, multi-task training gives 75 percent on pick and place, 90 on stacking, 70 on sorting, averaging 78.3, where ACT trained per task averaged 48.3. The same rows sit beside every published VLA in the SmolVLA arena entry and the ACT against SmolVLA comparison.

Why a 24 GB card is the right first run

LeRobot ships a compute sizing guide, the most useful page in the repo for this. It groups policies by backbone size and gives one VRAM envelope per group, measured at batch size 8 with AdamW, the lerobot default. Optimizer state alone adds 30 to 100 percent over a bare forward and backward pass, so these are not weights-only figures.

Ten to sixteen gigabytes at batch 8 is the argument: a 24 GB card fits that plus the dataloader. AY-Robots puts SmolVLA on the RTX 4090 or any 24 GB card, minimum 30 episodes, LeRobot v3.0 data, 245 ms per action step. Rented, that is 2 to 5 hours at 0.30 to 0.60 USD an hour, about 1 to 3 USD a fine-tuning run, against 4 to 12 USD on the 80 GB tier GR00T and Pi0.5 need (pricing). A failed SmolVLA run is a coffee; a failed GR00T run, lunch.

Step 0: the dataset decides the run, not the flags

Nothing below matters if the recording is bad. The LeRobot SmolVLA page is blunt: the reference dataset was 50 episodes across 5 cube positions, 10 per position, and the same task at 25 episodes performed badly. Repetition per variation generalises, raw episode count does not. Never recorded one? Start at record your first dataset with the desktop client, which writes LeRobot format out of a teleoperation session, or borrow one from the dataset directory.

• At least 30 episodes on AY-Robots, about 50 for the LeRobot reference recipe.
• Every variation you expect at rollout, repeated several times.
• One task string, spelled identically at record and rollout time. The model is conditioned on that text.
• Fixed cameras. A camera moved between recording and rollout is the most common reason a clean loss curve gives a motionless arm.
• A held-out variation you never trained on, so you have something honest to test against.

# v2.1 -> v3.0: aggregates per-episode files into shards and writes the episode offsets
python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=${HF_USER}/so100_pick_place

More on this in how to collect high quality VLA training data. The short version: 30 to 50 clean episodes of one task with deliberate variation beat 200 sloppy episodes of three, by a margin no hyperparameter closes.

Install lerobot 0.6.1

# LeRobot needs Python 3.12 or newer as of 0.6.x
conda create -y -n lerobot python=3.12
conda activate lerobot

# TorchCodec decodes the dataset videos and wants ffmpeg
conda install ffmpeg -c conda-forge

# Base package is deliberately thin; extras pull the rest
pip install 'lerobot[smolvla,training,core_scripts]'

# Sanity check: prints the lerobot version, the GPU torch sees, and the console scripts you got
lerobot-info

The base lerobot install is thin and gates heavy dependencies behind extras: smolvla adds transformers, num2words and accelerate, training the dataset stack and wandb, core_scripts the hardware and visualisation deps. On Linux the install path also decides your CUDA wheel: the PyPI default is a cu130 wheel with a driver floor of 580.65, so on an older driver install torch from the cu128 index first, then lerobot.

The training run, command by command

1. 1
   Authenticate against the Hub

   The base checkpoint comes from the Hub, and your dataset probably does too.

   bash

   hf auth login

2. 2
   Read the options once

   Every field of the pipeline and policy config is a flag. Skim it before digging in the source.

   bash

   lerobot-train --help

3. 3
   Start the fine-tune

   The docs example runs batch 64 on a single A100; the compute guide's own A100 40 GB anchor is batch 16, and 8 is the 24 GB equivalent. No scheduler flag here on purpose, see below.

   bash

   lerobot-train \
     --policy.path=lerobot/smolvla_base \
     --dataset.repo_id=${HF_USER}/so100_pick_place \
     --batch_size=8 \
     --steps=20000 \
     --save_freq=2000 \
     --log_freq=200 \
     --seed=1000 \
     --output_dir=outputs/train/smolvla_pick_place \
     --job_name=smolvla_pick_place \
     --policy.device=cuda \
     --wandb.enable=true

4. 4
   Read the log line, not just the loss

   Every --log_freq steps lerobot prints loss, grdn, lr, updt_s, data_s, smp/s and, on CUDA, mem_gb. mem_gb says whether the batch fits, lr whether the schedule is decaying, and data_s approaching updt_s means the dataloader is the bottleneck, not the GPU.

   bash

   # if data_s creeps toward updt_s
   lerobot-train ... --num_workers=8

5. 5
   Collect checkpoints you can compare

   save_freq defaults to 20000, so a 20000 step run leaves one checkpoint and nothing to compare it against. Set 2000. Pushing to the Hub needs --policy.repo_id.

   bash

   --save_freq=2000 \
   --policy.repo_id=${HF_USER}/smolvla_pick_place \
   --save_checkpoint_to_hub=true

6. 6
   Resume if the machine dies

   Point --config_path at the train_config.json next to the checkpoint. lerobot refuses to start into an existing output_dir unless you are resuming, so you cannot overwrite a run by accident.

   bash

   lerobot-train \
     --config_path=<path to the saved train_config.json> \
     --resume=true

The flags that actually change the outcome

The defaults you inherit if you touch nothing

A policy config in lerobot carries its own optimizer and scheduler preset, and unless you set use_policy_training_preset=false those presets win. Half the questions people ask about SmolVLA training are answered by a default they did not know existed.

The two lines that surprise people are freeze_vision_encoder and train_expert_only, both true. Out of the box you train roughly 100 M parameters, not 450 M, which is why it fits on 24 GB. LeRobot's own reference run on a four-GPU H100 cluster flips both to false; on one 24 GB card that turns a working run into an out of memory crash.

How long the run takes, and how many steps is enough

LeRobot publishes wall-clock anchors for five epochs over a roughly 50 episode dataset, about 45000 frames at 30 fps. Order of magnitude figures, the docs say, but they are the difference between expecting an hour and a day.

Running the fine-tuned policy back on the arm

lerobot-rollout \
  --strategy.type=base \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --task="Grasp the cube and put it in the box." \
  --policy.path=${HF_USER}/smolvla_pick_place

The 245 ms per action step is easy to misread: the policy emits chunk_size = 50 actions per forward pass and executes n_action_steps = 50 of them, so how often you pay that cost is set by those knobs, not by how often the servos get a command. That is what action chunking buys, and why a 245 ms model can drive a 30 Hz arm. What is left is inference latency at the end of the chunk.

That third row is what most write-ups skip. Async inference is about 30 percent faster and roughly doubles throughput in a fixed window, and the paper calls the success rates comparable, which on the average they are. Underneath it, sorting fell from 70 to 50 percent while pick and place gained 5. lerobot 0.6.1 carries the other lever in the same binary: --inference.type=rtc switches the rollout to real time chunking, which the script's own usage block recommends for the slow VLAs, Pi0, Pi0.5 and SmolVLA.

Two routes to the same checkpoint

pip install 'lerobot[smolvla,training,core_scripts]'
hf auth login

lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=${HF_USER}/so100_pick_place \
  --batch_size=8 --steps=20000 \
  --save_freq=2000 --seed=1000 \
  --output_dir=outputs/train/smolvla_pick_place \
  --job_name=smolvla_pick_place \
  --policy.device=cuda --wandb.enable=true

When SmolVLA is the wrong choice

The test for whether SmolVLA was the right first run is not whether it worked, but whether the failure told you something. Reach 60 or 70 percent and a bigger model is a reasonable next spend: the data carries signal. Reach 10 percent and a 3 B model most likely also reaches 10 percent, which you just learned for three dollars instead of twelve.

Worth reading before spending more: Pi0.5 against SmolVLA for more capacity on the same idea, and GR00T N1.7 against SmolVLA for the NVIDIA route, both 80 GB tier at 4 to 12 USD a run. The other way, ACT is the cheaper baseline: 80 M parameters, 20 ms per action step, no language conditioning. All five sit on the policies page; the arena has 85 models and 332 benchmark results.

The checklist before you scale anything

1. Did the lr reach its 2.5e-6 floor? Below 30000 steps lerobot rescales the decay and says so at startup; above it, set --policy.scheduler_decay_steps yourself.
2. More than one checkpoint, and episodes held out with --dataset.eval_split so the eval loss means something.
3. Does the policy move at all? A falling loss with a motionless arm has specific causes: loss falls, policy does nothing.
4. Does it survive a change of scene? If not: policy only works in one setup.
5. Did you write down the seed? lerobot defaults to 1000, so two untouched runs stay comparable.
6. Only then: more episodes, more variation, or a bigger model. In that order.

For why these models exist and what they do with the language input, vision language action models is the background; the SO-100 complete guide runs assembly through calibration to the first imitation learning run. For the finished checkpoint, run your first policy; if the arm never shows up, the failure mode index.