Imitation Learning vs Reinforcement Learning for Robot Arms

Open any of the five policies you can train on this platform and look at how it was made. ACT learns from 50 teleoperated episodes. SmolVLA, Pi0.5 and GR00T N1.7 are pretrained on large piles of human demonstrations and then fine-tuned on yours. Not one of them has a reward function. Not one of them ever tries something, fails, and updates on the failure. Every policy you can realistically put on a low-cost arm this year is trained by imitation learning.

That is not because reinforcement learning does not work on robots. It works, and on some tasks it beats imitation by a wide margin. It is because the bill is different. This article puts the two side by side with numbers from the papers, explains the one thing behaviour cloning provably gets wrong, and walks the hybrid loop that people actually run on a real arm today. Everything about lerobot below was checked against the v0.6.1 sources and the current LeRobot documentation, and every paper number was read out of the paper rather than out of a summary.

The difference is what you have to supply

Both approaches end with the same artifact: a checkpoint that maps camera images and joint states to the next action. What differs is the supervision you have to produce before training can start.

That last row is the whole argument for RL, and the second row is the whole argument against it. A demonstration dataset is finite work: you sit down, drive the arm, and stop. An RL run is open-ended interaction with a physical machine that is, by construction, mostly doing the task wrong.

Why imitation won the last three years

The short version: demonstrations turned out to scale better than reward engineering. Once action chunking and generative action heads made short demonstration sets usable, the field stopped fighting reward functions and started collecting data.

Every row is imitation. The differences between them are architectural, not about the learning signal: flow matching versus diffusion versus a plain transformer regressor. If you want the head-to-head on the five you can train here, the policy comparison has params, GPU tier and inference latency in one table.

What behaviour cloning provably gets wrong

There is one failure mode that is not an engineering bug and cannot be fixed with more of the same data. A behaviour-cloned policy is trained on the distribution of states a human visits. At run time it visits its own distribution. Every small error moves it slightly off the demonstrated manifold, where its next prediction is slightly worse, which moves it further off.

Ross, Gordon and Bagnell (2011), Theorem 2.1
credited in that paper to Ross and Bagnell (2010):

  if  E_{s ~ d_pi*} [ loss(s, pi) ] = e
  then  J(pi) <= J(pi*) + T^2 * e

  T = episode length in steps
  e = per-step 0-1 loss against the expert

  "Note that this bound is tight" - the extra cost really can grow
  quadratically in T, and behaves as Theta(T^2 * e) for small e.

Put numbers on it. A 30-second episode recorded at 30 Hz is T = 900 steps. A per-step error rate of one in a thousand is a very good policy by supervised-learning standards, and the bound still allows an extra cost of 900 squared times 0.001, which is far more than the total cost of the task. The bound is worst case, not typical case, but the shape is right, and it explains something every practitioner has seen: policies that look excellent on held-out validation loss and still stall halfway through the real task. That specific symptom has its own page under loss falls, policy does nothing.

What reinforcement learning costs on real hardware

RL fixes the distribution problem by definition: it trains on its own state distribution. The price is interaction. These are the published figures, not estimates.

The Rubik's cube row is the one people misremember. That policy is a real achievement, and it solved a fully scrambled cube in 2 of 10 real trials after months of continuous training at a scale of compute that no individual has. It is also worth reading the table it comes from: the 20 percent full-scramble and 60 percent half-scramble figures are for the variant that reads face angles from a sensor-equipped cube. The same policy reading face angles from vision alone managed 0 percent on the full scramble and 20 percent on the half. Simulation-first RL is a real strategy, and GPU-parallel simulators have made the sample cost much less frightening than it was in 2019. But sim-to-real for contact-rich manipulation on a 110 to 150 EUR arm with unmodelled backlash is a research project, not a Saturday.

What RL still wins, with the numbers

The honest answer is: the last stretch of reliability on contact-rich, tight-tolerance tasks, and cycle time. HIL-SERL (Luo, Xu, Wu and Levine, arXiv 2410.21845) is the cleanest published comparison because the imitation baselines were trained on the same amount of human data as the RL runs: the same number of demonstration episodes and interventions, on the same tasks and the same hardware.

The paper's second table is the one that should change how you plan a project, because it prices each ingredient separately. Read the bottom three rows as a ladder: strip the interventions and the system loses half its success rate, strip the demonstrations too and it learns nothing at all.

Two details in that paper matter more than the headline. First, the RL policies are faster than the human demonstrations they started from, which no imitation method can be. Second, the ablation: the same system started without demonstrations and without human corrections scored 0 percent on all three tasks it was evaluated on, and with demonstrations but no corrections it averaged 49 percent. The interventions are not a convenience feature. They are the reason it works at all.

The hybrid that people actually run: human-in-the-loop RL

The working recipe in 2026 is not RL versus imitation. It is imitation first, then a small amount of on-robot RL with a human ready to take over. lerobot ships an implementation of HIL-SERL, and it supports SO-100 class arms. The algorithm underneath is SAC with offline demonstration data mixed into every batch, following RLPD.

1. 1
   Install the RL extra

   The HIL-SERL code lives behind its own extra. You also need an NVIDIA GPU and a URDF for your arm, because the whole loop runs in end-effector space and needs inverse kinematics.

   bash

   pip install -e ".[hilserl]"

2. 2
   Find safe end-effector bounds

   Bounding the workspace is not a safety nicety, it is what makes the exploration problem small enough to solve. Move the arm through the region that solves the task and record the extremes.

   bash

   lerobot-find-joint-limits \
     --robot.type=so100_follower \
     --robot.port=/dev/tty.usbmodem58760431541 \
     --robot.id=black \
     --teleop.type=so100_leader \
     --teleop.port=/dev/tty.usbmodem58760431551 \
     --teleop.id=blue

3. 3
   Record a handful of demonstrations

   Set mode to record in the env config and drive the task. HIL-SERL is seeded with demonstrations, not started cold. The paper keeps a separate demo buffer, usually 20 to 30 episodes per task, and samples from it in every training batch.

   bash

   python -m lerobot.rl.gym_manipulator \
     --config_path src/lerobot/configs/env_config_so100.json

4. 4
   Crop the camera views

   Visual RL is far more distraction-sensitive than behaviour cloning. Crop each view down to the workspace and resize to 128x128, which is the resolution the policies here were validated on.

   bash

   python -m lerobot.rl.crop_dataset_roi --repo-id username/pick_lift_cube

5. 5
   Train the reward classifier

   This is the success detector. It is a small CNN on top of a pretrained vision model, helper2424/resnet10 in the reference config, with num_classes set to 2 for binary success and failure. Collect the labelling dataset with terminate_on_success set to false so the episodes keep running past the first success and you get enough positive frames.

   bash

   lerobot-train \
     --config_path path/to/reward_classifier_train_config.json

6. 6
   Start the learner, then the actor

   Two processes, talking over gRPC. The learner holds the replay buffers and does gradient steps; the actor drives the robot and streams transitions back. Run both on the machine next to the arm.

   bash

   # terminal 1
   python -m lerobot.rl.learner \
     --config_path src/lerobot/configs/train_config_hilserl_so100.json
   
   # terminal 2
   python -m lerobot.rl.actor \
     --config_path src/lerobot/configs/train_config_hilserl_so100.json

7. 7
   Intervene, then intervene less

   Press the gamepad trigger (or space on the keyboard) to take over, and again to hand control back. A run is going well when your intervention rate falls over time. If it does not fall, stop and fix the reward classifier or the bounds rather than training longer.

   bash

   # watch the intervention rate in the wandb dashboard
   # a healthy run: high at the start, near zero by the end

The defaults are worth comparing directly against the imitation trainer defaults this platform sends, because they are a different kind of number entirely. Imitation training is a batch job with a step count. RL training is a control loop with a discount factor.

The cheaper hybrid: corrections without any RL

If you do not want to build a reward classifier, there is a middle path that captures most of the distribution-shift fix and none of the reward engineering. Run the trained policy, let a human take over when it is about to fail, record the recovery, and fine-tune on the combined dataset. That is HG-DAgger, and lerobot exposes it as a rollout strategy.

lerobot-rollout --strategy.type=dagger \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM1 \
  --robot.cameras='{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }' \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM0 \
  --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \
  --dataset.repo_id=${HF_USER}/rollout_hil_dataset \
  --dataset.single_task="Put the cube in the bin" \
  --dataset.fps=30 \
  --strategy.num_episodes=50

The loop is pause, take over, drive back to a state the policy understands, correct, hand control back, all inside one continuous episode with no reset. Both the autonomous and the human segments are recorded, so the LeRobot dataset that comes out contains exactly the recovery behaviour the original demonstrations lacked. Then you fine-tune on the merged set and go again.

Two variants published in 2024 and 2025 are worth knowing if you go further. RLDG (arXiv 2412.09858) uses RL to generate the fine-tuning data and then distills it into a generalist policy, reporting up to 40 percent higher success rates than the same policy fine-tuned on human demonstrations. ConRFT (arXiv 2502.05450) fine-tunes a VLA with an offline stage that mixes behaviour cloning with Q-learning, then an online stage with human interventions, and reports 96.3 percent average success across eight real-world tasks after 45 to 90 minutes of online fine-tuning. Both are hybrids. Neither is RL from scratch.

Do it yourself, or do it here

# the imitation baseline first, always
lerobot-train \
  --dataset.repo_id=${HF_USER}/so101_test \
  --policy.type=act \
  --output_dir=outputs/train/act_so101_test \
  --job_name=act_so101_test \
  --policy.device=cuda \
  --wandb.enable=true

Where this platform does not help you

Being blunt about it, because the gap is structural rather than a missing feature flag. AY-Robots trains five policies, all by imitation. There is no reward classifier trainer, no replay buffer, no actor-learner split, and no way to express a return.

What the platform does do well for the hybrid story: it gets you the demonstration dataset and the imitation baseline cheaply and repeatably, which is the required first stage of every hybrid method above. HIL-SERL needs 20 demonstrations before it starts. ConRFT needs an offline stage. RECAP starts from demonstrations. None of them start from nothing, and the recording workflow plus the first training run is the cheapest way to get that stage done.

How to choose, in one table

Read this top to bottom and stop at the first row that matches your situation. Most people stop in the first three.

Two of those rows are free to act on right now. The live arm is a real SO-100 streaming in the browser with no signup, queue-based, and the arena holds 85 VLA models with 332 benchmark results, each value linked to its own paper or model card. If you are still deciding whether to buy hardware at all, the three ways to start lays out what each path costs before you commit.

One thing that does not appear in that table: choosing RL because it sounds more principled. The reason vision-language-action models are trained by imitation is not intellectual laziness. It is that a demonstration is a dense, unambiguous supervision signal that a person can produce at 30 Hz by moving their hand, and no one has found anything comparably cheap on the reward side. If you want the longer version of that argument, the VLA overview and the BridgeData V2 breakdown of which imitation and offline RL methods actually work both go deeper.