Failure Recovery Behaviours: Why Perfect Demos Break Policies

The gripper closes half a centimetre short. The cube stays on the table. And the policy carries on to the place position with nothing in its hand, because in every episode it was trained on, the cube was already between the fingers by that point in the trajectory. There is no frame anywhere in the dataset that shows an empty gripper at that height, so there is nothing for the model to imitate except moving on.

That is not a training bug. It is what the data asked for. A set of clean demonstrations describes one path through the task and says nothing about how to get back onto it. This article is about the other half of the data: recording failure and correction on purpose. What the published work measured, what it costs, and how to run it on an SO-100 with LeRobot datasets.

A dataset of perfect runs is a dataset with a hole in it

The theory is older than every model on this site. Ross and Bagnell, in Efficient Reductions for Imitation Learning (AISTATS 2010), state that the supervised treatment of imitation leads to compounding errors and a regret bound that grows quadratically in the time horizon of the task. The follow-up, DAgger (Ross, Gordon and Bagnell, AISTATS 2011), is the standard answer: train on states the learner actually visits, not the ones the expert visited. Both papers are about distribution, not about model capacity. A bigger VLA trained on the same one-path data has the same hole in it.

The ALOHA and ACT authors put the manipulation version plainly: a small error in a predicted action produces a large difference in state, the arm drifts off its training distribution, and it lands in states from which recovery is hard. Action chunking shortens the effective horizon and reduces how fast that compounds. It does not add a recovery behaviour. If the wrist ends up a centimetre to the left of anything in the dataset, a chunked policy predicts a confident chunk from a state it has never seen, and predicts it 100 steps at a time.

What the published numbers actually say

The RaC comparison is worth reading carefully rather than as a headline. Their Table 1 lines up three systems on similar shirt-hanging setups: ALOHA Unleashed, a 217M diffusion transformer, about 89 hours of data (5345 demonstrations), 75.0 percent success. Seed GR-3, a 4B VLA, 116 hours, roughly 63.6 percent. RaC, a 368M flow-matching transformer, 5 hours of expert plus recovery plus correction data, 78.3 percent. The paper says itself that it approximates the ALOHA Unleashed hours at one minute per trajectory and reads the GR-3 number off a figure, that the setups are only similar rather than identical, and all three are bimanual ALOHA-class rigs. None of this is an SO-100 with two USB webcams. The mechanism transfers; the numbers do not.

RACER needs the same care, in the opposite direction. Its real-world table is often quoted as an ablation of failure augmentation, and it is not one: every RACER variant in that table is trained on failure-augmented data. The 62.5 percent row is RACER minus the failure-analysis part of the rich language annotation, against 72.5 percent for the full model, over 40 episodes on four tasks with a Franka and no seed variance reported. What the augmented data itself buys is only visible against the RVT baseline at 25.0 percent, and that comparison changes the language conditioning and the supervisor-actor structure at the same time. Useful evidence, but not a clean isolation of recovery data.

Three ways to get failure onto disk

1. Perturb the operator during the demonstration

DART (Laskey et al., arXiv 1703.09327, Disturbances for Augmenting Robot Trajectories) injects noise into the supervisor's control stream while the human demonstrates. The human is then forced to show the way back, and the resulting dataset contains recovery states without anyone having to plan them. The paper optimises the noise covariance so that the disturbance matches the error the trained policy is expected to make. On MuJoCo Humanoid, collecting with DART cost 5 percent of the supervisor's cumulative reward, while DAgger-driven rollouts during collection cost 80 percent. On the physical grasping-in-clutter task with four human supervisors, DART averaged a 62 percent improvement over behaviour cloning.

2. Scripted disturbance episodes

RACER (Dai et al., arXiv 2409.14674) is the offline version of the same idea and it is the one that maps cleanly onto a hobby arm. Their pipeline perturbs the expert action at the keyframes that matter (aligning to an object, grasping it, releasing it) to induce a failure, then replays the original expert actions as the correction, so every perturbation yields a labelled recovery transition. They took 2250 RLBench expert episodes, perturbed each one five times, discarded the trajectories where the recovery did not succeed, and ended with 10,159 recovery episodes across 18 tasks. In their real-world Franka setup they collected 60 training demonstrations, 15 per task across four tasks, with three perturbations each.

• Nudge the target object 2 to 3 cm sideways after the reach has already started, then let the operator re-approach.
• Open the gripper one step early so the object drops, then pick it up again from wherever it landed.
• Start an episode from a pose the nominal set never begins in: gripper already low, wrist rotated, object behind the arm.
• Let a grasp slip on purpose and re-grasp without resetting the scene.
• Put a second object in the way mid-episode so the arm has to push it aside first.
• Drive the arm 3 cm past the correct approach and then correct back, which is the cheapest recovery segment there is.

The point of a scripted list is repeatability. If you can name the disturbance, you can record ten of them, and you can tell afterwards whether the checkpoint handles that specific one. ADC (Huang et al., arXiv 2503.11646) scales the same idea with a second person: while the teleoperator drives, an adversarial operator moves objects, changes conditions and rewrites the language instruction mid-episode. Their claim is that this compresses many failure-recovery behaviours into few episodes, and that models trained on 20 percent of the demonstration volume beat conventional full-volume datasets.

3. Intervene on a rollout

This is the on-policy version and it needs a trained checkpoint first, so it cannot be round zero. LeRobot 0.6.x ships it as a rollout strategy. The documented loop is: watch the policy run, trigger the pause input when failure looks imminent (the policy stops, the teleoperator moves to match the robot's position, and no frames are recorded during the pause), take over and teleoperate the recovery, then the correction, then hand control back. The docs name RaC's recover-then-correct decomposition as the protocol the dagger strategy follows, so Rule 1 is built in. Rule 2 is not: the documented loop resumes the same episode from the current state and lets you intervene again, which is exactly what RaC terminates early to avoid. If you want Rule 2 you end the episode yourself after the correction.

# LeRobot 0.6.1 (published on PyPI 2026-08-03). The HIL docs default to
# bi_openarm hardware; so_follower + so_leader are registered and CLI-selectable.
lerobot-rollout \
    --strategy.type=dagger \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_follower_arm \
    --robot.cameras='{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }' \
    --teleop.type=so_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=my_leader_arm \
    --policy.path=outputs/train/act_so100/checkpoints/last/pretrained_model \
    --dataset.repo_id=${HF_USER}/so100_recovery_round1 \
    --dataset.single_task="Put the red cube in the bowl" \
    --dataset.fps=30 \
    --strategy.num_episodes=30

1. 1
   Record the nominal set first

   You cannot record a recovery from a failure you have not seen yet. Record the clean demonstrations, keep the cameras fixed, keep the task string identical across every episode.

   bash

   lerobot-record \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_follower_arm \
       --robot.cameras='{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }' \
       --teleop.type=so101_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_leader_arm \
       --dataset.repo_id=${HF_USER}/so100_cube_nominal \
       --dataset.num_episodes=50 \
       --dataset.single_task="Put the red cube in the bowl" \
       --display_data=true

2. 2
   Record the disturbance episodes into a separate repo id

   Same task string, same cameras, different repo id. Separate ids keep the experiment reversible: if the recovery set makes things worse, you can retrain without it instead of hunting for episode indices.

   bash

   lerobot-record \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_follower_arm \
       --robot.cameras='{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }' \
       --teleop.type=so101_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_leader_arm \
       --dataset.repo_id=${HF_USER}/so100_cube_recovery \
       --dataset.num_episodes=20 \
       --dataset.single_task="Put the red cube in the bowl" \
       --dataset.episode_time_s=60 \
       --dataset.reset_time_s=30

3. 3
   Use the re-record key without mercy

   Right Arrow or n early-stops the current episode (or the reset phase) and moves to the next. Left Arrow or r cancels the current episode and re-records it. Escape or q stops the session, encodes the videos and uploads the dataset. If a disturbance episode turned into a genuine mess rather than a clean recovery, press Left Arrow. The robomimic result is the reason. These are the documented bindings for lerobot-record; the rollout script prints its own controls, and the docs tell you to trust those rather than any tutorial.

4. 4
   Train round zero

   Train the first checkpoint on the nominal set alone, so you have a baseline failure mode to aim the next round at.

   bash

   lerobot-train \
       --dataset.repo_id=${HF_USER}/so100_cube_nominal \
       --policy.type=act \
       --output_dir=outputs/train/act_so100 \
       --job_name=act_so100_r0 \
       --policy.device=cuda \
       --steps=20000

5. 5
   Roll out, intervene, retrain on the combination

   Run the dagger strategy against that checkpoint, then fine-tune from the previous checkpoint rather than from scratch, which is what the LeRobot HIL page does between rounds and what RaC does between rounds. Note the flag: --policy.path loads a policy for a rollout, --policy.pretrained_path is what the fine-tuning example passes to the trainer.

   bash

   lerobot-train \
       --dataset.repo_id=${HF_USER}/so100_recovery_round1 \
       --policy.type=act \
       --policy.pretrained_path=outputs/train/act_so100/checkpoints/last/pretrained_model \
       --output_dir=outputs/train/act_so100_r1 \
       --policy.device=cuda \
       --steps=20000

How much, and when to stop

There is no published episode count for an SO-100 doing your task, and anyone who gives you one is guessing. What the literature does give you is a protocol shape: collect a round-zero budget of full demonstrations, train, look at where the rollout leaves the demonstrated path, then spend the next budget on that specific failure. RaC formalises this as alternating rounds of intervention and retraining against a fixed data-collection budget, and reports a steeper success-rate-versus-data-size slope than either batched demonstration collection or HG-DAgger style collection.

Two ways to run this

pip install lerobot

# find the serial ports for follower and leader
lerobot-find-port

# check the cameras and dump one test frame per device
lerobot-find-cameras

# confirm the exact version you are reading docs for
pip show lerobot | head -2

What recording recovery does not fix

Recovery is reactive behaviour, which makes it the first thing that inference latency destroys. The control loop on this platform runs from 20 ms per action step for ACT to 485 ms for Pi0.5, with GR00T N1.7 at 152 ms and SmolVLA at 245 ms. Add public-internet round trips on top and a policy that would have caught a slipping cube instead notices it two chunks later. Remote inference is workable for slow pick-and-place and not for fast reactive motion. If your recovery behaviour needs to happen within 200 ms of the disturbance, inference has to sit next to the servos.

• A LeRobot v3.0 dataset crashes the GR00T loader and has to be converted down to v2.1 first. Your recovery episodes are recorded in whatever version your client writes, so check before the round starts, not after (dataset rejected as v3).
• GR00T runs are not bit-for-bit reproducible because the fine-tuning entry point exposes no seed, so a round-to-round comparison on GR00T carries seed noise you cannot remove.
• Five joints is five joints. If the recovery physically requires a wrist roll the arm does not have, no amount of data creates it (degrees of freedom).
• If the policy works on your table and nowhere else, that is a generalisation problem and recovery data is the wrong tool (policy only works in one setup).
• If the loss falls to nothing and the arm does not move, stop and look there first (loss falls, policy does nothing).
• A policy that stalls mid-motion may be modelling pauses in your demonstrations rather than lacking recovery data. The ACT authors identify pauses as a temporally correlated confounder that single-step policies handle badly (policy freezes mid-motion).

The protocol, compressed

1. Record the nominal set to the trainer's minimum and no further. Fifty episodes for ACT, GR00T or Pi0.5, thirty for SmolVLA.
2. Train round zero and watch twenty rollouts without touching anything. Write down where the arm leaves the path.
3. Write a disturbance list of three to six named perturbations that produce that state.
4. Record disturbance episodes into their own repo id, one perturbation type at a time, re-recording anything that turns into a mess.
5. For every intervention: recover first, correct second, then end the episode. Do not let the rollout run on after a takeover.
6. Retrain from the previous checkpoint on the combined data rather than from scratch.
7. Evaluate against the same twenty starting conditions. Count retry attempts, not just successes.
8. Stop when the failure mode changes each round rather than repeating. That is the signal that the hole is filled and something else is now the bottleneck.

Further reading on this site: how to collect high-quality VLA training data covers the nominal set this protocol sits on top of, the SO-100 complete guide covers the hardware and calibration, and the VLA overview covers what the pretrained models bring before you fine-tune them. If you want to see which architectures have been measured against each other, the arena lists 85 VLA models with 332 benchmark results, each value linked to its paper or model card.