Why a Policy Fails at Inference but Not in Training

The loss curve did what a loss curve is supposed to do. It fell hard for the first few thousand training steps, flattened, and stayed flat. You pull the last checkpoint, put the cube back on the table, start the arm, and it lifts five centimetres, drifts left and closes the gripper on air.

That is the normal outcome of a first policy, and it is usually not a bug in the training job. Training and inference are two different data distributions, two different clocks and often two different image pipelines; the loss sees one side of that. Below is what actually differs on an SO-100 class arm, cheapest check first.

The loss is measuring a different question

Supervised learning has a comfortable property: validation loss ranks your checkpoints, so early stopping works. Imitation learning breaks it. What you care about is task success under the states the policy itself visits; the loss is computed on the states the human visited. Mandlekar and colleagues measured that gap in the robomimic study (CoRL 2021), on simulated tasks where every checkpoint could be rolled out and scored.

Success rates in percent for BC-RNN over three seeds, 30 percent held out. The other half of the finding matters as much: best validation loss arrived early, around epoch 100 to 300, while best success arrived much later and kept climbing as validation loss climbed with it. Their summary is the sentence to keep: validation loss is a poor measure of policy performance.

There is a version of this specific to GR00T N1.7. In the Isaac-GR00T training config, eval_strategy defaults to "no", so a fine-tune reports no validation loss unless you pass --eval-strategy steps --eval-steps 500. The falling number in your terminal is training loss on data the model is memorising. That run also has no seed, so two identical commands do not give identical weights.

Why small errors turn into a missed grasp

The formal statement is older than any model here. Ross, Gordon and Bagnell showed that a policy with expected loss eps under the expert's own state distribution has cost bounded by the expert's plus T squared times eps over a horizon of T steps, and that the bound is tight: there are problems where the extra cost really does grow with the square of episode length.

Check 1: replay a recorded episode before you blame the model

Two minutes, and it settles the biggest question at once: is the failure in the network, or in everything around it. lerobot ships lerobot-replay, which pushes the recorded actions of one episode back onto the robot with no policy in the loop.

1. 1
   Rebuild the scene exactly as recorded

   Same table position, same object start pose, same lamp. You are reproducing a recording, not testing generalisation.

2. 2
   Replay episode 0 of the dataset you trained on

   The arm should complete the task. If it does not, the problem is upstream of the model. The actions come straight out of the LeRobot dataset you trained on.

   bash

   lerobot-replay \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=black \
       --dataset.repo_id=${HF_USER}/my-so100-dataset \
       --dataset.episode=0

3. 3
   Verify the transport without a model on the GR00T side

   Isaac-GR00T's policy server has the same idea built in. Give it --dataset-path and omit --model-path and it loads a ReplayPolicy, serving recorded actions over the real transport with no weights involved. NEW_EMBODIMENT has no built-in modality config, so --modality-config-path is not optional here, and --execution-horizon is required whenever --dataset-path is set.

   bash

   # ReplayPolicy: no model, real transport
   uv run python gr00t/eval/run_gr00t_server.py \
       --dataset-path ./demo_data/cube_to_bowl_5 \
       --embodiment-tag NEW_EMBODIMENT \
       --modality-config-path examples/SO100/so100_config.py \
       --execution-horizon 16 \
       --port 5555

Check 2: open-loop error against episodes the model has seen

A narrower question: given observations from a real episode, does the checkpoint predict roughly the actions the human produced? Entirely offline, and it catches the whole statistics bucket: wrong normalisation, wrong embodiment tag, wrong action space, a fine-tune that silently ran on a different joint order.

# Isaac-GR00T open-loop evaluation, one trajectory
uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path ./demo_data/cube_to_bowl_5 \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path /tmp/so100/checkpoint-2000 \
    --traj-ids 0 \
    --execution-horizon 16 \
    --steps 400 \
    --modality-keys single_arm gripper

Read the per-joint plot, not the single MSE number. NVIDIA deliberately publishes no target MSE here, because the demo set is five episodes and yours will differ, but they do publish the shape of a healthy run from one H100 at max-steps 2000. The GR00T N1.7 on SO-100 guide walks the same run.

• A flat or constant prediction curve means the action keys are not mapped. Check modality.json and --modality-config-path, not the learning rate.
• Enormous or NaN MSE means normalisation. Verify meta/stats and that your action ranges are physically sane.
• MSE flat or rising across checkpoints means the trainer never saw your data: wrong dataset path, or a dataloader serving nothing.
• Good on the training trajectory, poor on held-out episodes is data scarcity. The minimums on the policy pages are 50 episodes for GR00T, Pi0.5 and ACT, 30 for SmolVLA.

Check 3: the cameras are the most likely liar

In current lerobot the rollout entry point compares the visual features the policy declares against the cameras the robot reports, and raises before the arm moves if they do not line up.

Visual feature mismatch between policy and robot hardware.
Policy expects: {'observation.images.top', 'observation.images.wrist'}
Robot provides: {'observation.images.front', 'observation.images.wrist'}
Use --rename_map to map camera names, e.g. --rename_map='{"observation.images.top": "observation.images.cam0"}'

Read that guard carefully, because its condition is narrower than it looks. It fires only when neither key set is a subset of the other, and it is skipped entirely when you pass a --rename_map. A robot providing a strict subset of the expected cameras passes, and a robot with correct names but the cameras swapped between USB indices also passes, because names are all the guard can see.

• Same lens, different white balance. An auto-exposure webcam re-negotiates on each plug-in, so the deployment image is tinted against the recording.
• Same camera, different resolution. A 640x480 recording served at 1280x720 is resized, and many UVC webcams crop differently between capture modes, so the framing is not what the model trained on.
• Same camera, remounted slightly off its old pose. The most expensive shift on this list, and the next section has the number.
• Correct names, stale normalisation. Checkpoints trained before lerobot moved normalisation into processor pipelines carry stats inside the model state dict; migrate_policy_normalization.py converts them.

Check 4: the scene moved, and some moves cost far more than others

This is what people mean when they say the policy only works in one setup, and there is measurement behind it. Xie, Lee, Xiao and Finn ran a language-conditioned manipulation policy on a real robot across controlled single-factor shifts and reported each factor separately, which makes them rankable.

Two details matter. The camera row collapsed because the whole training dataset used one fixed head pose, which is how most people record an SO-100 dataset. And the same patterned paper was used for both background and table texture, so 88.9 against 52.8 percent is like for like: the surface the object sits on matters far more than the wall behind it. Most factor pairs also failed to compound, so chase the dominant factor.

Check 5: the clock, which is where 'it just freezes' comes from

A policy producing correct actions at the wrong rate looks broken in a way that has nothing to do with its weights. Three clocks must agree: the dataset frame rate, the control loop rate, and the model's own inference latency. The third is fixed by the model you picked.

Only ACT keeps up with a 30 fps control loop unaided. The others survive on action chunking: one forward pass produces a block of future actions and the arm executes it while the next pass runs. The GR00T N1.7 against Pi0.5 comparison is mostly a story about that number.

# Deploy a trained policy on a real arm
lerobot-rollout \
    --strategy.type=base \
    --policy.path=outputs/train/my_act_run/checkpoints/last/pretrained_model \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --task="pick up cube" --duration=60 --display_data=true

# Slow VLA: real-time chunking instead of one call per tick
lerobot-rollout \
    --strategy.type=base \
    --policy.path=lerobot/pi05_base \
    --inference.type=rtc \
    --inference.rtc.execution_horizon=10 \
    --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
    --task="pick up cube" --duration=60

Chunk length is where most apparent freezing comes from. lerobot's ACT configuration defaults to chunk_size 100 and n_action_steps 100, with temporal_ensemble_coeff at None, so temporal ensembling is off. At 30 fps that is 3.33 seconds of motion committed from one observation, and an object nudged 200 ms into a chunk is simply not seen, which reads exactly like a policy freezing mid-motion.

The ACT paper measured this directly: averaged over four settings, success went from 1 percent at chunk size 1 to 44 percent at chunk size 100, then tapered at 200 and 400 as reactivity was lost. Temporal ensembling, which needs n_action_steps set to 1, added 3.3 percent for ACT. The cheap dial is to shorten n_action_steps below chunk_size: the model still predicts 100 steps, you execute the first 25, and the arm re-looks four times as often.

Check 6: how many trials before the number means anything

Once the plumbing is clean you will want to compare checkpoints, and this is where real-robot evaluation quietly falls apart. A success rate on a physical arm is a binomial estimate from a tiny sample, and the exact Clopper-Pearson 95 percent interval is wide enough that most reported comparisons are not comparisons at all.

Read the last three rows together. Holding the estimate at 70 percent, going from 10 trials to 50 shrinks the interval from 58.6 points to 26.7. A checkpoint scoring 7 of 10 and one scoring 5 of 10 have heavily overlapping intervals, so picking the first is a coin flip wearing a lab coat. Kress-Gazit and colleagues (2024) argue for reporting run counts, initial conditions and failure modes rather than a bare success rate; Vincent and colleagues (2024) bound the whole performance distribution from as few rollouts as possible.

Running the diagnosis: by hand or on AY-Robots

pip install 'lerobot[core_scripts]'

# 1. plumbing
lerobot-replay --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
    --dataset.repo_id=${HF_USER}/my-so100-dataset --dataset.episode=0

# 2. what the robot actually reports
lerobot-find-cameras
lerobot-info

# 3. deploy with a shorter chunk so the arm re-looks sooner
lerobot-rollout --strategy.type=base \
    --policy.path=outputs/train/my_run/checkpoints/last/pretrained_model \
    --policy.n_action_steps=25 \
    --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
    --task="pick up cube" --duration=60

When the honest answer is that the data cannot support the task

Some failures survive every check above. The policy is then telling you something true about the dataset, and the tell is behavioural: confident and consistently wrong in the same place, rather than flailing.

• The task needs information no camera captures. A grasp that depends on seeing behind the object cannot be learned from one overhead view, however many episodes you record.
• The demonstrations are multi-modal in a way the loss averages away. Two operators approached from opposite sides and the policy learned the mean, which goes through the object.
• The policy learned a shortcut that holds only in your data. de Haan, Jayaraman and Levine named this causal misidentification: the training procedure ignores the causal structure of the demonstration, so more information can make behaviour cloning worse.
• The demonstrations are too clean. Nothing shows recovery from a bad approach, so the compounding-error bound stays quadratic.

If the diagnosis lands here, the fix is a recording session, not a training run. DAgger's argument was that the valuable new data is expert labels on the states your policy visits when it goes wrong. lerobot implements this as --strategy.type=dagger, where a human on the leader arm takes over when the policy drifts. That mode records the corrections only; --strategy.record_autonomous=true records the autonomous phase too. Our notes on collecting high-quality VLA training data, on what scale really means in BC-Z and the full SO-100 guide cover what to vary.

# Collect corrections on the states your policy actually visits
lerobot-rollout \
    --strategy.type=dagger \
    --strategy.num_episodes=20 \
    --policy.path=outputs/train/my_run/checkpoints/last/pretrained_model \
    --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
    --teleop.type=so100_leader --teleop.port=/dev/ttyACM1 \
    --dataset.repo_id=${HF_USER}/my-so100-dagger \
    --dataset.single_task="Grab the cube"

The order, one more time

1. 1
   Replay a recorded episode

   No policy involved. If this fails, fix calibration or geometry and stop.

   bash

   lerobot-replay --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
       --dataset.repo_id=${HF_USER}/my-so100-dataset --dataset.episode=0

2. 2
   Look at one frame from every camera

   Catches the swap the name-based guard cannot see.

   bash

   lerobot-find-cameras

3. 3
   Run open-loop eval on a training episode

   A flat prediction curve is a mapping problem, not a training problem.

4. 4
   Shorten the chunk

   If the arm commits multi-second blocks, cut n_action_steps well below chunk_size before touching the weights.

   bash

   lerobot-rollout --strategy.type=base --policy.n_action_steps=25 \
       --policy.path=outputs/train/my_run/checkpoints/last/pretrained_model \
       --robot.type=so100_follower --robot.port=/dev/ttyACM0 --task="pick up cube"

5. 5
   Vary one factor at a time

   Table surface alone, lighting alone, start pose alone, 20 trials each.

6. 6
   Only now, retrain

   More variation along the factor that failed, or DAgger corrections.

None of these checks are clever, only cheap. The order matters because each eliminates a whole class of explanation, so the expensive lever gets pulled last. If you have no arm on the desk to reproduce the failure on, the three ways to start and the live arm put you on a real one without a signup.