Reading Loss Curves in Imitation Learning

You start a fine-tune, the loss drops from 1.2 to 0.08 in the first two thousand training steps, flattens, and does nothing for eighteen thousand more. Is it done? Broken? Should you have stopped at step 6000? With no validation curve on the chart, the loss cannot tell you. This article is about what it can.

The context is imitation learning on a small arm: 30 to 200 episodes recorded by teleoperation, a policy fine-tuned on a rented GPU for a few hours, then run back on the hardware. That is the loop the training guides here describe, and where loss curves get misread most. Everything below is checked against LeRobot v0.6.1 (3 August 2026) and Isaac-GR00T at commit 51d4c89 (20 August 2026).

What the training loss actually measures

A behaviour-cloning trainer samples a batch of frames from your LeRobot dataset, asks the model to predict the demonstrated action chunk, and reduces the error. The important word is samples. The loss is an expectation over independently drawn frames. A rollout is not: the observation at step t+1 depends on the action taken at step t. Ross, Gordon and Bagnell put it plainly in the DAgger paper: sequential prediction problems where future observations depend on previous predictions violate the i.i.d. assumptions supervised learning rests on.

That mismatch is the whole story. Your loss is computed on states a human produced. Your policy runs on states it produced itself. The ACT paper describes the same failure from the practical side: errors in the policy can compound over time, and human demonstrations can be non-stationary. A loss curve is blind to both, because every frame it sees came from the human.

Why there is usually no validation curve

For most of LeRobot's life there was no held-out loss at all. That changed recently and it is worth dating precisely, because half the tutorials on the internet still describe the old behaviour. Pull request 3824, "Add inline offline validation with train/eval split", was merged on 25 June 2026 and shipped in v0.6.0 on 6 July 2026. It added dataset.eval_split, eval_steps and max_eval_samples. LeRobot v0.5.1, released 7 April 2026, has none of them.

When you do turn it on, read the split rule before you trust the curve. The docstring of make_train_eval_datasets in LeRobot's dataset factory says: the last ceil(n_episodes * eval_split) episodes per task are held out. That is chronological, not random. Recording sessions drift, so your held-out episodes are the ones where the lighting had changed, your wrist had tired and the object had migrated two centimetres left. A rising eval loss there may be measuring your session, not your model.

# LeRobot v0.6.x: hold out the last 10 percent of episodes per task
lerobot-train \
  --dataset.repo_id=${HF_USER}/so100_pick_place \
  --dataset.eval_split=0.1 \
  --eval_steps=1000 \
  --max_eval_samples=512 \
  --policy.type=act \
  --steps=100000 \
  --save_freq=5000 \
  --log_freq=100 \
  --output_dir=outputs/train/act_so100 \
  --job_name=act_so100 \
  --wandb.enable=true

step:2K smpl:64K ep:2K epch:2.13 loss:0.083 grdn:1.204 lr:1.0e-04 updt_s:0.412 data_s:0.003 smp/s:78 mem_gb:41.20
step:2K smpl:65K ep:2K epch:2.16 loss:0.081 grdn:0.998 lr:1.0e-04 updt_s:0.409 data_s:0.002 smp/s:78 mem_gb:41.20
step 1000: eval_loss=0.0912

The number that decides your run is not on the chart

The strongest evidence is still the robomimic study, What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, Mandlekar et al., CoRL 2021. They list it as challenge C4, "Train Objective is not Eval Objective", and their phrasing is blunt: policies are trained with surrogate losses, which makes it hard to know which checkpoints are good without trying each one directly on the robot.

This is also the mechanism behind the most common support question we see, which has its own page at loss falls, policy does nothing. The loss went where it should and the arm still sits there. Nothing is broken; the objective never promised what you wanted. The related pattern, a policy that works in the recorded setup and nowhere else, is under policy only works in one setup.

Curve shapes and what they actually mean

The right-hand column is the part people get wrong: most bad-looking curves do not justify killing a run, and one good-looking curve does.

A lot of curve shape is schedule shape rather than model behaviour. ACT returns None from get_scheduler_preset, so it trains at a constant 1e-5 and its loss creeps down for a very long time. SmolVLA and Pi0.5 use a cosine decay with warmup that auto-scales down when your step count is below scheduler_decay_steps, default 30000. GR00T N1.7 uses lr_scheduler_type "cosine" with warmup_ratio 0.05. In all three the last few thousand steps flatten because the learning rate is nearing its floor, not because the model stopped improving.

What to do instead: an evaluation ladder

Since the loss will not select your checkpoint, something else has to. The practical answer is a ladder, cheapest first, where each rung filters candidates for the next.

1. 1
   Rung 1: sanity checks on the training log

   A liveness signal, not a quality one. Confirm the run is learning and the epoch count is sane, in the first ten minutes rather than at the end.

   bash

   # is the run alive, and how many passes over the data has it done?
   grep -E 'epch:|eval_loss' outputs/train/act_so100/train.log | tail -20

2. 2
   Rung 2: held-out loss, if your trainer has it

   On LeRobot v0.6.0 or newer. Treat a rising eval_loss as a prompt to investigate, not a stopping rule, and remember the split is chronological.

   bash

   lerobot-train \
     --dataset.repo_id=${HF_USER}/so100_pick_place \
     --dataset.eval_split=0.1 \
     --eval_steps=500 \
     --max_eval_samples=256 \
     --policy.type=smolvla \
     --policy.pretrained_path=lerobot/smolvla_base \
     --steps=20000 \
     --policy.scheduler_decay_steps=20000 \
     --save_freq=2000

3. 3
   Rung 3: open-loop evaluation against recorded trajectories

   Feed a held-out episode's recorded observations to the checkpoint and compare the predicted chunk against what the human did. Not success, but it catches gross failures before you burn arm time.

   bash

   uv run python gr00t/eval/open_loop_eval.py \
     --dataset-path /data/so100_pick_place \
     --embodiment-tag NEW_EMBODIMENT \
     --model-path /checkpoints/step-8000 \
     --traj-ids 0 1 2 \
     --execution-horizon 16 \
     --denoising-steps 4

4. 4
   Rung 4: rollouts on the arm

   The only signal that answers your question. Run each surviving checkpoint the same number of times from the same start states and count completions. Ten trials each is a usable minimum.

   bash

   # same start pose, same object placement, count successes out of 10
   for ck in 6000 8000 10000 12000; do
     echo "checkpoint $ck"
     # start the policy server on this checkpoint, then run 10 scripted attempts
   done

Rung 3 is the least known, so here are its details. The script prints "Unnormalized Action MSE across single traj" and "Average MSE across all trajs", plus MAE variants, and writes a plot of ground truth against predicted actions to /tmp/open_loop_eval/traj_{traj_id}.jpeg. --save-plot-path overrides that with the literal filename you give it, for every trajectory, so with several --traj-ids the plots overwrite each other; leave it off. Defaults: execution_horizon 16, denoising_steps 4, steps 200, traj_ids [0]. The plot beats the scalar. You see at a glance whether the gripper channel is flat, whether a joint sits at a limit, and whether the prediction lags.

One thing to check before comparing checkpoints: make sure they still exist. Isaac-GR00T's FinetuneConfig defaults to save_steps 1000 with save_total_limit 5, so an unattended run keeps the last five and silently deletes everything earlier. Raise it at launch time, not after the run.

Two ways to get a checkpoint you can trust

Using the loss curve as your primary signal

What it costs to be wrong

People over-read loss curves because the alternative feels expensive. On the spot market this platform uses, an A100 80 GB or H100 run for GR00T N1.7, GR00T N1.5 or Pi0.5 takes 3 to 6 hours at 1.20 to 2.00 USD per hour, roughly 4 to 12 USD per run. A 24 GB card for SmolVLA or ACT takes 2 to 5 hours at 0.30 to 0.60 USD per hour, roughly 1 to 3 USD. Full numbers on the pricing page.

Against that, an afternoon spent staring at a flat line, shipping the final checkpoint because it had the lowest loss, and concluding the model does not work is far more expensive. Retraining is cheap. Evaluating properly is what costs, and it costs arm time rather than GPU time.

• Minimum episodes before a run is worth attempting: 50 for GR00T N1.7, GR00T N1.5, Pi0.5 and ACT, 30 for SmolVLA. Below that the loss still falls and the policy still does not work.
• GR00T and Pi0.5 are cloud-only here. SmolVLA and ACT also run locally, the sensible choices while you iterate on evaluation method.
• ACT has no base model, so there is no zero-shot baseline to compare a checkpoint against.

A checkpoint protocol you can actually follow

The rules that survive contact with a real arm are short. Recording is covered in record your first dataset and running in run your first policy. What follows is only the part about deciding when a run is done.

To run that shortlist without owning hardware, the live arm streams a physical SO-100 with no signup, and the try page lists the three ways to start. The same operations are exposed to a terminal through the CLI and to agents through the MCP server. Public datasets are in the dataset directory.

Related reading

On how much data has to exist before a loss curve means anything, BC-Z and what scale really means is the read, and how to collect high quality VLA training data covers recording. For the hardware end, start with the SO-100 complete guide and the SO-100 robot page. Fine-tuning, action chunking and flow matching have short glossary entries if any of those terms were new.