Choosing a Checkpoint Without a Validation Loss

The number your trainer prints is not the number you care about

You fine-tuned a GR00T N1.7 policy on 60 recorded episodes of your SO-100 picking up a cube. Twenty thousand training steps later the loss has fallen from 1.2 to 0.04 and the output directory holds five checkpoints. Which one goes on the robot?

The trainer did not tell you, and in most robot learning stacks it structurally cannot. Supervised learning solves model selection by holding out data and taking the lowest validation loss. Imitation learning breaks that, because what you minimise (per-timestep agreement with a demonstrator) and what you want (task success in a closed loop) come apart.

The lerobot row changed recently. On the Pi0.5 path the platform runs lerobot 0.5.1, where eval_freq means "run the policy in a gym environment every N steps". A real SO-100 dataset has no gym environment, so the flag does nothing. Version 0.6.0 added an offline train/eval split (pull request 3824), the first upstream held-out loss for a real robot dataset.

The evidence: lowest validation loss is close to the worst rule available

The reference measurement is Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021). They evaluated every checkpoint in simulation, so they had ground truth for what each selection rule would have picked. Table 29 is the result.

Success rates in percent for BC-RNN, three seeds each. Read the first row again: the validation-loss rule picked a 7.3 percent policy out of a run containing an 84 percent one, and blindly taking the last checkpoint was ten times better. The project page states the general result: the best validation policy is "50 to 100% worse than the best performing policy".

Diffusion Policy (Chi et al., 2023) reacted by refusing to report a single number. Every benchmark entry is written as (max performance) / (average of last 10 checkpoints), each averaged over 3 seeds and 50 environment initial conditions. The distance between the two columns is the size of the checkpoint-selection problem.

A 42 point spread between the best checkpoint a run produced and a typical late one is larger than most published architecture differences. The same paper notes that Implicit BC was so unstable its authors evaluated every checkpoint and reported the best, which in the real world "necessitates the evaluation of many policies on hardware".

The three signals you actually have

A policy fine-tuned on your own SO-100 data leaves exactly three sources of information, in increasing order of cost and usefulness.

1. Training loss. Free, and almost useless for ranking. It says the run did not diverge and the loader is feeding real data. A loss that falls to near zero while the arm does nothing has its own page: loss falls, policy does nothing.
2. Open-loop error on held-out episodes. Runs on the GPU pod that just finished training, no robot needed. Feed the policy observations it never saw and compare its actions to the human's, as MSE and MAE in joint units.
3. Closed-loop rollouts on the real arm. Slow, noisy, and the only thing that answers the question. Everything above it is triage.

DAgger (Ross, Gordon and Bagnell, 2011) showed that a policy trained only on expert states accumulates error once its own mistakes push it off that distribution. Simchowitz et al. (2025) sharpened it for continuous actions: any smooth deterministic imitator "necessarily suffers error on execution that is exponentially larger, as a function of problem horizon, than the error under the distribution of expert training data". Your held-out MSE measures the small quantity; the robot experiences the large one.

Step 1: hold out episodes before you press train

None of this works if every episode went into training. You need a slice of your LeRobot dataset the model never saw, held out at episode granularity. Frames within one episode are near-duplicates at 30 Hz, so a random frame split leaks the answer.

1. 1
   lerobot 0.6.0 and later: use the built-in split

   dataset.eval_split holds out a fraction of episodes per task; eval_steps sets how often the held-out loss is computed. Setting eval_steps without eval_split aborts the run with a config error; eval_split on its own is legal and just holds the episodes back.

   bash

   pip install 'lerobot==0.6.1'
   
   lerobot-train \
     --dataset.repo_id=your-name/so100_cube_pick \
     --policy.type=act \
     --output_dir=outputs/act_cube \
     --steps=100000 \
     --save_freq=5000 \
     --dataset.eval_split=0.15 \
     --eval_steps=2000 \
     --max_eval_samples=2000 \
     --seed=1000

2. 2
   lerobot 0.5.1: split by hand

   0.5.1 has no eval_split, but dataset.episodes lets you name the training episodes explicitly and leaves the rest of the dataset untouched. dataset.exclude_episodes is the inverse of the same knob, but it is not in 0.5.1, 0.6.0 or 0.6.1 - it only exists on main, so do not reach for it on a released version.

   bash

   TRAIN_EPS=$(python -c "print(list(range(51)))" | tr -d ' ')
   
   lerobot-train \
     --dataset.repo_id=your-name/so100_cube_pick \
     --dataset.episodes="$TRAIN_EPS" \
     --policy.type=smolvla \
     --output_dir=outputs/smolvla_cube \
     --steps=20000 \
     --save_freq=2000 \
     --seed=1000

3. 3
   Isaac-GR00T: split the dataset on disk

   launch_finetune.py exposes no episode filter and no eval split, so the split must exist in the filesystem. GR00T needs LeRobot v2.0 or v2.1; a v3.0 dataset crashes its loader and must be converted down before the split.

   bash

   cp -r ~/.cache/huggingface/lerobot/so100_cube_pick /data/cube_train
   cp -r ~/.cache/huggingface/lerobot/so100_cube_pick /data/cube_heldout
   
   # then edit meta/episodes.jsonl in each copy so that
   # cube_train holds episodes 0-50 and cube_heldout holds 51-59,
   # dropping the matching parquet and video files on each side.

4. 4
   Record enough that a split is affordable

   Holding out 15 percent of 30 episodes rounds up to 5 held out and leaves 25 for training, under the minimum for most policies. The stated minimums are 50 episodes for GR00T N1.7, GR00T N1.5, Pi0.5 and ACT, and 30 for SmolVLA. Plan the slice on top.

   bash

   python -c "
   from lerobot.datasets.lerobot_dataset import LeRobotDataset
   ds = LeRobotDataset('your-name/so100_cube_pick')
   print('episodes:', ds.num_episodes, 'frames:', ds.num_frames, 'fps:', ds.fps)
   "

Step 2: score every surviving checkpoint open-loop

Isaac-GR00T ships the tool. gr00t/eval/open_loop_eval.py loads a checkpoint, replays recorded trajectories and prints unnormalized MSE and MAE against ground-truth actions, plus per-joint plots. On main as of 24 August 2026 its defaults are steps=200, traj_ids=[0], execution_horizon=16 and denoising_steps=4.

uv run python gr00t/eval/open_loop_eval.py \
  --dataset-path /data/cube_heldout \
  --embodiment-tag NEW_EMBODIMENT \
  --model-path /tmp/so100_finetune/checkpoint-10000 \
  --traj-ids 0 \
  --execution-horizon 16 \
  --steps 400

One trajectory is not a measurement. Pass every held-out episode index to --traj-ids and loop over every checkpoint; the script averages across trajectories, so a small shell loop turns a directory of checkpoints into a ranked list.

#!/usr/bin/env bash
set -euo pipefail

RUN=/tmp/so100_finetune
HELDOUT=/data/cube_heldout
TRAJS="0 1 2 3 4 5 6 7 8"
mkdir -p /tmp/oleval

for ckpt in "$RUN"/checkpoint-*; do
  step=$(basename "$ckpt" | cut -d- -f2)
  echo "=== step $step ==="
  uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path "$HELDOUT" \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path "$ckpt" \
    --traj-ids $TRAJS \
    --execution-horizon 16 \
    --steps 400 2>&1 \
  | tee "/tmp/oleval/$step.log" | grep -Ei 'average m(s|a)e'
done

# rank the checkpoints by mean MSE over the held-out episodes
for f in /tmp/oleval/*.log; do
  mse=$(grep -o 'Average MSE across all trajs: [0-9.e+-]*' "$f" | awk '{print $NF}')
  printf '%s  %s\n' "$mse" "$(basename "$f" .log)"
done | sort -g

For ACT and SmolVLA there is no equivalent upstream script, because lerobot's lerobot-eval expects a gym environment. On 0.6.x the held-out eval_loss printed every eval_steps is the closest thing. Same signal, same caveat: reject with it, do not rank with it.

Step 3: spend rollouts where they change the decision

Open-loop scoring gets you from ten candidates to two or three. The rest is arithmetic about binomial proportions, and this is where hobby projects and a fair share of papers go wrong: a rollout either succeeds or it does not, so you are estimating a proportion, and the uncertainty is wider than intuition suggests.

Ten rollouts at 70 percent is compatible with a true rate anywhere from a third to nearly always. The Toyota Research Institute large behavior model project page puts it in one sentence: "With 50 rollouts, for example 10 rollouts each on 5 behaviors, the resulting CI width is generally 20% to 30% absolute success rate, making all but the largest-sized effects impossible to reliably measure." They add that "many robotics papers may be measuring statistical noise due to insufficient statistical power".

from scipy.stats import beta, fisher_exact

def clopper_pearson(k, n, alpha=0.05):
    lo = 0.0 if k == 0 else beta.ppf(alpha / 2, k, n - k + 1)
    hi = 1.0 if k == n else beta.ppf(1 - alpha / 2, k + 1, n - k)
    return lo, hi

for n in (10, 20, 30, 50, 100, 200):
    k = round(0.7 * n)
    lo, hi = clopper_pearson(k, n)
    print(f"n={n:3d}  {k/n:.0%}  [{lo:.3f}, {hi:.3f}]  width={100*(hi-lo):.0f} pp")

# smallest gap reaching p < 0.05 against a 50% baseline
for n in (10, 20, 30, 50, 100, 200):
    kb = round(0.5 * n)
    for ka in range(kb, n + 1):
        if fisher_exact([[ka, n - ka], [kb, n - kb]])[1] < 0.05:
            print(f"n={n:3d}: challenger must hit {ka}/{n} = {ka/n:.0%}")
            break

With ten rollouts each you cannot distinguish a 50 percent checkpoint from an 80 percent one. The consequence is liberating: if you can afford 20 rollouts per candidate, do not test five checkpoints with four each. Test two with 20 each, and filter hard enough open-loop that only two survive.

A protocol that fits an SO-100 and an afternoon

One arm, one task, and a run that produced five to twenty checkpoints. This finishes before dinner and is honest about its limits.

1. 1
   Reject on open-loop error, keep the top three

   Score every checkpoint on all held-out episodes. Discard anything whose MSE is more than roughly double the best, and anything whose per-joint plot shows a channel flatlining, usually the gripper. Do not rank inside the top three.

   bash

   for f in /tmp/oleval/*.log; do
     mse=$(grep -o 'Average MSE across all trajs: [0-9.e+-]*' "$f" | awk '{print $NF}')
     printf '%s  %s\n' "$mse" "$(basename "$f" .log)"
   done | sort -g | head -3

2. 2
   Add the last checkpoint as a free fourth candidate

   The robomimic numbers make it a strong default: 74.0 percent against 84.0 for the best and 7.3 for the validation-loss rule. It costs nothing and protects you when the open-loop filter misleads.

3. 3
   Fix the scene, then interleave

   Mark 10 named object start positions with tape and shuffle a schedule of (position, checkpoint) pairs before touching the arm. Serve every candidate from the same endpoint so inference latency is identical.

   python

   import random
   positions = [f"P{i}" for i in range(10)]
   checkpoints = ["ckpt-8000", "ckpt-16000", "ckpt-20000", "last"]
   schedule = [(p, c) for c in checkpoints for p in positions]
   random.seed(0)
   random.shuffle(schedule)
   for i, (p, c) in enumerate(schedule, 1):
       print(f"{i:3d}  place at {p}  run {c}")

4. 4
   Log binary outcomes plus a failure tag

   Success or failure only; a partial success is a failure. Add one word for how it failed. The distribution of failure modes is often more informative than the rate.

   bash

   echo "$(date -Iseconds),P3,ckpt-16000,0,stalled" >> rollouts.csv

5. 5
   Compute the interval before announcing a winner

   Feed the counts into the function above. If your top two intervals overlap heavily you did not find a winner, you found two checkpoints you cannot separate on this budget: ship the faster one and say so. Reuse the same slice and the same 10 positions next run; the training docs and the CLI let you pin the dataset revision.

Two ways to run this

git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
uv sync --python 3.12

CUDA_VISIBLE_DEVICES=0 uv run python \
  gr00t/experiment/launch_finetune.py \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path /data/cube_train \
  --embodiment-tag NEW_EMBODIMENT \
  --modality-config-path examples/SO100/so100_config.py \
  --num-gpus 1 \
  --output-dir /tmp/so100_finetune \
  --max-steps 20000 \
  --save-steps 2000 \
  --save-total-limit 20 \
  --global-batch-size 32 \
  --dataloader-num-workers 4

bash score_checkpoints.sh

Where this platform does not help

AY-Robots does not solve checkpoint selection, and no current product does. Ranking checkpoints by task success requires running the robot.

• No automatic held-out metric across all five policies. GR00T's open-loop MSE comes from the upstream script; lerobot's eval_loss arrived in 0.6.0. Neither is a success rate.
• No success classifier. Deciding whether a rollout succeeded is manual. Systems like AutoEval automate it with learned classifiers and reset policies, but on two fixed WidowX stations with four fixed tasks.
• No simulator for your scene. SIMPLER makes simulated evaluation predictive of real performance, but only after someone builds a simulated environment matched to the real one. The paper's sim-to-real correlation rests on around 1500 evaluation episodes across two standard embodiments, Google Robot and WidowX. That is a project, not a checkbox.
• Remote inference changes what you measure. The control loop is 20 to 485 ms per action step depending on the model, and public-internet round trips turn a working policy into a hesitant one. Keep the inference location constant across candidates, or you are ranking network paths.

Failure modes that look like a bad checkpoint but are not

Before concluding that every checkpoint in a run is bad, rule out the causes that make all of them look bad at once. A systematic problem in the data or the serving path produces a flat, uniformly poor evaluation.

• Loss near zero and the arm barely moves: loss falls, policy does nothing. Usually a normalisation or action-space mismatch, identical across every checkpoint.
• Works on your desk and nowhere else: policy only works in one setup. Your held-out episodes came from that same setup, so open-loop error will not catch it.
• The arm stops halfway: policy freezes mid-motion, often an action chunking horizon issue rather than checkpoint quality.
• GR00T refuses a v3.0 dataset: dataset rejected as v3. The full index is at the fix pages, and run your first policy covers the serving side.

One free sanity check: run the fine-tuned checkpoint and a base checkpoint through the same open-loop script on the same episodes. If fine-tuning did anything the gap is obvious; if not, the problem is upstream of checkpoint selection. For how these models compare on published benchmarks, the arena lists 85 VLA models with 332 benchmark results, each value linked to its source.

The short version

The loss curve is a liveness check. Open-loop error on episodes the model never saw filters out broken checkpoints. Rollouts on the real arm are the only measurement, and how many you can afford sets how confidently you may speak. Earlier in the pipeline, the SO-100 setup guide covers the hardware and the VLA overview covers the models. The dataset directory has datasets to practise on; the SO-100 page has the arm.