Policy Reliability: From a Working Demo to an Honest Success Rate

The first time a fine-tuned policy picks up the cube, the instinct is to reach for the phone and film it. That clip is real. It is also one sample from a distribution nobody has measured. Run the same policy twenty more times, move the cube two centimetres, turn on the overhead light, and it stops being representative of anything.

This article is about the distance between those two states: a policy that works once, and a policy whose behaviour you can predict tomorrow morning. That distance is almost never closed by a better model. It is closed by a reset you perform the same way every time, a tally sheet, a confidence interval, a retry rule, and a log you actually read.

The demo is one sample, and you chose it

Selection bias in robot demos is not dishonesty, it is convenience. The policy fails, you nudge the cube, it fails, you nudge again, it works, you film that one. The belief that follows, it works, is built from a single sample drawn after several unrecorded resets.

The failures that surface on attempt eleven are rarely new physics. They are the thin parts of your demonstration coverage, plus the environment drifting while you are not looking.

None of these are model problems, which is why swapping ACT for SmolVLA rarely fixes them. They are measurement problems, and the failure mode index is a faster diagnostic than another training run.

One number, and the interval you left off it

Success rate is a binomial proportion. If you ran n attempts and k succeeded, the honest report is k/n plus an interval. Clopper-Pearson is the conservative choice and it is two lines of Python. At 95 percent:

from scipy.stats import beta

def clopper_pearson(k, n, alpha=0.05):
    """Exact two-sided binomial interval for k successes in n trials."""
    lo = 0.0 if k == 0 else beta.ppf(alpha / 2, k, n - k + 1)
    hi = 1.0 if k == n else beta.ppf(1 - alpha / 2, k + 1, n - k)
    return lo, hi

print(clopper_pearson(24, 30))   # (0.6143, 0.9229)
print(clopper_pearson(8, 10))    # (0.4439, 0.9748)

The second table ends arguments. To claim policy A beats policy B, the attempts you need depend on how big the real gap is. Two-proportion sample sizes at alpha 0.05 and 80 percent power, per policy:

Read the last two rows carefully. A 10 point improvement in an already good policy is beyond what one arm and one human can measure in a week. That is not a reason to give up, it is a reason to stop claiming it. If two checkpoints land at 82 and 88 percent over 30 attempts each, you cannot tell them apart.

What published work actually runs

The numbers behind the tables you compare yourself against are smaller than most people assume. All of the following was read from the papers themselves.

The reset is the experiment

The highest-leverage change most people can make is to write down how the scene is reset and then follow that written procedure instead of their memory. TRI did this with an image overlay: the person resetting the cell matched the objects to a picture of the desired scene. ACT constrained object placement to a 15 cm reference line on the table. Robot Utility Models reused a fixed grid of start positions across every environment.

For one SO-100 on a desk, the cheap version is a printed sheet taped down with the target zone marked, plus a photo of the start scene on your phone. Ten minutes of work, and it removes the largest single source of variance. If the arm itself is not yet repeatable, fix that first with the SO-100 setup guide.

1. Mark the object start zone physically. Tape or a printed template, not "about here".
2. Return the arm to a defined start pose every time. --strategy.reset_to_initial_position does this in lerobot.
3. Photograph the reference scene once and match to the photo, not to memory.
4. Fix the lighting. Same lamps, same blinds. Note it if you cannot.
5. Do not touch the cameras. If you must, before and after the move are two experiments.
6. Set the time limit up front and let attempts fail on the clock, not on your patience.
7. Randomise order when comparing policies, and if a second person is free, let them run the arm blind.

Running the evaluation on real hardware

The tooling exists and most people skip it. In lerobot, lerobot-rollout is the single CLI for deploying a trained policy on a real robot, and its episodic strategy is built for scored attempts: the policy drives each episode, then an optional leader arm drives during a timed reset phase. Flags below were read from the lerobot deployment docs on 2026-08-24.

1. 1
   Check the checkpoint offline first

   Comparing predicted actions against ground truth from a held-out episode costs nothing and catches checkpoints that will not move.

   bash

   uv run python gr00t/eval/open_loop_eval.py \
       --dataset-path /data/so100_cube \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path /checkpoints/gr00t_n17_cube/checkpoint-20000 \
       --traj-ids 0 \
       --execution-horizon 16

2. 2
   Do one throwaway run to shake out the cell

   Base strategy, no recording. This is where you find a moved camera index or an upside-down wrist view. Nothing here counts.

   bash

   lerobot-rollout \
       --strategy.type=base \
       --policy.path=${HF_USER}/act_so100_cube \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
       --task="Pick up the red cube and put it in the box" \
       --duration=60

3. 3
   Run the scored block with episodic

   Numbered episodes, a fixed limit per attempt, a reset phase between them. Pin the revision rather than trusting main; each pushed checkpoint is tagged with its step.

   bash

   lerobot-rollout \
       --strategy.type=episodic \
       --policy.path=${HF_USER}/act_so100_cube \
       --policy.pretrained_revision=010000 \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1 \
       --strategy.reset_to_initial_position=true \
       --dataset.repo_id=${HF_USER}/eval_cube_ckpt10000 \
       --dataset.num_episodes=30 \
       --dataset.episode_time_s=30 \
       --dataset.reset_time_s=15 \
       --dataset.single_task="Pick up the red cube and put it in the box"

4. 4
   Score each attempt while the reset runs

   Live scoring, not retrospective scoring from video. Video is slower and reliably kinder to the policy.

   text

   ep  outcome   note
   01  S         clean
   02  F-grasp   closed 1 cm short
   03  S
   04  X         leader arm bumped during reset, rerun
   05  F-stall   froze above the cube, 12 s left

5. 5
   Compute the interval, write the honest sentence

   Excluded attempts do not count in n. Report interval, n and reset procedure together; any one alone is not reproducible.

   text

   24 / 30 = 80.0%  (95% CI 61.4% to 92.3%)
   reset: taped 10 cm zone, arm homed, overhead lamp only, 30 s limit

Binary success is a blunt instrument

Pass or fail throws away most of what a rollout tells you. Pi0.5 reports task progress rather than binary success for its home evaluations, and SmolVLA decomposes real-world tasks into subtasks. A policy that reaches the object and fails the grasp is a different engineering problem from one that never leaves its start pose, and averaging them into one zero hides which you have.

Two ways to get to a number you trust

pip install lerobot

# record (defaults: 60 s episode, 60 s reset, 50 episodes)
lerobot-record --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
  --teleop.type=so100_leader --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=${HF_USER}/so100_cube --dataset.num_episodes=50

# evaluate
lerobot-rollout --strategy.type=episodic --policy.path=${HF_USER}/act_so100_cube \
  --dataset.num_episodes=30 --dataset.episode_time_s=30 --dataset.reset_time_s=15

Latency quietly changes the number you measure

A success rate is comparable across setups only if the control loop was the same, and it usually is not. Per-action inference latency across the five trainable policies spans more than an order of magnitude, a direct consequence of how vision-language-action models are built. Those figures are the floor; anything you add comes out of the same budget.

There is real work on this. Real-Time Chunking generates the next action chunk while the current one is still executing, freezing the actions guaranteed to run and inpainting the rest, which keeps motion continuous despite inference delay. lerobot exposes it as an inference backend.

lerobot-rollout \
    --strategy.type=base \
    --inference.type=rtc \
    --inference.rtc.execution_horizon=10 \
    --inference.rtc.max_guidance_weight=10.0 \
    --policy.path=${HF_USER}/pi05_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --task="Pick up the cube" \
    --duration=60 \
    --device=cuda

Monitoring: read the loop, not just the outcome

Every lerobot control loop reports what it actually achieved, per episode and for the whole run. This is the closest thing to free monitoring you will get, and almost nobody reads it. It landed after the 0.6.1 release, so you need lerobot from main to see it. It gives you effective control frequency against target, how often the loop blew its budget, where the time went, and how much pacing headroom is left.

Cadence summary - whole run, 2 episodes - target 30 Hz
  effective cadence: 29.84 Hz policy over 40.2 s measured
  cycles over the 33.3 ms work budget: 12/1197 (1.0%)
  loop-body steps (share of measured work):
    observe      mean 3.13 ms  ...  31.7% of work
    infer        mean 5.18 ms  ...  52.4% of work
  pacing headroom: 7.4 ms slept per tick on average

The number to watch across a long session is pacing headroom. Near zero means the loop is saturated, and any hiccup, a camera stall, a slow disk write, a thermal throttle, costs you actions. A policy that scored 24 of 30 in the morning and 15 of 30 in the afternoon has usually not forgotten anything; something in the loop got slower. The client guide is a better first stop than a retraining run.

• Log the cadence summary from every scored block beside the tally.
• Treat any camera reconnect during a block as grounds to exclude that attempt. See camera not detected.
• Record evaluation runs as datasets. --strategy.type=sentry records continuously with auto-upload; episodic records per attempt.
• Keep the failure code distribution, not just the total. A drift from F-grasp toward F-approach usually means a camera moved.

What honest evaluation actually costs

The objection is time, and it is fair. Thirty attempts at 30 seconds plus 15 seconds of reset is about 22 minutes of arm time, but with scoring, exclusions and the servo that needs re-homing, budget an hour. Eighty attempts is an afternoon. That cost is real and worth naming.

The GPU side is comparatively cheap; an RTX 4090 tier run for SmolVLA or ACT is roughly 1 to 3 USD. The expensive resource is the human next to the arm, which is why so much published work stops at 10 rollouts, and why distributed efforts like RoboArena exist: seven academic institutions pooling more than 600 pairwise double-blind real-robot evaluation episodes.

The short version

The difference between a demo and a system is not a bigger model. It is a written reset procedure and a number with an interval on it. Once you have those, running a policy stops being a party trick, and the data collection choices that actually move the number become visible instead of guessed at.