Automatic Success Detection for Robot Policy Rollouts

A rollout either worked or it did not. Turning that into a number sounds trivial until you have to do it two hundred times without a person in the room. Then it becomes a concrete engineering question: which piece of software decides, what exactly does it check, and how often is it wrong in a direction that flatters your policy?

This page is about that piece of software: the four places a success bit can come from, only three of which produce a number you can publish; what vision-based detectors actually score and at what precision; why a judge that is right 90 percent of the time bends a leaderboard in a predictable direction; and why two published success rates almost never measure the same thing. Every threshold below comes out of source code or a paper linked at the bottom.

Where the success bit comes from

There are four mechanisms in common use, and only the first three produce something you can put on a leaderboard. Knowing which one produced a number is the first thing to ask about any published success rate, and it is almost never stated in the abstract.

The fourth row is worth separating out early, because it gets conflated with the others constantly. A failure detector that fires when a diffusion policy starts oscillating is useful, but it is not a success detector and its output does not belong in a success column.

Simulator predicates: read the code, not the paper

In simulation the success bit is a pure function of scene state, written by hand by whoever built the task. SIMPLER, the real-to-sim evaluation suite from the VLA community, is a good place to look because the code is short. Here is the grasp predicate for the coke-can family of tasks, from grasp_single_in_scene.py in the ManiSkill2_real2sim repository.

is_grasped = self.agent.check_grasp(self.obj, max_angle=80)
if is_grasped:
    self.consecutive_grasp += 1
else:
    self.consecutive_grasp = 0
    self.lifted_obj = False

# ... contact loop elided here: it sets flag = False while the object is still
# resting on a non-robot actor with a contact impulse above 1e-6 ...

consecutive_grasp = self.consecutive_grasp >= 5
diff_obj_height = self.obj.pose.p[2] - self.obj_height_after_settle
self.lifted_obj = self.lifted_obj or (flag and (diff_obj_height > 0.01))
lifted_object_significantly = self.lifted_obj and (diff_obj_height > 0.02)

if self.require_lifting_obj_for_success:
    success = self.lifted_obj
else:
    success = consecutive_grasp

if self.success_from_episode_stats:
    # During evaluation, if policy puts down coke can in the end but has lifted it
    # significantly before, it is still a success
    # However, if you want to perform RL training on this environment,
    # make sure to turn off this option
    success = success or (self.episode_stats["n_lift_significant"] >= 5)

Read what that says. A grasp counts if the gripper contact normal is within 80 degrees. It has to hold for 5 consecutive simulation frames. The object has to rise 1 cm above where it settled to arm the lift flag, and 2 cm for the lift to count as significant. And on the evaluation path, a rollout that lifts the can and then sets it back on the table is still a success.

The move-near task in the same repository is a conjunction of four conditions, every one carrying a constant that somebody chose.

• No other object dropped more than 2 cm below where it settled, and neither the source nor the target object fell more than 15 cm.
• The correct source object moved more than 0.03 m in the xy plane, and moved further than every other object in the scene.
• The source object ends up within the sum of the two objects' half bounding-box diagonals plus 0.10 m of the target.
• The source object is the closest object to the target, with a 0.01 m tolerance.

Change the 0.10 m to 0.05 m and every published number on that task moves. Nothing in the paper title tells you which constant was in effect. The same goes for the episode budget, which is part of the success criterion whether or not anyone calls it that. The widely copied OpenVLA evaluation script for LIBERO hard-codes a per-suite horizon derived from the longest training demonstration.

num_steps_wait: int = 10          # Number of steps to wait for objects to stabilize in sim
num_trials_per_task: int = 50     # Number of rollouts per task

if cfg.task_suite_name == "libero_spatial":
    max_steps = 220   # longest training demo has 193 steps
elif cfg.task_suite_name == "libero_object":
    max_steps = 280   # longest training demo has 254 steps
elif cfg.task_suite_name == "libero_goal":
    max_steps = 300   # longest training demo has 270 steps
elif cfg.task_suite_name == "libero_10":
    max_steps = 520   # longest training demo has 505 steps
elif cfg.task_suite_name == "libero_90":
    max_steps = 400   # longest training demo has 373 steps

That matters more than it looks. A policy with a 485 ms action step and one with a 20 ms action step are given the same number of steps, not the same number of seconds, so the horizon penalises hesitancy rather than latency. Check the per-step figures on the policy comparison page before reading anything into a horizon-limited benchmark.

Vision-based success detection on real hardware

On a real SO-100 there is no scene state to query. LeRobot's guide for adding a benchmark states that your environment must return info["is_success"] on every step() call, because that is how the evaluation loop knows the task was completed. The reduction matters too: lerobot_eval.py collapses the per-step flags with einops.reduce(..., "b n -> b", "any"), so an episode passes if the flag was ever true, not if it is true at the last step. The mean over episodes, times 100, is pc_success. Nothing on a physical arm produces that key.

# lerobot-eval works because the simulator supplies info["is_success"].
# EvalConfig defaults: n_episodes=50, batch_size=0 (auto-tuned from CPU cores).
# The seed lives one level up, on EvalPipelineConfig, and defaults to 1000.
lerobot-eval \
    --policy.path=lerobot/diffusion_pusht \
    --env.type=pusht \
    --eval.batch_size=10 \
    --eval.n_episodes=10 \
    --policy.use_amp=false \
    --policy.device=cuda

So you build the instrument. AutoEval, from Berkeley, is the most complete published recipe: a per-scene classifier fine-tuned from PaliGemma-3B as a yes/no visual question, plus a learned reset policy so the station runs unattended.

1. 1
   Collect labelled images by teleoperating the scene

   Drive the arm through the success state and the failure state, dumping frames into one pickle per label. AutoEval used roughly 1000 images per scene, under ten minutes of teleoperation.

   bash

   python scripts/teleop.py --ip <ROBOT_IP> --log_type pkl \
       --log_dir ~/datasets/record-open_drawer.pkl
   python scripts/teleop.py --ip <ROBOT_IP> --log_type pkl \
       --log_dir ~/datasets/record-close_drawer.pkl

2. 2
   Fine-tune PaliGemma as a yes/no VQA head

   The question is closed-set and scene-specific: 'is the drawer open?', 'is the eggplant in the sink or in the basket?'. It is not a general 'did it succeed?' judge.

   bash

   huggingface-cli login   # paligemma is a gated repo
   python scripts/ft_paligemma.py --working_dir ~/datasets/ --dataset_type drawer

3. 3
   Score it on a held-out set before you trust it

   AutoEval evaluated on roughly 100 held-out images and only deployed a classifier above 95 percent accuracy. Below that, collect more data.

   bash

   python scripts/ft_paligemma.py --working_dir ~/datasets \
       --model_id ~/datasets/checkpoints/... --eval

4. 4
   DAgger the classifier on its own mistakes

   Run an evaluation with data saving on, relabel the frames the classifier got wrong, retrain. This is where most of the accuracy comes from.

   bash

   python run_eval.py --save_classifier_data
   python scripts/filter_images.py --input_folder ~/auto_eval_log/... \
       --output_folder ~/datasets/

5. 5
   Run the unattended evaluation

   With a working classifier and a reset policy the station scores rollouts on its own. AutoEval cut human evaluator time by more than 99 percent.

   bash

   python run_eval.py --robot_ip <ROBOT_IP> \
       --config scripts/configs/eval_config.py:open_drawer \
       --policy_server_ip <POLICY_SERVER_IP> \
       --policy_server_port <POLICY_SERVER_PORT>

The honest part of that paper is the error accounting. The authors walked through all 50 episodes of one AutoEval run on the "put eggplant in blue sink" task with the Open-pi0 policy. Three of the 50 produced a wrong result, in each case because the episode was misclassified as a success and the reset policy had failed. Motor failures were more frequent but get re-run automatically rather than scored. Aggregated over five tasks, six generalist policies and 50 rollouts per policy per task, agreement with human-run evaluation was a Pearson r of 0.942 and an MMRV of 0.015.

The false positive tax

A judge that is wrong 10 percent of the time does not add 10 percent of noise. It adds bias, in a predictable direction. If the true success rate is p and the judge has sensitivity s and specificity t, the rate you observe is s*p + (1-t)*(1-p). Two lines of arithmetic, and it changes how you read every automated leaderboard.

def observed(p_true, sensitivity, specificity):
    """What an imperfect judge reports when the real success rate is p_true."""
    return sensitivity * p_true + (1.0 - specificity) * (1.0 - p_true)

# The gap between two policies is scaled by Youden's J = sensitivity + specificity - 1
for s in (0.99, 0.95, 0.90, 0.80, 0.71):
    j = 2 * s - 1
    print(f"sens=spec={s:.2f}: a true 10 point gap is measured as {j * 10:.1f} points")

Read the columns rather than the rows. A symmetric judge pulls every number toward 50 percent: weak policies look better than they are, strong ones look worse, and the gap shrinks by exactly Youden's J. At 0.90 sensitivity and specificity a real 10 point improvement shows up as 8 points. At the 0.71 GVL-SD reports zero-shot it shows up as 4.2, smaller than the sampling noise of a 50-rollout evaluation. You would be measuring your judge, not your policy.

This is why the false positive direction deserves more attention than the false negative one. A false negative costs you a rollout. A false positive tells you a broken checkpoint is fine, and you ship it. If you can only tune one threshold, tune for precision.

Runtime monitors answer a different question

Two lines of work get filed under success detection and should not be. They are worth running, just not worth averaging.

• Sentinel (CoRL 2024) splits failure into erratic behaviour, caught with a statistical measure of temporal action consistency at negligible compute cost, and task-progression failure, caught with a VLM. Combining the two detects 18 percent more failures than either alone, and it is built to warn mid-rollout, not to score afterwards.
• SAFE (NeurIPS 2025) reads the last-layer features of a vision-language-action model and emits one scalar failure likelihood, tested on OpenVLA, pi0 and pi0-FAST, with functional conformal prediction calibrating the threshold. Its headline metric is ROC-AUC over rollouts, a detector metric rather than a policy metric.
• Both are the right tool for stopping a rollout before the arm knocks something over. Neither produces a number comparable to a published success rate, because neither measures task completion.

How many rollouts you actually need

Even a perfect judge leaves a binomial sampling problem, and robot evaluations run in the regime where that problem dominates. The Clopper-Pearson interval is the conservative default; here it is on an observed 70 percent success rate at several trial counts.

from scipy.stats import beta

def clopper_pearson(k, n, alpha=0.05):
    lo = 0.0 if k == 0 else beta.ppf(alpha / 2, k, n - k + 1)
    hi = 1.0 if k == n else beta.ppf(1 - alpha / 2, k + 1, n - k)
    return lo, hi

for n in (10, 20, 30, 50, 100, 200, 500):
    k = round(0.7 * n)
    lo, hi = clopper_pearson(k, n)
    print(f"n={n:4d}  observed 70.0%  95% CI [{lo*100:5.1f}, {hi*100:5.1f}]")

The STEP paper (Sequential Testing for Efficient Policy Comparison), from Toyota Research Institute and Princeton, is the most useful practical treatment. It starts from the constraint everyone works under, a feasible sample size of about 10 or 50 trials. On its bimanual FoldRedTowel task the two policies sat at 56 and 92 percent, a 36 point gap, and STEP stopped after 19 to 23 of the 50 available trials depending on the budget it was configured for. On the CarrotOnPlate comparisons, where the gaps were 59 against 68 percent and 68 against 76, the paper estimates 500 trials per policy would be needed to call a winner at the standard significance level. Against state-of-the-art sequential baselines it cuts the required trials by up to 32 percent.

Why two published success rates rarely compare

Put the pieces together and the reason is mechanical rather than mysterious. Seven independent knobs sit between a rollout and a percentage, and papers usually report none of them in the table caption.

The most interesting response is to stop reporting absolute rates at all. RoboArena crowd-sources evaluation across a network of labs on DROID hardware, lets each evaluator pick their own task and scene, and requires only that comparisons be double-blind and pairwise. From more than 600 pairwise real-robot episodes across seven generalist policies at seven institutions, the aggregated preference ranking tracked real performance better than centralised evaluation did.

SIMPLER attacks the same problem from the simulation side, and its real-world reference numbers were not cheap: on the Google Robot setup, 75 pick-coke-can, 60 move-near, 54 drawer and 27 open-drawer-then-place-apple trials per policy across six open-source checkpoints, plus 24 real trials per Bridge task. Against those, visual matching reports a Pearson r of 0.976 with MMRV 0.031 on pick-coke-can and 0.942 with MMRV 0.027 on the drawer tasks. Good correlations for ranking, and still not a licence to quote a simulation percentage as a real-world one. If simulation is where you want to spend effort, the GPU simulation stack is a separate discussion.

Two ways to get a scored rollout

# lerobot[evaluation] is the eval framework only; the environment and policy
# extras are separate, so a pusht eval needs the pusht extra as well.
pip install -e '.[evaluation,pusht]'

# Simulation: the env supplies info["is_success"], lerobot-eval reports pc_success
lerobot-eval --policy.path=lerobot/diffusion_pusht --env.type=pusht \
    --eval.n_episodes=50 --policy.device=cuda

# Real SO-100: no success bit exists. Record the rollouts, score them later.
lerobot-rollout \
    --strategy.type=episodic \
    --policy.path=user/my_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --teleop.type=so100_leader \
    --teleop.port=/dev/ttyACM1 \
    --dataset.repo_id=user/rollout_episodic_data \
    --dataset.num_episodes=20 \
    --dataset.single_task="Grab the cube"

A scoring protocol you can run on an SO-100 this week

None of this needs a research budget. It needs writing things down before you run them, which is the part that gets skipped.

1. 1
   Write the predicate in one sentence, before the first rollout

   Not 'picks up the cube'. Something a stranger could apply from the video alone: 'the cube is off the table and inside the gripper when the episode ends'. If you cannot write it in one sentence, you have a demo, not a task.

   text

   SUCCESS: at the final frame, the cube is fully inside the jaws and its
            lowest point is above the table surface.
   FAILURE: anything else, including a lift followed by a drop.
   VOID:    the arm never moved because the port was wrong (re-run, do not score).

2. 2
   Fix the initial states and their order

   Twenty numbered start positions marked on tape, in a seeded order identical for every checkpoint you compare. Most apparent policy differences are differences in which start poses got drawn.

   python

   import random
   rng = random.Random(1000)          # lerobot's default seed, for what it is worth
   order = list(range(1, 21))
   rng.shuffle(order)
   print(order)                       # use this exact order for every checkpoint

3. 3
   Record every rollout, do not score live

   Live scoring is where rater drift enters. Record first, score later, from video, with the checkpoint name hidden.

   bash

   lerobot-rollout --strategy.type=episodic \
       --policy.path=user/my_policy \
       --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
       --dataset.repo_id=user/rollout_eval_run7 \
       --dataset.num_episodes=20 --dataset.single_task="Grab the cube"

4. 4
   Label twice and measure your own agreement

   Have two people label the same clips independently and compute agreement before you trust any automated judge. If two humans disagree on 8 of 50 clips, no classifier trained on one person's labels will beat that.

   python

   from sklearn.metrics import cohen_kappa_score
   print(cohen_kappa_score(labels_alice, labels_bob))
   # below about 0.8 means the predicate is ambiguous, not that the raters are careless

5. 5
   Only then automate, and report both numbers

   Calibrate the classifier against the human labels on the same clips, report its sensitivity and specificity beside the success rate, and keep the human labels for the runs that matter.

   text

   ACT ckpt-40000: 68% (34/50) auto-scored
                   judge: sens 0.93 / spec 0.90 on 50 human-labelled clips
                   95% CI on the raw count: 53.3 to 80.5 percent

Where this platform does not help

AY-Robots has no automatic success detection. There is no scoring endpoint, no built-in classifier, no success column that fills itself in. Training, GPU rental, checkpoint storage and pod-based serving are handled; deciding what counts as done is not.

If you have not run a policy on hardware yet, the first-policy walkthrough is the shorter path, and recording a dataset comes first. Data quality sets the ceiling your success rate is measured against, covered in the data collection guide. And ACT at 20 ms and SmolVLA at 245 ms per action step behave very differently under a fixed step budget: see ACT on SO-100 and GR00T N1.7 on SO-100.