Benchmarking Your Own Robot Task: Trials, Randomisation, Reporting

A benchmark is not a number you produce at the end of a training run. It is a procedure you write down before you run anything, that another person could follow and get roughly the same answer. Most people skip the writing-down part, run twelve rollouts, get eight successes, and tell everyone the policy is at 67 percent. That number is close to meaningless, and this article is about why, and what to do instead.

The good news is that an honest benchmark for one task on one SO-100 is mostly bookkeeping: a fixed list of starting conditions, a written success rule, a frozen configuration, and the habit of reporting raw counts instead of a percentage. The expensive part is trial count, so that is where this starts.

Why a leaderboard number cannot answer your question

The AY-Robots arena lists 85 vision-language-action models with 332 benchmark results, and every value links back to the paper or model card it came from. That is useful for orientation. It cannot tell you what a fine-tuned checkpoint will do on your table, with your lighting, your gripper and your task. Different embodiment, different objects, different success rule, different operator.

The SIMPLER paper makes the same point from the other direction. Ranking six open-source policy checkpoints on Google Robot tasks, validation mean squared error reached an average Pearson correlation of only 0.308 with real-world performance, while their visual-matching simulated evaluation reached 0.924. The loss curve you watch during training is a poor predictor of the ranking you care about, which is why you have to run the robot.

It also helps to see how small real published protocols are. People assume the numbers in VLA papers rest on thousands of rollouts. In the real-robot sections they usually do not.

Two things stand out. First, the per-condition sample sizes are small, often 10 to 25. Second, almost none of these report a confidence interval on a single policy's success rate. PhAIL, a 2026 preprint that benchmarks four public VLAs on a Franka FR3, opens by describing the field in exactly those terms: real-world VLA evaluation still rests on binary success rate at a fixed timeout with 25 or fewer rollouts per condition, almost always without confidence intervals or paired statistical comparison. That is the paper's own framing of the status quo rather than a counted survey, but it matches the table above. Copy the field's habits and you inherit the field's uncertainty.

How many trials you actually need

Success on a rollout is a coin flip with an unknown bias. That makes the statistics simple and unforgiving. The table below is a 95 percent Wilson score interval for an observed 70 percent success rate at different trial counts. Wilson rather than the textbook Wald interval because Brown, Cai and DasGupta showed in Statistical Science in 2001 that the Wald interval's coverage is erratic and recommended Wilson or Jeffreys for small samples; the NIST/SEMATECH e-Handbook gives the same formula in section 7.2.4.1.

# pip install statsmodels
from statsmodels.stats.proportion import proportion_confint

for n in (10, 20, 25, 30, 50, 100, 200, 500):
    k = round(0.70 * n)
    lo, hi = proportion_confint(k, n, alpha=0.05, method="wilson")
    print(f"{k}/{n}  [{lo*100:5.1f}, {hi*100:5.1f}]  width {100*(hi-lo):4.1f} pp")

Comparing two policies costs much more than measuring one

Measuring one policy roughly is cheap. Deciding which of two checkpoints is better is not, because you are now stacking two noisy estimates. The numbers below are for a standard two-proportion test, two-sided, alpha 0.05, 80 percent power, independent trials for each policy.

Read the last two rows again. A ten-point improvement, the size most fine-tuning changes actually produce, needs hundreds of rollouts per policy with independent trials. If your budget is 20 rollouts per checkpoint, you cannot resolve ten points, and your notes should say so.

The mirror-image question is more useful in practice: how often will a short evaluation rank the worse policy higher? Suppose the true rates are 60 and 70 percent, and you run n trials of each.

Define success before you run anything

Half the disagreements about a benchmark are definitional, not statistical. Write the success rule down first, in a form an unfamiliar person could apply from the video alone, and never edit it after seeing results.

• Terminal state: the cube is inside the bin footprint and the gripper has released it. Not 'roughly in the bin'.
• Tolerance: fully inside the printed rectangle counts, touching the rim does not.
• Time limit: 60 seconds of wall clock, or a fixed number of control steps. Say which, because the two diverge when inference latency changes.
• Retries: does a second grasp attempt after a slip still count as success? Decide before, not after.
• Abort conditions: what stops the rollout early and how that outcome is scored. Hitting a joint limit is a failure, not a discarded trial.
• Who scores it: one named person, or a video reviewed later by someone blind to which policy ran.

Binary success throws away most of what you saw

A binary outcome on 25 trials is a low-bandwidth measurement, and every serious recent protocol adds partial credit. RoboChallenge splits each task into stages worth 10 progress points per rollout, runs 10 rollouts per task so a task totals 100 points, deducts 0.5 progress points for each retry, and terminates the rollout once the number of successive failed retries exceeds four. pi0.5 scores against a rubric that roughly corresponds to the percentage of steps completed, so placing half the dishes in the sink scores around 50. RoboArena asks evaluators for a continuous progress score from 0 to 100 alongside the binary preference.

The failure category is the field you will thank yourself for. Tags like 'missed grasp', 'grasped then dropped', 'wrong object', 'froze' or 'collided' turn 25 rollouts into a debugging plan, and the failure-mode pages split along roughly those lines: a policy that freezes mid-motion and a gripper that does not close are different problems even when both show up as a zero in the success column.

Randomisation you can reproduce

There is a real tension here. If every trial starts from an identical cube position you are measuring one point in the task distribution, and the number will not survive a slightly different Tuesday. If every trial starts wherever you happened to drop the cube, the evaluation is not repeatable and, worse, it is manipulable. The RoboChallenge team measured this: they compared experienced testers who had collected the training data, ignorant testers who only read the task description, and adaptive testers who were the model authors and placed objects strategically based on how previous runs had gone. Their summary of real-machine testing is blunt: with the same props, task and model, the measured success rate can change even from 0 to 100 percent.

The fix is a written condition grid. Enumerate the factors you want to vary, take the product, and sample from that list with a fixed seed. SIMPLER does this explicitly: pick coke can is 3 orientations times 25 marked grid positions for 75 trials; move object near object is 5 object triplets times 2 triangle patterns times 6 role assignments for 60 trials; the drawer tasks are 9 robot base positions times 3 drawers times 2 directions for 54 trials. Nothing is left to the operator's judgement in the moment.

1. 1
   Write the condition grid

   Pick two or three factors that matter for your task and enumerate their levels. Tape a grid onto the work surface and label the cells, so a position is a name and not a guess.

   python

   import itertools, random
   
   positions    = ["A1", "A2", "B1", "B2", "C1", "C2"]   # taped cells on the mat
   orientations = ["flat", "upright"]
   distractor   = [False, True]
   
   conditions = list(itertools.product(positions, orientations, distractor))  # 24 cells
   trials = conditions * 2                                                   # 48 trials
   random.Random(1000).shuffle(trials)                                       # fixed seed
   
   print("trial,position,orientation,distractor")
   for i, (pos, ori, dis) in enumerate(trials, start=1):
       print(f"{i:03d},{pos},{ori},{int(dis)}")
   

2. 2
   Print the sheet and follow it

   Print the 48 lines and work down the list. The order is randomised so that anything drifting across the session, servo temperature, daylight through a window, your own fatigue, is spread across conditions instead of piling into the last ten trials.

   bash

   python make_trials.py > trials_2026-08-24.csv
   head -4 trials_2026-08-24.csv
   # trial,position,orientation,distractor
   # 001,B2,flat,1
   # 002,A1,upright,0
   # 003,C1,flat,0
   

3. 3
   Reproduce the scene, do not eyeball it

   RoboChallenge superimposes a reference image on the live camera stream and asks the tester to move objects until the images match. You can do the same with a still frame and any camera viewer: it is the cheapest reproducibility tool in this whole article.

   bash

   # capture the reference frame for one condition, once
   ffmpeg -f avfoundation -framerate 30 -i "0" -frames:v 1 ref_B2_flat.png
   
   # later, overlay the reference at 50 percent on the live view while you reset the scene
   ffplay -f avfoundation -framerate 30 -i "0" \
     -vf "movie=ref_B2_flat.png[r];[in][r]blend=all_mode=average"
   

4. 4
   Log the trial before you run it

   Write down the condition and the checkpoint identifier before the rollout starts, not after you have seen the outcome. This is the difference between a record and a memory.

   json

   {"trial": 17, "policy": "act_so100_ckpt100000", "grid_seed": 1000,
    "position": "B2", "orientation": "flat", "distractor": true,
    "operator": "pk", "started_at": "2026-08-24T14:05:11Z",
    "outcome": "fail", "stage_reached": 2, "stages_total": 4, "retries": 1,
    "time_to_success_s": null, "timeout_s": 60,
    "failure_tag": "grasped_then_dropped", "notes": "caught the rim, slipped"}
   

What to hold fixed

Everything not in the condition grid must be pinned, written down and re-checked between policies. None of the items below show up in the success column when they go wrong; they just quietly make the comparison meaningless. Two of them, the action chunking settings and the inference latency you actually run at, change the controller itself rather than the weights.

One thing you cannot hold fixed is worth naming. The GR00T N1.7 fine-tuning entry point, the tyro CLI in Isaac-GR00T's launch_finetune.py, exposes no seed argument, so GR00T runs are not bit-for-bit reproducible and trainer variation becomes part of the noise in your comparison. LeRobot does expose a seed and defaults it to 1000, so ACT, SmolVLA and Pi0.5 runs can at least be repeated. If you are comparing two GR00T checkpoints trained on the same data, some of the difference you measure is the trainer, not your change.

Pair the trials and blind the operator

The cheapest way to buy statistical power is to stop running two independent batches. RoboArena is explicit about it: the evaluator arranges a scene, is handed two anonymous policy endpoints, runs A and B back to back, and is required to closely match the initial conditions within that pairwise comparison, though conditions may change freely between comparisons. pi0.5 does the same by holding the set of items constant for both policies within a comparison. Pairing removes scene-to-scene variance, and a paired test then only looks at the trials where the two policies disagreed.

The paired numbers come from the McNemar sample-size formula and depend on how often the two policies differ on the same scene. Two similar checkpoints agree often, so pairing wins big. Either way you avoid the failure mode where policy A was evaluated in the morning and policy B after the sun moved.

Two ways to run this

# the evaluation extra alone is not enough: you also need the policy extra
# (e.g. lerobot[pi]) and, for a simulated env, the env extra (e.g. lerobot[pusht])
pip install 'lerobot[evaluation,pusht]'

lerobot-eval \
  --policy.path=outputs/train/my_run/checkpoints/100000/pretrained_model \
  --env.type=pusht \
  --eval.n_episodes=200 \
  --eval.batch_size=50 \
  --seed=1000 \
  --policy.device=cuda

# the simulated reference protocol, for comparison
git clone https://github.com/openvla/openvla.git
python experiments/robot/libero/run_libero_eval.py \
  --model_family openvla \
  --pretrained_checkpoint openvla/openvla-7b-finetuned-libero-spatial \
  --task_suite_name libero_spatial \
  --center_crop True
# defaults: num_trials_per_task=50, seed=7, num_steps_wait=10
# 10 tasks x 50 rollouts = 500 trials per suite, averaged over 3 seeds

Reporting a result someone else can check

A benchmark result is a claim, and a claim needs enough attached information for a reader to decide whether to believe it. If you cannot fill in a row, say so rather than letting the reader assume.

Two conventions from outside robotics are worth stealing. Agarwal and colleagues, writing about deep reinforcement learning in 2021, argued that with few runs the interquartile mean with stratified bootstrap confidence intervals beats a bare mean, and that performance profiles beat single numbers. And OpenVLA reports 84.7 +/- 0.9 percent on LIBERO-Spatial, where the interval is a standard error across three training seeds of 500 rollouts each rather than a spread over rollouts. Those are two different uncertainties: rollout noise says how well you measured this checkpoint, seed noise says how much of a difference is the trainer.

A worked protocol for one SO-100 pick and place

Putting it together for a concrete case: you have fine-tuned ACT on a LeRobot dataset of roughly 60 episodes recorded through teleoperation, and you want to know whether a second checkpoint trained on 120 episodes is actually better.

1. 1
   Decide what gap is worth detecting

   Be honest before you start. If you only care about improvements of 20 points or more, 45 to 60 paired trials will do. If you want to resolve 10 points, the paired budget is over 200 trials and you should probably spend that time collecting more data instead.

   text

   target gap        20 points (60% -> 80%)
   design            paired A/B, matched scene
   expected disagreement  ~30% of trials
   budget            57 paired trials = 114 rollouts
   at 60 s + 30 s reset   about 3 hours of bench time
   

2. 2
   Pin the configuration

   Write both checkpoints, both sets of normalisation statistics, and the chunk settings into one file. ACT on this platform defaults to chunkSize 100 and nActionSteps 100; if the two checkpoints do not share those values you are comparing two controllers, not two datasets.

   yaml

   policy_a:
     name: act_so100_60ep
     checkpoint: run-4a1f/step-100000
     chunk_size: 100
     n_action_steps: 100
     seed: 1000
   policy_b:
     name: act_so100_120ep
     checkpoint: run-9c02/step-100000
     chunk_size: 100
     n_action_steps: 100
     seed: 1000
   inference: local        # 20 ms per action step for ACT
   instruction: "pick up the red cube and place it in the bin"
   

3. 3
   Run the sheet

   For each line: reset the scene against the reference image, start the wrapper that randomly picks A or B, run, score, then run the other policy from the same reset. Two rollouts per line, one line per scene.

   bash

   while read -r line; do
     echo "scene: $line"
     read -rp "scene matched to reference? [enter]" _
     ./run_pair.sh --config bench.yaml --trial "$line" >> results.jsonl
   done < trials_2026-08-24.csv
   

4. 4
   Count the disagreements, not the totals

   The paired comparison lives entirely in the trials where one policy succeeded and the other failed. Everything else cancels.

   python

   import json
   from statsmodels.stats.contingency_tables import mcnemar
   
   rows = [json.loads(l) for l in open("results.jsonl")]
   b = sum(1 for r in rows if r["a"] and not r["b"])   # A wins the scene
   c = sum(1 for r in rows if r["b"] and not r["a"])   # B wins the scene
   n11 = sum(1 for r in rows if r["a"] and r["b"])
   n00 = sum(1 for r in rows if not r["a"] and not r["b"])
   
   print(f"both {n11}  neither {n00}  A only {b}  B only {c}")
   print(mcnemar([[n11, b], [c, n00]], exact=True))
   

5. 5
   Write the result down in full

   One paragraph with counts, interval, configuration and seed. If the paired test does not reach significance, say that, and say what gap your sample size could have detected. A null result you can trust is more useful than a positive result you cannot.

   text

   ACT/SO-100 pick-and-place, 2026-08-24, unit 2, local inference.
   57 paired scenes, grid seed 1000, blind endpoint selection, one operator.
   A (60 episodes): 34/57 = 59.6% (95% Wilson 46.7 to 71.3)
   B (120 episodes): 44/57 = 77.2% (95% Wilson 64.8 to 86.2)
   Discordant: A only 4, B only 14. McNemar exact p = 0.031.
   Powered to detect a 20-point gap; a 10-point gap would not have shown.
   

One caution about which weights you pick. Running the benchmark on eight checkpoints at 25 trials each and reporting the best is the same mistake as tuning on the test set: you have selected for luck. Screen on a small pass, benchmark the survivor properly, and say in the report that you did it that way.

Where this stops being worth it

Honest benchmarking has a point of diminishing returns, and knowing where it is saves you weeks. If your policy is at 20 percent, you do not need statistics, you need better data or a different setup; the data collection guide is a better use of an afternoon than 200 rollouts. If your policy works in exactly one lighting condition, that is a known failure pattern with a known set of causes, described on the page about policies that only work in one setup, and a bigger trial count will only measure it more precisely.

Rigorous evaluation pays off at the margin: when two options look similar and you have to choose, when you are about to spend real money scaling a dataset, or when someone outside your team will act on the number. Work on making it cheaper is active. SureSim combines large-scale simulation with a small number of real trials using prediction-powered inference and reports saving over 20 to 25 percent of hardware evaluation effort for similar bounds; RoboArena found its crowd-sourced ranking converged within roughly 100 pairwise comparisons. Neither removes the need to run the robot.