Open-Loop vs Closed-Loop Evaluation of Robot Policies

A fine-tuning run just finished. There are eight checkpoints in the bucket and one arm on the desk. Two different tests are now available, and they answer two different questions. Open-loop evaluation replays a recorded episode and asks how close the model's predicted action is to the action a human actually sent. Closed-loop evaluation hands the model the robot and asks whether the task got done. Teams routinely run the first, report the number, and are then surprised that the arm never closes the gripper.

This page is about which test to run when, what each number really means, and the specific ways both of them mislead you. Every command below was read out of the upstream repository on 24 August 2026; flag names and defaults change, so the version is named wherever it matters.

The two questions, side by side

Both tests take the same artifact, a trained policy checkpoint, and both produce a single number. That is where the similarity ends. Open loop is a supervised-learning measurement: the model sees a real observation from your LeRobot dataset and its output is scored against the recorded label. Closed loop is a control measurement: the model sees whatever its own previous action produced, which after a few seconds is a state no human demonstrator ever visited.

The compounding-error asymmetry is the entire reason the two disagree. Imitation learning trains on a distribution of states a human produced. At rollout time the policy leaves that distribution the moment it makes a small mistake, and the errors it makes off-distribution are not the errors the training loss ever penalised. A model can be very good at the first question and useless at the second. See imitation learning for the background, and loss falls, policy does nothing for the version of this that shows up in a support ticket.

What open-loop evaluation actually measures

NVIDIA's GR00T N1.7 repository has a section literally called Open-Loop Evaluation, and one called Closed-Loop Evaluation right underneath it. The open-loop entry point is a tyro CLI at gr00t/eval/open_loop_eval.py. Point it at a dataset and a checkpoint and it walks a trajectory, asks the model for actions, and compares them against the recorded ones.

uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path <DATASET_PATH> \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path <CHECKPOINT_PATH> \
    --traj-ids 0 \
    --execution-horizon 16

Leave out --model-path and the script does not load a checkpoint at all: it falls back to a PolicyClient talking to a running policy server on 127.0.0.1:5555. That is the same server the closed-loop path uses, which is convenient, and it is also the first thing that silently changes the meaning of your run. Here are the real defaults from the ArgsConfig dataclass.

The MSE number and what it hides

The script computes mean squared error and mean absolute error over the unnormalized action vector, averaged across every action dimension and every time step, then averages those per-trajectory values. Four consequences follow, and all four bite in practice.

• Units are whatever your dataset stores. Unnormalized means degrees, or radians, or raw servo ticks. An MSE of 0.8 is meaningless without the unit. It is a within-project number, never a cross-project one.
• All joints are averaged together. On an SO-100 that is five arm joints plus a gripper, so the gripper is one sixth of the score. A policy that tracks the arm perfectly and never closes the gripper still posts a respectable MSE.
• One trajectory by default. --traj-ids defaults to [0]. One episode of one task is a sample size of one.
• It is scored against a human, not against success. Two different valid grasp approaches score as a large error against each other. A slower but correct trajectory scores worse than a fast wrong one.

What closed-loop evaluation actually measures

Closed loop means the environment steps. GR00T runs it as a server plus a client: the checkpoint is served over ZMQ, and a separate process owns the simulator or the robot and calls get_action in the control loop. The split exists because the simulator and the model usually want incompatible Python environments, and it is the same split the platform's inference pods use.

1. 1
   Serve the checkpoint

   One process holds the weights. --use-sim-policy-wrapper adapts the observation format for the simulator client.

   bash

   uv run python gr00t/eval/run_gr00t_server.py \
       --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
       --embodiment-tag LIBERO_PANDA \
       --use-sim-policy-wrapper

2. 2
   Run rollouts against it

   The client owns the environment. Note that it runs from the benchmark's own virtualenv, not the project root's.

   bash

   gr00t/eval/sim/LIBERO/libero_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
       --n-episodes 10 \
       --policy-client-host 127.0.0.1 \
       --policy-client-port 5555 \
       --max-episode-steps 720 \
       --env-name libero_sim/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it \
       --n-action-steps 8 \
       --n-envs 5

3. 3
   Sanity-check the harness before you trust it

   Start the server with --dataset-path and no --model-path and it replays recorded actions instead of predicting them. --execution-horizon is required on this path even though the README example omits it; without it the server raises before it binds. If ReplayPolicy does not solve the task, your environment is wrong and no checkpoint will save you.

   bash

   uv run python gr00t/eval/run_gr00t_server.py \
       --embodiment-tag NEW_EMBODIMENT \
       --dataset-path <DATASET_PATH> \
       --execution-horizon 16

On a real arm there is no is_success

In simulation the environment hands you info["is_success"]. On an SO-100 on a desk, nothing does. Somebody watches the rollout and writes a tick or a cross on paper. That is not a gap in the tooling, it is the actual state of the field, and it is why RoboArena went to double-blind pairwise comparisons across seven institutions rather than trying to standardise an absolute success number.

LeRobot's real-hardware path is lerobot-rollout. In the current docs it is the documented way to deploy a trained policy on the arm, and depending on the strategy it also records what happened. It gives you the rollout; it does not give you a verdict.

lerobot-rollout \
  --strategy.type=episodic \
  --policy.path=${HF_USER}/my_policy \
  --policy.pretrained_revision=010000 \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM1 \
  --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}}" \
  --task="Put lego brick into the transparent box" \
  --duration=600

Two details in that command carry more weight than they look. --policy.pretrained_revision lets you pin a specific pushed checkpoint by its step tag, so you can evaluate step 10000 and step 40000 without renaming directories. And all strategies accept --inference.type=rtc, real-time chunking, which exists because a 485 ms model cannot be run naively at 30 Hz. See action chunking and inference latency for what that is compensating for.

How many rollouts is enough

This is the question that decides whether a closed-loop number means anything, and it is almost never stated. A success rate over N rollouts is a binomial proportion. The confidence interval at small N is much wider than people's intuition, which is why two checkpoints that look 15 points apart are frequently indistinguishable.

The table below is the 95 percent Wilson score interval for an observed 70 percent success rate at different rollout counts, plus the smallest gap between two checkpoints that reaches significance in a two-proportion test around 60 percent. Because successes are whole numbers, the achievable gaps are coarse at small N: at 10 rollouts per arm the counts move in 10-point steps, and 4/10 against 8/10 still does not reach significance. These are standard statistics you can recompute in a few lines of Python, not measurements from any particular robot.

The vendors are not consistent about this either, which is worth knowing before you compare published numbers. NVIDIA's LIBERO results for GR00T N1.7 are reported over 200 rollouts per suite, ranging from 189/200 on the long-horizon suite to 197/200 on the object suite. The openpi LIBERO client defaults to num_trials_per_task = 50, which over ten tasks is 500 rollouts per suite. Same benchmark, two and a half times the sample size, and a confidence interval about 1.6 times wider at 200 rollouts than at 500.

Simulation benchmarks sit in between

There is a middle option that is genuinely closed loop but does not require your robot: run the rollouts in a simulator. It removes the reset labour and the human judgement, and it introduces a new question, namely whether the simulator ranks policies the way reality does.

SIMPLER is the interesting one methodologically, because it is a benchmark about benchmarks. Li et al. (2024) build simulated environments by green-screening simulated assets onto real backgrounds and matching object textures, then score the evaluation pipeline itself with two metrics: Mean Maximum Rank Violation and the Pearson correlation coefficient, both computed between simulated and real policy performance. The repository ships mean_maximum_rank_violation and pearson_correlation plus a table of real-world performance so you can score your own offline eval approach the same way. If you are building any kind of proxy evaluation, that is the honest way to justify it: show that it ranks policies the way the robot does.

Which test to run when

The decision is not open loop versus closed loop as a philosophy. It is: what am I trying to rule out right now, and what is the cheapest test that rules it out. Open loop is a smoke test with a numeric output. Closed loop is the acceptance test.

That latency column changes how closed-loop evaluation feels in practice. ACT at 20 ms per action step runs comfortably above 30 Hz, so a rollout looks like a normal robot motion. Pi0.5 at 485 ms per action step only works because the model emits a chunk and the controller plays it out. That is why real-time chunking exists, and why an evaluation that ignores the control rate is measuring the wrong thing. GR00T N1.7 sits at 152 ms, SmolVLA at 245 ms.

Running both, two ways

A loop that survives contact with a real week

Here is the sequence that wastes the least time, assuming one task, one arm, and a fine-tuning run that saved checkpoints along the way. It is built around the fact that open-loop evaluation is nearly free and closed-loop evaluation is not.

1. 1
   Replay one recorded episode on the arm first

   Before evaluating any model, replay a recorded episode. If the recorded actions do not reproduce the task, your calibration or your dataset is wrong and every evaluation after this is noise.

   bash

   lerobot-replay \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM1 \
       --robot.id=my_follower \
       --dataset.repo_id=${HF_USER}/my-task \
       --dataset.episode=0

2. 2
   Open loop on three held-out episodes, every checkpoint

   Same --execution-horizon and same --denoising-steps for all of them, or the comparison is meaningless. Look at the per-joint plots, not only the scalar.

   bash

   for CKPT in 5000 10000 15000 20000; do
     uv run python gr00t/eval/open_loop_eval.py \
       --dataset-path ./my_so100_data \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path /tmp/run/checkpoint-$CKPT \
       --traj-ids 47 48 49 \
       --execution-horizon 16 \
       --save-plot-path ./plots/ckpt-$CKPT.jpeg
   done

3. 3
   Throw away only the obviously broken ones

   Use open loop to eliminate, not to rank. A checkpoint whose predictions are in the wrong numeric range or flat on the gripper dimension is dead. Two checkpoints within a few percent of each other are tied, whatever the decimal places say.

4. 4
   Closed loop on the survivors, fixed N, fixed start states

   Pick N before you start. Twenty rollouts is enough to answer does this work at all. Fifty is the smallest number that lets you rank two candidates. Vary the object start pose deliberately across the rollouts, otherwise you are measuring one initial condition.

5. 5
   Write down N, the successes, and the failure mode

   Not just 14/20. Note what the eight failures looked like: overshoot, early gripper close, freeze, wrong object. That distribution is what tells you whether to record more episodes or change the model, and it is the thing open-loop MSE can never give you.

Where neither test helps, and what to do instead

Both tests share a blind spot: they assume the thing you are measuring is the policy. Often it is not. If inference runs across the public internet, a closed-loop rollout measures your network. The control loop on these models is 20 ms for ACT up to 485 ms for Pi0.5 per action step, and adding round trips on top of that turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place and not for fast reactive motion. For an evaluation you intend to trust, put the inference next to the servos.

The second blind spot is the dataset. If the recorded episodes contain a systematic defect, a wrist camera that shows a frozen frame, or an action stream sampled on a different raster than the images, then the open-loop error is measuring the defect and the closed-loop failures have a data cause that no checkpoint choice will fix. That check belongs before evaluation, not after it: see collecting high-quality VLA training data and, for the underlying dataset mechanics, episodes and the dataset docs.

Finally, if you want context on how the field reports these numbers at all, the arena collects 85 VLA models and 332 benchmark results with every value linked to its paper or model card. Reading two or three of those source pages is the fastest way to see how differently people define success, and how rarely the rollout count is printed next to the percentage. Related reading on this blog: vision-language-action models for the model class, and the SO-100 complete guide for the hardware end of the loop.