Quality Control for Robot Demonstrations: A Review Workflow

The problem is triage, not detection

Everybody who records LeRobot datasets eventually writes a script that finds broken episodes. The script is the easy part. The hard part is that a 300 episode dataset contains roughly 300 things you could look at, each one takes about a minute to watch properly, and you have a GPU rental waiting. So the script gets written, gets run once, produces a list of 40 suspicious episodes, and then nobody reviews the list because reviewing the list is the expensive bit.

A review workflow fixes that by ordering the checks so that every stage only receives what the stage before it could not settle. Cheap and automatic first, expensive and human last. The point is not to catch every defect. The point is to spend your review minutes on the episodes most likely to be worth reviewing, and to know what you skipped.

The rest of this article walks each gate with the commands that run it. The worked example is a real public dataset, lerobot/svla_so101_pickplace: 50 episodes, 11,939 frames at 30 fps, two cameras, six joints on an SO-100 class follower arm. You can point every command below at your own dataset instead, or at anything on the public dataset directory.

Gate 0: does the trainer accept the format at all

This gate takes ten seconds and saves the most wasted GPU hours, because the failure it catches happens after the pod is already billing. Read meta/info.json and check four things: the codebase version, the frame rate, the feature schema, and the episode count.

python - <<'EOF'
import json
from huggingface_hub import hf_hub_download

p = hf_hub_download("lerobot/svla_so101_pickplace", "meta/info.json", repo_type="dataset")
info = json.load(open(p))
print(info["codebase_version"], info["robot_type"], info["fps"], "fps")
print(info["total_episodes"], "episodes,", info["total_frames"], "frames")
for k, v in info["features"].items():
    print(" ", k, v["dtype"], v["shape"])
EOF

v3.0 so100_follower 30 fps
50 episodes, 11939 frames
  action float32 [6]
  observation.state float32 [6]
  observation.images.up video [480, 640, 3]
  observation.images.side video [480, 640, 3]
  timestamp float32 [1]
  frame_index int64 [1]
  episode_index int64 [1]
  index int64 [1]
  task_index int64 [1]

Three checks fail here often enough to be worth naming. First, the codebase version against the policy you plan to train: GR00T N1.7 and GR00T N1.5 want LeRobot v2.0 or v2.1, and a v3.0 dataset crashes the GR00T loader, so it has to be converted down first (see dataset rejected as v3). ACT, SmolVLA and Pi0.5 take v3.0. Second, the episode count against the minimum the policy needs: 30 for SmolVLA, 50 for ACT, Pi0.5, GR00T N1.5 and GR00T N1.7. Third, the camera keys, because a policy config that names observation.images.front will not find a dataset that recorded observation.images.up.

Gate 1: the statistics pass that costs 84 KB

This is the gate most people skip, and it is the one with the best ratio of information to effort. A LeRobot v3.0 dataset stores per-episode summary statistics in the metadata itself, in meta/episodes/chunk-000/file-000.parquet. Reading that file gives you, for every episode and every feature, the min, max, mean, std and count, plus the episode length in frames, the task string, and the pointers into the shared shards: a chunk and file index, the row range into the parquet (dataset_from_index and dataset_to_index), and the start and end timestamp inside each MP4.

The size difference is the whole argument for running this first. For the example dataset the full repository is 86.1 MB, of which 85.7 MB is the two MP4 shards. The entire meta/ directory is 84,121 bytes, about one tenth of one percent of the repository. You can rank every episode in the dataset before downloading any video.

#!/usr/bin/env python3
"""Gate 1: rank LeRobot v3.0 episodes from metadata alone. No video is decoded."""
import json, sys
import numpy as np
import pyarrow.parquet as pq
from huggingface_hub import snapshot_download

repo = sys.argv[1] if len(sys.argv) > 1 else "lerobot/svla_so101_pickplace"
root = snapshot_download(repo, repo_type="dataset", allow_patterns=["meta/*"])
info = json.load(open(f"{root}/meta/info.json"))
t = pq.read_table(f"{root}/meta/episodes").to_pandas()

def flat(v):                    # image stats arrive as nested per-channel arrays
    a = np.asarray(v, dtype=object).ravel()
    return np.array([flat(x)[0] if isinstance(x, np.ndarray) else float(x) for x in a])

def col(name):
    return np.stack([flat(v) for v in t[name]])

def rz(x):                      # robust z score: median absolute deviation, not std
    med = np.median(x)
    mad = np.median(np.abs(x - med)) or 1e-9
    return (x - med) / (1.4826 * mad)

joints = info["features"]["action"]["names"]
cams = [k.split("observation.images.")[1] for k in info["features"] if ".images." in k]
length = t["length"].to_numpy().astype(float)
span = col("stats/action/max") - col("stats/action/min")

flags = {"length": np.abs(rz(length)) > 3}
for j, name in enumerate(joints):
    flags["span:" + name.split(".")[0]] = np.abs(rz(span[:, j])) > 3
for cam in cams:
    lum = col(f"stats/observation.images.{cam}/mean").mean(1)
    flags["light:" + cam] = np.abs(rz(lum)) > 3

for i in range(len(t)):
    hit = [k for k, v in flags.items() if v[i]]
    if hit:
        print(f"episode {i:>4}  {int(length[i]):>4} frames  {', '.join(hit)}")

n = sum(any(v[i] for v in flags.values()) for i in range(len(t)))
print(f"\n{n}/{len(t)} episodes flagged for review, {len(t) - n} clean")

$ python audit.py lerobot/svla_so101_pickplace
episode   13   231 frames  span:gripper
episode   27   203 frames  light:up
episode   28   224 frames  light:up, light:side
episode   30   288 frames  span:gripper, light:side
episode   37   227 frames  span:gripper

5/50 episodes flagged for review, 45 clean

Five episodes out of fifty go to a human. Forty-five do not. That is the entire job of gate 1, and the output is already more useful than a flat list, because of how the flags group.

• Episodes 27, 28 and 30 are flagged on brightness, and they are consecutive. Consecutive flags on an environment feature are not three bad operators, they are one lighting change: a lamp switched on, a blind moved, the sun came round. Peak to peak brightness across the whole run is 9.2 percent on the up camera and 12.3 percent on the side camera.
• Episodes 13, 30 and 37 are flagged on gripper travel, and they are scattered. Scattered flags on an action feature are operator variance: the gripper opened less far on those attempts. Median gripper span across the set is 23.22 units, the smallest is 17.47.
• Episode 30 carries two independent flags. Episodes that trip more than one rule are where a reviewer should start.

One honest limit of this gate, visible in the metadata itself. The count field for the image statistics is 100 on every episode, while the count for observation.state equals the episode length (303 for episode 0). Image statistics are computed from a sample of 100 frames per episode. That is plenty to detect a lighting shift across a whole episode. It is useless for finding a single frozen frame or a two second camera dropout, which is a different check that does require decoding video.

The other structural limit: summary statistics describe the envelope, not the path. Two episodes with identical per-joint min and max can trace completely different trajectories, one smooth and one full of the jerk that comes from a stuttering leader-follower link. Gate 1 will not separate them. That is what gate 2 is for.

Gate 2: replay only what gate 1 flagged

Now you decode video, but only for five episodes instead of fifty. LeRobot ships a viewer that renders camera streams, joint states and actions on a shared timeline.

1. 1
   Open the worst offender first

   Episode 30 tripped two rules, so it goes first. The viewer opens a rerun.io window with every camera and every joint on one timeline. It sits behind an optional extra: install lerobot[viz] first, or the command exits telling you rerun-sdk is missing. The same extra ships the Foxglove backend used in step three.

   bash

   lerobot-dataset-viz \
       --repo-id lerobot/svla_so101_pickplace \
       --episode-index 30

2. 2
   Point it at a local recording instead

   If the dataset came off your own machine rather than the Hub, add --root and switch the mode. This is the normal case right after a recording session with the desktop client.

   bash

   lerobot-dataset-viz \
       --repo-id my-user/pick-place-run3 \
       --root ./my_local_data_dir \
       --mode local \
       --episode-index 30

3. 3
   Scrub instead of watching, if you prefer a timeline

   The Foxglove display mode starts a WebSocket server on port 8765 and serves the episode as a seekable timeline, which is faster than watching linearly when you already know roughly where the problem is.

   bash

   lerobot-dataset-viz \
       --repo-id lerobot/svla_so101_pickplace \
       --episode-index 30 \
       --display-mode foxglove
   # then connect Foxglove to ws://127.0.0.1:8765

4. 4
   Write the verdict down as you go

   Keep a plain text file with one line per reviewed episode: index, verdict, reason. You will need the index list intact for gate 4, and you will want the reasons when you decide whether to re-record.

   bash

   cat >> review.tsv <<'EOF'
   30	drop	light changed mid-episode, gripper never fully opened
   13	keep	short grip travel but the cube was already close
   27	keep	brightness only, task completed cleanly
   EOF

One more viewer is worth knowing, and it is worth being clear that it is one tool and not two. The hosted LeRobot Dataset Visualizer Space is the deployed instance of the open source huggingface/lerobot-dataset-visualizer repo (Apache-2.0, same README, run it yourself with bun install and bun dev). Either way you get the same app: synchronized video and time series charts, and a filtering panel that flags low movement, jerky motion and outlier episode length, then exports the flagged episode ids as a ready to run LeRobot CLI command. Its heuristics overlap with gate 1, but it is built for looking at one episode closely rather than ranking fifty at once. Use it when you want to look, not when you want to rank. The upstream question of how to record demonstrations that need less of this in the first place is covered in collecting high quality VLA training data.

Gate 3: the sampled human review

Everything so far finds episodes that are statistically odd. None of it finds an episode where the arm moved smoothly, the lighting was fine, the gripper closed properly, and the operator picked up the wrong object. That failure looks completely normal in every summary statistic, and it is the one that quietly teaches your policy the wrong thing.

Sample size, not full coverage. At 30 fps, the 11,939 frames of the example dataset are 6.6 minutes of footage per camera, 13 minutes for both. That is watchable. Scale to 500 episodes and it is more than an hour per camera, which nobody does twice. A workable rule: review every episode gate 1 flagged, plus a fixed random 10 percent of the unflagged ones. The random sample is not there to catch defects. It is there to measure your false negative rate, so you find out whether gate 1 is calibrated for your task before you trust it on the next dataset.

If somebody else recorded the data, this gate is also where operator variance shows up. DROID was collected by 50 people across three continents over 12 months, 76,000 trajectories in 564 scenes; the RoboTurk work collected 137.5 hours from remote workers. At that scale the reviewer is not checking whether a single episode is broken, they are checking whether one operator's habit has become a systematic bias in the data. The same applies at small scale as soon as two people take turns on the teleoperation station.

Gate 4: act on the verdict, then re-check

LeRobot has a first-party editing CLI. It handles the bookkeeping that makes manual deletion dangerous: renumbering the remaining episodes, rewriting the parquet and MP4 offsets, and pruning tasks that no episode references any more.

# delete every episode you marked "drop", in ONE call, into a NEW repo
lerobot-edit-dataset \
    --repo_id my-user/pick-place-run3 \
    --new_repo_id my-user/pick-place-run3-clean \
    --operation.type delete_episodes \
    --operation.episode_indices "[30, 41, 44]"

# fix a language annotation instead of deleting the episode
lerobot-edit-dataset \
    --repo_id my-user/pick-place-run3-clean \
    --operation.type modify_tasks \
    --operation.episode_tasks '{"12": "Pick up the blue cube and place it in the bin"}'

# confirm what survived
lerobot-edit-dataset \
    --repo_id my-user/pick-place-run3-clean \
    --operation.type info \
    --operation.show_features true

Then run gate 1 again against the cleaned repo. This is not ceremony. Pruning changes the reference distribution, so episodes that sat just inside the threshold before may sit outside it now, and, more usefully, you find out whether you removed a defect or removed the evidence of one. If your gripper-span flags disappear entirely after deleting three episodes, the problem was three episodes. If new ones appear, the problem is your calibration and no amount of deleting will fix it. That distinction is the difference between a dataset you can train on and a recording session you need to repeat.

What the published curation work actually found

It is easy to treat pruning as damage control. The research points the other way: filtering is a lever, and a fairly strong one. Three results are worth carrying into your own workflow.

The Belkhale result is the one that changes how you read your own flags. If diversity were unambiguously good, every unusual episode would be worth keeping and gate 1 would be pointing you at your best data. It is not that simple: the paper separates variation in the action you took at a given state, which hurts, from variation in how the world responds, which is just noise you have to model. An episode where the operator took a different but equally valid route to the goal is diversity worth keeping. An episode where the operator took a worse route to the same goal is action divergence, and it makes the policy hesitate at exactly that state. Both look identical in a min-max table. Only a person watching can tell them apart, which is why gate 3 does not go away.

The CUPID and DemInf numbers come with a caveat worth stating plainly, because it constrains what you can do before your first training run. CUPID ranks demonstrations by their influence on a policy's expected return, which requires evaluation rollouts, which requires a trained policy. You cannot run it on day one. DemInf is unsupervised and needs no policy, which makes it the closer analogue to what gate 1 does, just with a learned embedding instead of hand-written rules. Neither replaces looking at the video.

Two ways to run this

pip install "lerobot[viz]"   # the viz extra is what ships lerobot-dataset-viz

# gate 0 + 1: metadata only, no video
python audit.py my-user/pick-place-run3

# gate 2: replay only the flagged indices
lerobot-dataset-viz --repo-id my-user/pick-place-run3 --episode-index 30

# optional: automated scoring across nine dimensions
git clone https://github.com/RoboticsData/score_lerobot_episodes.git
cd score_lerobot_episodes && pip install -e .
python score_dataset.py --repo_id my-user/pick-place-run3 --threshold 0.5

# gate 4: prune into a new repo
lerobot-edit-dataset \
    --repo_id my-user/pick-place-run3 \
    --new_repo_id my-user/pick-place-run3-clean \
    --operation.type delete_episodes \
    --operation.episode_indices "[30, 41, 44]"

What to automate, and what never to automate

The split is not about difficulty. It is about whether the check has a ground truth that exists independently of your intent.

What this workflow cannot tell you

Being honest about the boundary matters more than the checks themselves, because a green audit report creates confidence that the audit did not earn.

• It cannot tell you the task is learnable. A dataset can pass every gate and still be impossible to learn from, because the information the policy needs (an occluded object, a force, a state the cameras never see) was never recorded. Nothing in the metadata reveals a missing observation.
• It cannot predict closed-loop success. Every metric here is computed on the demonstration in isolation. The published methods that do relate data to closed-loop return, such as CUPID, need evaluation rollouts from a trained policy first.
• It cannot separate a hard episode from a bad one. An episode that took 40 percent longer because the object started in an awkward pose is exactly the data you want; the same statistics describe an episode where the operator was distracted.
• It cannot fix a policy that fails at inference but trained fine. That is usually an environment gap or a latency problem rather than a data defect, and it lives on the loss falls but the policy does nothing and policy freezes mid-motion pages.
• It does not document the dataset. The Datasheets for Datasets proposal is worth following here: record who collected it, on what hardware, under what lighting, with what task wording, and what you pruned and why. Six months later that file is worth more than the audit output.

One last framing. The reason to run this before renting a GPU is not that the GPU is expensive. On the 24 GB tier a SmolVLA run is a couple of dollars, and even a GR00T N1.7 run on an A100 lands between 4 and 12 USD. The reason is attribution. If you train on unaudited data and the policy is bad, you cannot tell whether the data was bad, the hyperparameters were wrong, or the task is hard, so the next thing you do is guess. An audited dataset does not guarantee a good policy. It guarantees that when the policy is bad, you are debugging something else. That is worth three seconds and 84 KB.