Finding Broken Episodes in a Robot Dataset Before You Train

Training is no longer the expensive part. A GR00T N1.7 job is 4 to 12 USD of rented A100 or H100 time; an ACT or SmolVLA run on a 24 GB card is 1 to 3 USD (see pricing). What costs real money is the two days afterwards, when the arm reaches for the cube, stops ten centimetres short, and you have no idea why.

Usually the answer was in the dataset, findable in twenty minutes with ffmpeg and a parquet reader. This is the audit to run on a LeRobot dataset before it goes near a GPU. Four defects recur in teleoperated recordings: frozen frames, swapped camera streams, joints parked at a calibrated limit, and timing skew between image and action.

Four defects and what each one does to the policy

Each defect has one signature in the data and another in the behaviour of the trained policy.

Know the layout before you grep it

An audit is a set of file reads, so you need to know which file holds what. LeRobotDataset v3.0 shipped in lerobot 0.4.0: many episodes are concatenated into shared files, and boundaries are resolved through metadata. See the docs on datasets.

your-dataset/
  meta/
    info.json      # features, dtypes, shapes, fps, codebase_version, path templates
    stats.json     # dataset-wide min/max/mean/std/count per feature (quantiles only
                   #   if the writer was recent enough to compute them)
    tasks.parquet  # task strings mapped to integer task ids
    episodes/
      chunk-000/file-000.parquet   # per episode: length, tasks, the row range into
                                   #   data/, a timestamp range per video, and stats
  data/
    chunk-000/file-000.parquet     # frame rows, MANY episodes per file
  videos/
    observation.images.front/chunk-000/file-000.mp4   # MANY episodes per file

# v2.1 -> v3.0, exactly as the v3.0 docs page writes it
python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=<HF_USER/DATASET_ID>

# v3.0 -> v2 for GR00T: the downgrade lives in the Isaac-GR00T repo
python scripts/lerobot_conversion/convert_v3_to_v2.py --help

Defect 1: frozen frames

A USB camera that loses its stream usually does not throw. It hands back the last decoded frame while the loop keeps running. LeRobot is explicit: a background thread stores a frame plus a capture timestamp, and read_latest, which is what the SO-100 follower calls to build an observation, raises only when the buffered frame is older than max_age_ms, default 500 ms. At 30 fps that is fifteen stale frames passing without complaint.

The consequence is specific. If your wrist camera is what tells the model when to close the end effector, frozen wrist frames teach it that gripper closure is not predictable from that camera, so it leans on proprioception. The policy reaches correctly and grips at the wrong moment.

# freezedetect compares the mean absolute difference of all components against a
# noise floor. Defaults: noise n=-60dB (a 0.001 difference ratio), duration d=2 s.
# It logs, and sets, lavfi.freezedetect.freeze_start / freeze_duration / freeze_end.
ffmpeg -hide_banner -i videos/observation.images.wrist/chunk-000/file-000.mp4 \
  -vf freezedetect=n=-60dB:d=0.5 -map 0:v:0 -f null - 2>&1 | grep freezedetect

# mpdecimate marks near-duplicates of the previous frame.
# Defaults: hi=64*12, lo=64*5, frac=0.33. Count the DROPS, not every line that
# mentions drop_count: at -loglevel debug the filter prints one line per frame.
ffmpeg -hide_banner -i videos/observation.images.wrist/chunk-000/file-000.mp4 \
  -vf mpdecimate -loglevel debug -f null - 2>&1 | grep -c ' drop pts:'

Defect 2: cameras that swapped places

This produces the most confusing bug reports: the dataset is internally consistent and only wrong relative to the arm. The OpenCVCamera docstring says the quiet part out loud, that camera indices can be unstable across reboots or port changes, especially on Linux, where lerobot enumerates by globbing /dev/video* in name order. A reboot or a different hub puts the wrist stream into the column labelled front, and a block of episodes is mislabelled.

1. 1
   Pin the devices before recording again

   lerobot-find-cameras enumerates what is attached and writes a sample image per device. On Linux, prefer a by-id path over an integer index.

   bash

   lerobot-find-cameras opencv          # enumerates devices, saves one sample image each
   ls -l /dev/v4l/by-id/                # pass these stable paths as index_or_path

2. 2
   Dump a contact sheet

   One thumbnail every 20 seconds of shard, tiled 8x8 into a PNG per camera. If episodes 40 to 61 carry the wrist view in the front column, scrolling shows it.

   bash

   for f in videos/observation.images.front/chunk-*/file-*.mp4; do
     ffmpeg -hide_banner -loglevel error -i "$f" \
       -vf "fps=1/20,scale=160:-1,tile=8x8" -frames:v 1 "sheet_$(basename $f .mp4).png"
   done

3. 3
   Confirm numerically

   Not the msad filter: it compares two videos to each other, needs matching resolution and pixel format, and says nothing about which moves more. Measure motion energy inside each stream.

   bash

   # Mean luminance of the frame-to-frame difference, per stream. A wrist camera
   # moves with the arm, an overhead camera does not, so the wrist number is much
   # larger. If that ordering flips for a block of episodes, the streams swapped.
   for v in front.mp4 wrist.mp4; do
     printf "%-12s " "$v"
     ffmpeg -hide_banner -i "$v" \
       -vf "tblend=all_mode=difference,signalstats,metadata=print:key=lavfi.signalstats.YAVG" \
       -f null - 2>&1 | awk -F= '/YAVG/{s+=$2;n++} END{printf "mean %.3f over %d frames\n", s/n, n}'
   done

4. 4
   Relabel or delete, not half of each

   If the swap is a clean block and both cameras share resolution and geometry, renaming columns is legitimate. If the fields of view differ, delete it.

   bash

   lerobot-edit-dataset --repo_id you/your-dataset \
     --new_repo_id you/your-dataset-clean \
     --operation.type delete_episodes \
     --operation.episode_indices "[40, 41, 42, 43]" 

A missing camera is a different failure, camera not detected. The walkthrough is record your first dataset; what makes a session worth keeping is in how to collect high-quality VLA training data.

Defect 3: joints parked at their calibrated limit

The behaviour here is deliberate, and it hides the problem. Calibration on an SO-100 writes four numbers per motor: drive_mode, homing_offset, range_min, range_max. Normalising a servo reading clamps first.

# src/lerobot/motors/motors_bus.py, _normalize()
bounded_val = min(max_, max(min_, val))
if self.motors[motor].norm_mode is MotorNormMode.RANGE_M100_100:
    norm = (((bounded_val - min_) / (max_ - min_)) * 200) - 100
    normalized_values[id_] = -norm if drive_mode else norm
elif self.motors[motor].norm_mode is MotorNormMode.RANGE_0_100:
    norm = ((bounded_val - min_) / (max_ - min_)) * 100
    normalized_values[id_] = 100 - norm if drive_mode else norm
elif self.motors[motor].norm_mode is MotorNormMode.DEGREES:
    mid = (min_ + max_) / 2
    max_res = self.model_resolution_table[self._id_to_model(id_)] - 1
    normalized_values[id_] = (val - mid) * 360 / max_res   # note: val, not bounded_val

In robots/so_follower/so_follower.py the five arm joints get RANGE_M100_100 and the gripper RANGE_0_100, unless you set use_degrees. If calibration used a narrower range than the arm can reach, or the leader was pushed past the follower's range during teleoperation, the column becomes a flat line at exactly -100, +100, 0 or 100. That line is what the model learns as the correct action for the whole stretch.

Detect the clamp as a run, not as a threshold

The obvious test, flagging a joint that spends some share of an episode near its extreme, does not work, and real data shows why. The public lerobot/svla_so101_pickplace dataset is 50 episodes, 11939 frames, 30 fps, v3.0. Its elbow_flex column sits within 0.5 of the +100 limit in 29.1 percent of frames, flagging 49 of 50 episodes. Almost all of that is servo quantisation near the top of travel: values step 99.09, 99.18, 99.27, which is motion, not a clamp. Only 0.49 percent read exactly 100.0.

import pyarrow.parquet as pq
import numpy as np, json, glob

info    = json.load(open("meta/info.json"))
fps     = info["fps"]
names   = info["features"]["action"]["names"]
MIN_RUN = int(0.5 * fps)          # half a second held against a stop

def longest_run_at_limit(col, lo, hi, eps=1e-3):
    at = (col <= lo + eps) | (col >= hi - eps)
    best = cur = 0
    for pinned in at:
        cur = cur + 1 if pinned else 0
        best = max(best, cur)
    return best

for path in sorted(glob.glob("data/chunk-*/file-*.parquet")):
    t   = pq.read_table(path, columns=["episode_index", "action"])
    ep  = np.asarray(t["episode_index"])
    act = np.stack(t["action"].to_numpy(zero_copy_only=False))
    for e in np.unique(ep):
        a = act[ep == e]
        for j, name in enumerate(names):
            lo, hi = (0.0, 100.0) if name.startswith("gripper") else (-100.0, 100.0)
            run = longest_run_at_limit(a[:, j], lo, hi)
            if run >= MIN_RUN:
                print(f"ep {e:>4}  {name:<18} {run:>4} frames "
                      f"({run / fps:.1f} s) pinned at a calibrated stop")

# Reuses names and act from the snippet above. How much of each
# calibrated range does the task actually use?
for j, name in enumerate(names):
    lo, hi = (0.0, 100.0) if name.startswith("gripper") else (-100.0, 100.0)
    c = act[:, j]
    print(f"{name:<18} [{c.min():>7.1f}, {c.max():>6.1f}]  "
          f"uses {(c.max() - c.min()) / (hi - lo):>4.0%} of the calibrated range")

The hardware fix is to recalibrate with the joint driven to its true mechanical stops; lerobot-find-joint-limits does that by teleoperating for a fixed window and recording the extremes reached. Also: SO-100 and SO-101 run Feetech STS3215 servos on 7.4 V and 12 V destroys them, so check your arm on the SO-100 page. For symptoms, see gripper does not close and joint stops early.

Defect 4: timing skew between image and action

Every row carries a timestamp, an observation and an action, and the contract is that they describe the same instant. Two things break it: spacing, when the control loop misses a slot and a gap opens that is not 1/fps, and capture latency, since the camera thread and the loop are separate. Neither is checked at load.

The recording script is talkative about row two. Its CycleTimer warns: Control loop is running slower (X Hz) than the target FPS (Y Hz). Dataset frames might be dropped and robot control might be unstable. Common causes are: 1) Camera FPS not keeping up 2) Policy inference taking too long 3) CPU starvation. If that appeared while recording, those episodes are suspect.

import pyarrow.parquet as pq
import numpy as np, json, glob

fps = json.load(open("meta/info.json"))["fps"]

for path in sorted(glob.glob("data/chunk-*/file-*.parquet")):
    t  = pq.read_table(path, columns=["episode_index", "timestamp"])
    ep = np.asarray(t["episode_index"]); ts = np.asarray(t["timestamp"])
    for e in np.unique(ep):
        d = np.diff(ts[ep == e])
        bad = int((np.abs(d - 1.0 / fps) > 1e-4).sum())    # lerobot's tolerance_s
        if bad:
            print(f"ep {e:>4}  {bad:>5} gaps off 1/fps  "
                  f"median {np.median(d)*1e3:.2f} ms  worst {d.max()*1e3:.2f} ms")

The twenty minute audit, start to finish

Cheapest checks first, because each can save you the next. These apply equally to files written by the desktop client, whose guide is at docs/client-guide.

1. 1
   Read the header and check the episode gate

   Format version, fps, features, episode count. GR00T, Pi0.5 and ACT want 50 episodes here, SmolVLA 30; below that, nothing to audit.

   bash

   lerobot-edit-dataset --repo_id you/your-dataset --root . \
     --operation.type info --operation.show_features true
   
   python -c "import json;i=json.load(open('meta/info.json'));\
   print(i['codebase_version'], i['fps'], i['total_episodes'], i['total_frames'])" 

2. 2
   Histogram the episode lengths

   The fastest signal in the audit. A two second episode is a mis-trigger; three times the median usually means a mid-take recovery. The reference dataset runs a median 230 frames.

   bash

   python - <<'EOF'
   import pyarrow.parquet as pq, numpy as np, glob
   L = []
   for p in sorted(glob.glob('meta/episodes/chunk-*/file-*.parquet')):
       L += list(np.asarray(pq.read_table(p)['length']))
   L = np.array(L); m = np.median(L)
   print('n', len(L), 'median', m, 'p05', np.percentile(L,5), 'p95', np.percentile(L,95))
   print('suspects:', np.where((L < 0.4*m) | (L > 2.5*m))[0])
   EOF

3. 3
   Scan the video for freezes

   One pass per camera shard. Map freeze_start back to episodes with the from_timestamp fields in meta/episodes.

   bash

   for f in videos/*/chunk-*/file-*.mp4; do
     echo "== $f"
     ffmpeg -hide_banner -i "$f" -vf freezedetect=n=-60dB:d=0.5 \
       -map 0:v:0 -f null - 2>&1 | grep freeze_start
   done

4. 4
   Look at what you flagged

   Never delete on a threshold alone. Watch each flagged episode against its joint traces.

   bash

   pip install 'lerobot[dataset_viz]'
   lerobot-dataset-viz --repo-id you/your-dataset --episode-index 47
   
   # headless: save the .rrd recording and pull it down
   lerobot-dataset-viz --repo-id you/your-dataset --episode-index 47 \
     --save 1 --output-dir /tmp/rrd

5. 5
   Delete into a new repo id

   Pass --new_repo_id so the original survives; without it the operation edits in place.

   bash

   lerobot-edit-dataset \
     --repo_id you/your-dataset \
     --new_repo_id you/your-dataset-clean \
     --operation.type delete_episodes \
     --operation.episode_indices "[7, 12, 40, 41, 42, 43, 47, 88]" 

Doing this by hand versus doing it on AY-Robots

The audit is not something this platform does for you. What changes is the loop around it: where the dataset comes from, whether the trainer accepts its format, and how fast you get from suspicion to a second run.

pip install "lerobot[dataset_viz]"
lerobot-info                    # python, torch, ffmpeg versions in one block

hf download you/your-dataset --repo-type dataset --local-dir ./ds

# --root is the folder holding meta/, data/, videos/. Without it the CLI looks
# in $HF_LEROBOT_HOME/<repo_id>, not in the directory you happen to be sitting in.
lerobot-edit-dataset --repo_id you/your-dataset --root ./ds \
  --operation.type info --operation.show_features true

# audit, then prune into a NEW repo id
lerobot-edit-dataset --repo_id you/your-dataset --root ./ds \
  --new_repo_id you/your-dataset-clean \
  --operation.type delete_episodes \
  --operation.episode_indices "[7, 12, 47]" \
  --push_to_hub true

What the published curation work found

Deleting episodes feels wasteful. The literature says it is not. Data quality has been a first-class variable in imitation learning since robomimic, which ran six offline algorithms on five simulated and three real tasks and found performance depended on demonstration quality and stopping criterion, not just quantity. Every method below needs a trained policy to score against, making them second-pass tools; the four defects above are findable first.

After the prune: what to re-check

Deleting episodes writes a new dataset and re-aggregates the statistics the trainer normalises with, from the per-episode stats you kept. Reload once and print the ranges before queueing.

from lerobot.datasets import LeRobotDataset

ds = LeRobotDataset("you/your-dataset-clean")

# meta.info is a DatasetInfo dataclass, not a dict, so index it with attributes.
# The metadata object also exposes the common counters directly:
print(ds.meta.total_episodes, ds.meta.total_frames, ds.meta.fps)
print(ds.meta.stats["action"]["min"], ds.meta.stats["action"]["max"])

# q01/q10/q50/q90/q99 exist only if the dataset was written by a lerobot new
# enough to compute them. Ask before you index.
print(sorted(ds.meta.stats["action"].keys()))

# load a subset without rewriting anything, to A/B a suspicion quickly
sub = LeRobotDataset("you/your-dataset", episodes=[0, 1, 2, 3, 4])
print(len(sub))

Then run the cheapest training you have. ACT or SmolVLA on the 24 GB tier costs 1 to 3 USD and finishes in 2 to 5 hours, a reasonable canary even when the model you want is GR00T. ACT has no base model, so the run tells you about your data. Compare the five at /policies; guides such as GR00T N1.7 on SO-100 list the defaults each trainer sends.

What an audit cannot tell you

A dataset can pass every check above and still be a bad dataset.

• Task coverage. Thirty flawless episodes with the cube in one corner give a policy that only works in that corner, and no file check sees it.
• Operator style drift. Two people recording, one from above and one from the side, gives data consistent per episode and bimodal overall.
• Whether the demonstration was good. Reaching the goal through three corrections is a successful episode and a poor demonstration; that is what influence-function methods are for.
• Whether the arm was calibrated correctly rather than consistently, and whether your model's inference latency fits the control loop.

The latency point is where remote inference breaks. Per-step action latency runs from 20 ms for ACT to 152 ms for GR00T N1.7, 165 for GR00T N1.5, 245 for SmolVLA and 485 for Pi0.5. Public-internet round trips on top turn a working policy into a hesitant one: remote inference suits slow pick-and-place, not fast reactive motion. Background is in vision-language-action models; the hardware side is the SO-100 complete guide.