What a Good Robot Demo Video Shows, and What It Hides

A demo video is a sample of size one

A robot demo video is evidence. It is just very weak evidence, and it is weak in a specific, measurable way. What you are shown is one execution of a task. What you want to know is the distribution that execution was drawn from. Those two things are related by a number, the number of attempts, and that number is almost never on screen.

This is not an accusation of dishonesty. Recording a clean take is a normal part of communicating research, and the same clip can be an honest illustration or a misleading claim depending entirely on what is said around it. The problem is that the surrounding information is usually missing, and the viewer fills the gap with an optimistic assumption. This article is about closing that gap: what to look for in the footage, what the policy papers actually report, and what you can verify yourself in about ten minutes.

The arithmetic of a lucky take

Start with the part that needs no domain knowledge. If a policy succeeds with probability p on an independent attempt, the probability of getting at least one success in m attempts is 1 minus (1 minus p) to the power m. Put realistic numbers in and the result is uncomfortable.

from math import sqrt

def at_least_one(p, m):
    return 1 - (1 - p) ** m

def wilson(k, n, z=1.96):
    """95% confidence interval for a success rate of k out of n."""
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return max(0.0, centre - half), min(1.0, centre + half)

for p in (0.2, 0.3, 0.5):
    print(p, [round(at_least_one(p, m), 2) for m in (5, 10, 20)])

for k, n in ((4, 5), (9, 10), (20, 25), (45, 50), (170, 200)):
    lo, hi = wilson(k, n)
    print(f"{k}/{n} = {100 * k / n:.0f}%  95% CI {100 * lo:.0f}-{100 * hi:.0f}%")

The first loop prints the numbers that should make you cautious. A policy with a true success rate of 20 percent produces at least one clean take in 10 attempts 89 percent of the time. At 30 percent it is 97 percent. Filming twenty attempts and keeping one is not fraud, it is an afternoon. The cost of producing a convincing video is close to zero even for a policy that fails four times out of five.

The second loop prints the interval you should mentally attach to any reported success rate. This is the number that decides whether two policies in a comparison are actually different.

None of this makes small evaluations useless. It makes single numbers from small evaluations uninformative, which is a different and more actionable problem. The fix that the field has converged on is not more trials for their own sake, it is better metrics and matched conditions. Vision-language-action models in particular tend to fail in graded ways rather than binary ones, which is why partial-credit scoring has started to replace binary success.

What the papers actually report

Before criticising a video, it helps to know what the primary literature commits to. These are the evaluation protocols as stated in the papers, not as summarised by press coverage. They vary by more than an order of magnitude.

The gap between the top and the bottom of that table is the whole story. A 25-trial single-seed result and a 2700-rollout blind randomised study are both published, both peer-reviewed in some form, and both get turned into a 30-second video that looks the same. The video is not where the difference lives.

# Terminal 1: serve the policy
uv run python gr00t/eval/run_gr00t_server.py \
    --model-path nvidia/GR00T-N1.7-3B \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
    --device cuda:0

# Terminal 2: open-loop evaluation against recorded trajectories
uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path demo_data/droid_sample \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
    --host 127.0.0.1 \
    --port 5555 \
    --traj-ids 1 2 \
    --execution-horizon 8

Five edits that change what you see

1. Cuts

Evan Ackerman, who has written the IEEE Robotics and Automation Society guidance on robot videos, puts it plainly: he assumes multiple cuts means multiple takes. That is the correct prior. A cut between robot approaches object and robot lifts object removes exactly the part where these policies fail most often.

You can count the cuts without watching frame by frame. ffmpeg's scene-change filter reports every frame where the visual content jumps, and ffprobe tells you the real duration and frame rate.

# duration, declared frame rate and frame count
ffprobe -v error -select_streams v:0 \
  -show_entries stream=r_frame_rate,avg_frame_rate,nb_frames \
  -show_entries format=duration \
  -of default=noprint_wrappers=1 demo.mp4

# list every hard cut (scene change above the threshold)
ffmpeg -hide_banner -nostats -i demo.mp4 \
  -filter:v "select='gt(scene,0.3)',showinfo" \
  -f null - 2>&1 | grep pts_time

A high cut count is not proof of anything on its own. A promotional edit is allowed to have cuts. What matters is whether the cuts fall inside a single claimed execution. Ackerman's advice to producers is to show one uninterrupted take first and only then cut to other angles, and that is exactly the structure worth looking for.

2. Speed

Playback speed is the cheapest edit and the hardest to spot by eye, because most viewers have no calibrated sense of how fast a robot arm should move. The relevant number is the control loop. On the five policies AY-Robots trains, a single action step costs between 20 ms and 485 ms.

Those figures come from the AY-Robots policy comparison. Read them with one correction in mind: all of these models use action chunking, so a single forward pass emits a chunk of future actions that the robot plays out while the next chunk is computed. That is why a Pi0.5 rollout does not look like two twitches per second. It also means raw per-step latency alone cannot tell you whether a video was sped up.

The honest way to settle it is instrumentation rather than inference. LeRobot's lerobot-rollout prints a cadence report at the end of every run: effective control frequency against the target, how many cycles blew the budget, and where the time went across observe, process, infer, send and record. If a demo is accompanied by that block, the speed question is answered. If it is not, and the video has no on-screen speed label, treat the pace as unknown.

3. Retries

A retry inside a take is not a problem, it is often the most interesting thing in the clip. Recovery behaviour is a real capability and a policy that regrasps after a slip is better than one that does not. The problem is retries between takes, presented as one. This is the failure mode the Toyota Research Institute large behavior model study designs against by running policies in randomised bundles: each bundle covers one initial condition, every policy in the comparison gets that condition, and the order is shuffled so that drift in lighting or object wear does not favour whoever went first.

There is a subtler version. The University of Pennsylvania GRASP lab ran over 300 trials of pi0 in the wild and then published six clips they describe, in their own words, as cherry-picked rollouts that highlight the model's strengths. That is fine, because they say so, and because they also publish the failure taxonomy: out-of-distribution objects, spatial reasoning errors on height, freezing mid-task, collisions with unfamiliar appliances. They call the whole exercise a vibe check rather than a benchmark. Selection plus disclosure is honest. Selection alone is not.

4. The reset you do not see

Between two takes, someone puts the objects back. How precisely they do that is one of the largest and least discussed sources of variance in robot evaluation. The TRI study handles it by giving the human resetter a camera overlay of the desired initial scene and asking them to match it. RoboArena requires evaluators to closely match initial conditions within an A/B pair while deliberately allowing variation between pairs.

Why this matters more than it sounds: The Colosseum benchmark varies 20 manipulation tasks along 14 axes of environmental perturbation, things like object colour, texture, size, lighting, distractors and camera pose. A single perturbation cuts success rates by 30 to 50 percent. Combined perturbations cut them by 75 percent or more. A demo filmed in the exact scene the data was collected in is measuring something close to the training distribution, which is a much easier problem than the one the video appears to claim. This is common enough on small custom datasets that it has a dedicated failure page.

5. Who is driving

Autonomous footage and teleoperated footage are visually identical from outside the robot. This is not hypothetical. When 1X demonstrated its NEO home humanoid in October 2025, the complex portions of the demo were run by remote human operators, and the company was open that teleoperation is part of the product plan rather than a temporary crutch. The IEEE RAS guidance asks producers to state on the video itself how the robot is being controlled, ranging over human in the loop, human supervised, scripted actions, partial autonomy and full autonomy, and to put that text on the footage rather than in the description where it will be stripped by every repost.

For a small arm this question is less loaded but still worth asking, because leader-follower teleoperation produces motion that is smoother and more decisive than most trained policies. If a clip of an SO-100 looks better than everything else you have seen from that hardware class, teleoperation is the first hypothesis, not the last.

Run the demo yourself

The fastest way to stop arguing about a video is to reproduce the number. For anything built on imitation learning with an open checkpoint, that is a few commands, and the result is worth more than any amount of frame analysis. Here is the protocol, borrowed from the studies above and cut down to what one person with one arm can actually run in an afternoon.

1. 1
   Fix the initial conditions before you start

   Photograph the scene you will reset to and keep the picture open on a second screen. This is the camera-overlay trick from the TRI study, done cheaply. Write down object positions in centimetres from a table edge so a second person can reproduce them.

2. 2
   Decide the trial count in advance

   Twenty trials per condition is the practical floor for a claim about one policy: it gets a 90 percent observed rate down to roughly a 20 point interval. Fifty is what the TRI study uses per task per policy per condition. Decide before you look at the results, otherwise you will stop at a flattering moment.

3. 3
   Run the rollouts and log the cadence

   Use the same command every time and let it print its timing report. The cadence block is your proof that the run was real time.

   bash

   lerobot-rollout \
     --strategy.type=base \
     --policy.path=${HF_USER}/act_policy \
     --robot.type=so101_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
     --display_data=true \
     --task="put the brick in the box" \
     --duration=60

4. 4
   Score with partial credit, not just success or failure

   Record a progress score from 0 to 100 alongside the binary outcome, the way RoboArena does. Recent work on sequential testing (arXiv 2603.13616, March 2026) finds that fine-grained progress separates competing policies faster than binary success, cutting evaluation burden by up to 70 percent against standard batch methods.

5. 5
   Repeat under one perturbation

   Move the object 10 cm, or change the lighting, and run 20 more. The Colosseum result says to expect a 30 to 50 percent relative drop from a single perturbation. If your drop is much larger, you have a generalisation problem rather than a capability.

6. 6
   Publish the denominator

   Whatever you release, put the trial count, the autonomy level and the playback speed on the video itself. That single caption is the difference between a demo and a claim.

   bash

   ffmpeg -i raw.mp4 -vf \
     "drawtext=text='18/20 trials, autonomous, 1x speed':\
   x=20:y=h-60:fontsize=28:fontcolor=white:box=1:boxcolor=black@0.5" \
     -c:a copy demo.mp4

Where this platform helps, and where it does not

Being specific about the limits is more useful than a pitch. AY-Robots can shorten the path from a claim to your own measurement. It cannot tell you whether someone else's video was edited.

• It does not analyse videos, detect cuts, or estimate playback speed. The ffprobe commands above are the whole toolkit for that, and they are free.
• It does not verify third-party benchmark claims. The arena links each number to its source; reading the source is still your job.
• It cannot make a remote evaluation as valid as a local one. Inference has to sit next to the servos for fast tasks. With a control loop of 20 to 485 ms per action step, adding public-internet round trips turns a working policy into a hesitant one. Remote inference is workable for slow pick-and-place and unsuitable for fast reactive motion, and a video recorded over a fast local link will not reproduce over a slow remote one.
• It does help with the parts that are pure friction: renting the right GPU tier, converting dataset versions, serving a checkpoint, and getting a policy back onto an arm so you can count rollouts instead of arguing about frames.
• The same operations are available from a terminal via the CLI and to agents via the MCP server, which is what you want if you are scripting a twenty-rollout evaluation rather than clicking through it.

A checklist you can paste into a review

Six questions. If a video answers all six, it is doing its job. If it answers none, it is a moodboard.

1. How many attempts? Give the numerator and the denominator, per task, and say whether the shown clip is representative or selected.
2. Autonomous, supervised, or teleoperated? State it on the footage, not in the description.
3. What playback speed? Real time, or the multiplier, on screen. Publish the cadence log if you have one.
4. One take or several? If several, say so, and show one uninterrupted take first.
5. Who set the scene, and how precisely? Matched initial conditions, or free arrangement, and whether the evaluator knew which policy was running.
6. What happens one perturbation away? Move the object, change the light, and report that number too.

The interesting consequence of taking this seriously is that it gets easier to be impressed by the right things. A 40 percent success rate on a genuinely hard task, stated openly with 25 trials behind it and a failure reel attached, is a much stronger result than a flawless clip with no denominator. Once you have run your own twenty rollouts on a fine-tuned ACT or SmolVLA policy and watched the number come out lower than you hoped, other people's honest numbers start to read as competence rather than weakness.

If you want the practical route rather than the critical one, the run your first policy walkthrough and the dataset recording tutorial cover the mechanics, and the data collection guide covers why most demos that fail were doomed at recording time rather than at training time. The SO-100 complete guide is the starting point if you do not have an arm yet.