Seen Versus Unseen Objects: What Zero-Shot VLA Numbers Measure

What the word "unseen" is actually doing

Every vision-language-action model paper has a column called something like "unseen objects". That number is the one people quote, and it is the one that decides whether you buy an arm, because it implicitly promises that the policy you train on your desk will survive a change of scene. The promise is doing far more work than the experiment behind it.

The papers are not lying. "Unseen" is a protocol decision, not a property of the world, and every lab draws the line somewhere different. One holds out the object instance but keeps the category. One holds out the task sentence but keeps every object. One holds out the room. These produce wildly different numbers and all get printed under the same word. If you want to know whether a fine-tuned policy survives contact with your kitchen, read where each paper drew the line.

BC-Z, and the gap between 44 percent and 32 percent

BC-Z (arXiv 2202.02005, submitted 4 February 2022) is where "zero-shot task generalization" entered imitation learning for manipulation at scale. The team collected 25,877 robot demonstrations across 100 tasks, 125 hours of robot time, plus 18,726 human videos. They evaluated on 29 held-out tasks for which no robot demonstrations existed. The abstract reports "24 unseen manipulation tasks with an average success rate of 44%".

In the results section the number changes shape. 44 percent is the mean over the 24 held-out tasks that scored above zero. Over the full held-out set, zeros included, it was 32 percent with four or five distractor objects and 38 percent with one. Conditioned on a human video instead of a language embedding, the same policy scored 4 percent.

Two other results matter more to a desk setup than the headline. A single-task policy trained on 1,000 demonstrations of one task reached 5 percent; the multi-task policy trained across all 100 reached 52 percent on that same task. And on the four easiest held-out tasks, single-task policies trained on 300-plus demonstrations of the held-out task itself scored 0 percent. The dominant failure was "last-centimeter" error: the arm reaches the right object, then fails to close the gripper. Full write-up in the BC-Z article.

RT-2, where the seen number stopped being interesting

RT-2 (arXiv 2307.15818, 28 July 2023) ran about 6,000 real robot trials and split the unseen set three ways, each into easy and hard cases. The appendix table is the most useful table in the VLA literature for this question.

Look at the first column. RT-1 scored 92 on seen tasks; RT-2-PaLI-X scored 91. Inside the training distribution the web-pretrained giant was slightly worse than the small robotics transformer. Everything RT-2 bought sits on the right, where the average goes from 32 to 62. Report only the seen number and RT-2 and RT-1 are the same model.

The ablation is blunter. A 5B RT-2 trained from scratch on robot data, no web pretraining, averaged 9 percent across the unseen splits. The same architecture from a web-pretrained checkpoint reached 42 percent fine-tuned, 44 percent co-fine-tuned, and 63 percent at 55B. Generalisation came from web pretraining and parameter count, not from robot demonstrations. Longer treatment in the RT-2 article, per-benchmark numbers in the RT-2 arena entry.

Open X-Embodiment, and the thing extra robots did not buy

Open X-Embodiment (arXiv 2310.08864, 13 October 2023) is usually summarised as "more robots, better generalisation". The ablation says something more specific. Note first that the RT-X models were trained on a robotics mixture of 9 embodiments, not the full 22 in the dataset, because the dataset kept growing after the experiments ran.

Twelve extra datasets moved unseen-object generalisation from 62 to 61. Within noise, nothing. What moved was the emergent-skills column, 27.3 to 75.8, and those skills are tasks that exist in the Bridge dataset for a different robot. Remove Bridge and it falls to 42.8. Cross-embodiment data did not make the model better at unseen objects; it let the model perform skills somebody else had demonstrated. Only one of those is generalisation in the usual sense.

The bottom row is the one to remember. Strip out web pretraining, train the same 5B from scratch on pooled robot data from 9 embodiments, and both columns go to roughly zero. Robot-data scale alone did not produce generalisation. There is a negative-transfer result too: on the data-rich Google Robot domain RT-1 scored 92 and RT-1-X, the same architecture co-trained on everything, scored 73, because it underfit. Details in the Open X-Embodiment write-up.

Which shift actually breaks a policy

If you are going to hold something out, hold out the thing that matters. Xie, Lee, Xiao and Finn measured this directly in arXiv 2307.03659 (7 July 2023): a real robot policy trained with random crop and photometric augmentation, one environment factor perturbed at a time, six tasks, twelve trials per factor.

Two things fall out. Moving the camera cost more than twice what unfamiliar distractors cost, which the paper attributes to the training set using a fixed head pose throughout. And the factors did not compound: table texture plus background was no worse than table texture alone, so you can fix these one at a time.

LIBERO-Plus (arXiv 2510.13626, 15 October 2025) reached the same conclusion with a much bigger hammer: 10,030 simulated tasks across seven perturbation factors and twenty-one sub-components, run against ten VLA models. The ordering held.

Camera viewpoint and the robot's initial joint configuration were worst for both models, matching the 2023 real-robot study. The authors also report the models were largely insensitive to language variation and, on probing, tend to ignore the instruction entirely. A model can post a strong "unseen instruction" score without reading the instruction, because in a scene holding one plausible object there is only one thing to do.

What published zero-shot numbers do not measure

• Statistical resolution. OpenVLA (arXiv 2406.09246) evaluated 170 rollouts on BridgeData V2, 17 tasks at 10 trials each, and 60 on the Google robot, 12 tasks at 5 trials. RT-2 ran each emergent-capability instruction 5 times. Honest, expensive, standard numbers, and small ones.
• Which subset the average covers. BC-Z's 44 percent excludes the zeros. Check whether a reported mean covers all held-out conditions or only the ones that worked.
• The initial state. Nearly every protocol resets the arm to the same home pose. Perturbing the initial state alone cost OpenVLA-OFT 59.9 points, more than any other factor.
• Partial credit. Long-horizon evaluations often score progress rubrics, not binary success. Pi0.5 scores roughly the percentage of task steps completed, so half the dishes in the sink counts as about 50 percent. Reasonable, but not comparable to a binary pick-and-place rate.
• Anything about your arm. None of it was measured on an SO-100, with your cameras, your lighting and your gripper.

# How much can you conclude from 10 trials?
# Wilson score interval, 95 percent, for a measured success rate.
import math

def wilson(successes, trials, z=1.96):
    p = successes / trials
    denom = 1 + z * z / trials
    centre = (p + z * z / (2 * trials)) / denom
    half = z * math.sqrt(p * (1 - p) / trials + z * z / (4 * trials * trials)) / denom
    return 100 * (centre - half), 100 * (centre + half)

for k, n in [(5, 10), (10, 20), (25, 50), (50, 100)]:
    lo, hi = wilson(k, n)
    print(f"{k}/{n} = {100 * k / n:.0f}%   95% CI {lo:.1f}% to {hi:.1f}%")

# 5/10   = 50%   95% CI 23.7% to 76.3%
# 10/20  = 50%   95% CI 29.9% to 70.1%
# 25/50  = 50%   95% CI 36.6% to 63.4%
# 50/100 = 50%   95% CI 40.4% to 59.6%

Building your own seen and unseen split on one arm

You cannot reproduce a 6,000-trial evaluation on a desk. You can define one split before training and measure it honestly. The unit you are splitting is the episode inside a LeRobot dataset, so the split has to exist in the recording plan, not in a post-hoc filter.

1. 1
   Write the split down before you record anything

   Pick one axis of variation and commit to it in a text file next to the dataset. One axis, not three. "Objects A, B, C are seen, object D is unseen, everything else fixed" is a real experiment. "Some stuff will be different" is not.

   text

   SPLIT: object identity
   seen      : red cube, blue cube, yellow cube  (episodes 0-59)
   unseen    : green cube                       (never recorded)
   held fixed: camera pose, table cloth, lighting, arm mount,
               gripper, home pose, instruction template
   protocol  : 25 rollouts per condition, 5 start positions x 5 repeats
   date      : 2026-08-23

2. 2
   Record the seen set

   The desktop client records episodes, camera streams and joint states from a teleoperation session into LeRobot format. Keep the camera bolted down for every episode. For the manual path, lerobot exposes the same operation as a CLI, verbatim below from the README at version 0.6.1, released 3 August 2026.

   bash

   pip install lerobot     # 0.6.1, requires Python >= 3.12
   
   lerobot-record \
     --robot.type=<robot_name> \
     --teleop.type=<teleoperator_name> \
     --dataset.repo_id=${HF_USER}/my-dataset

3. 3
   Train once, on the seen set only

   No peeking. If the unseen object appears in even one episode the exercise is void. lerobot's default seed is 1000, so a repeated run is reproducible for ACT, SmolVLA and Pi0.5. GR00T's fine-tuning entry point, the tyro CLI in launch_finetune.py, exposes no seed at all, so two GR00T runs on identical data will differ.

   bash

   lerobot-train \
     --policy.type=act \
     --dataset.repo_id=lerobot/aloha_mobile_cabinet

4. 4
   Sanity-check the checkpoint open-loop first

   Open-loop evaluation replays recorded episodes and compares predicted actions against ground truth. It is cheap and catches a dead checkpoint in a minute. It cannot measure generalisation: it scores against a dataset the model trained on unless you held episodes back. Command from the Isaac-GR00T README.

   bash

   uv run python gr00t/eval/open_loop_eval.py \
       --dataset-path <DATASET_PATH> \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path <CHECKPOINT_PATH> \
       --traj-ids 0 \
       --execution-horizon 16
   
   # writes /tmp/open_loop_eval/traj_{traj_id}.jpeg with MSE metrics

5. 5
   Run seen and unseen back to back, same session

   25 rollouts seen, then 25 unseen, identical start positions, alternating if you can. Record every trial as a 0 or a 1. Do not stop early because the numbers look good, and compute the intervals before concluding anything.

   bash

   # For a simulated dry run of the same idea, LIBERO-Object holds out
   # object identity by construction: 10 tasks, each a pick-and-place
   # of a different object.
   lerobot-eval \
     --policy.path=lerobot/pi0_libero_finetuned \
     --env.type=libero \
     --env.task=libero_object \
     --eval.n_episodes=10

One note on that command. lerobot's LiberoEnv defaults to task libero_10, accepts libero_spatial and libero_object, and runs at 20 Hz to match robosuite's control frequency. LIBERO itself (arXiv 2306.03310) is four suites and 130 tasks; LIBERO-Spatial, LIBERO-Object and LIBERO-Goal have 10 tasks each and isolate exactly one thing. It is simulation, so treat a result there as a smoke test, never as evidence about your hardware.

pip install lerobot

# 1. record the seen split
lerobot-record --robot.type=<robot_name> \
  --teleop.type=<teleoperator_name> \
  --dataset.repo_id=${HF_USER}/seen-objects

# 2. train on it and nothing else
lerobot-train --policy.type=act \
  --dataset.repo_id=${HF_USER}/seen-objects

# 3. simulated smoke test on held-out object identity
lerobot-eval --policy.path=<checkpoint> \
  --env.type=libero --env.task=libero_object \
  --eval.n_episodes=10

# 4. real rollouts: your own harness, your own tally sheet

What actually buys generalisation with one arm and fifty episodes

The literature is consistent and the answer is unwelcome. RT-2's ablation says web pretraining and parameter count. Open X-Embodiment's bottom row says a from-scratch model on pooled robot data generalises to nothing. Pi0.5 (arXiv 2504.16054, 22 April 2025) says scene count: roughly 400 hours of mobile manipulator data across about 100 home environments, with a scaling study over 3, 12, 22, 53, 82 and 104 locations. Only at 104 did the model match a control trained directly on the test homes. Generalisation to the 101st home was bought by collecting data in 100.

You are not going to record 100 rooms. What you can do is pick a policy family whose pretraining already carries the semantic breadth, and spend your fifty episodes on the motion rather than the vocabulary. The five trainable policies here differ sharply on that axis.

Practical recommendation: train ACT first to prove the task is learnable and your data is clean, then fine-tune SmolVLA or GR00T N1.7 on the same dataset and measure the unseen column on both. The head-to-head comparison lists what changes. If the VLA does not beat ACT on your unseen split, the pretraining is not helping on your task and the money belongs in more episodes.

Where this platform does not help

Being clear about this beats another feature list. AY-Robots does not measure generalisation for you. No button runs a held-out evaluation, scores it and hands you a confidence interval. The split is your design decision, the rollouts your labour, the tally sheet your spreadsheet. What the platform removes is GPU provisioning, training-step plumbing and the cost of a second run.

If you have no hardware at all, the live arm streams a physical SO-100 with no signup on a queue, which at least shows how much a two-centimetre offset in object placement changes the problem. And if your policy trains fine but does nothing on the arm, the split is not your issue yet: start at loss falls, policy does nothing.