Generalisation: What Breaks a Trained Robot Policy

A policy that works is a policy that works under the conditions it was measured in. Move the camera three centimetres and a checkpoint that scored 90 percent yesterday can score 20 today. Change the tablecloth and it may not notice. That asymmetry has been measured factor by factor.

Below is what the ablations say about which changes a trained policy survives, plus a test you can run on an SO-100 in an afternoon. Every number comes from a paper linked beside it.

What generalisation means when you trained the thing yourself

In the papers it means held-out tasks. For anyone who fine-tuned ACT or SmolVLA on fifty of their own episodes that is the wrong frame: the task is fixed, what moves is the room. Xie, Lee, Xiao and Finn made this precise by decomposing an environment into independent factors.

A behaviour cloning policy maps pixels and joint angles to actions. It cannot tell which pixels are the task and which are the room. Whatever correlated with the right action in your LeRobot dataset is a valid feature, including your monitor's shadow and the table edge in frame.

The measured ranking, from two independent studies

Xie et al. (arXiv, July 2023) ran one checkpoint on a real mobile manipulator across two lighting conditions, three sets of distractors, three table textures, three backgrounds and two camera poses, twelve trials each. That policy already used random crop and photometric augmentation, so these are numbers for a hardened model.

Two details matter more than the ordering. Background and table texture used the same patterned paper, so 88.9 against 52.8 is clean: not the pattern, but where it is. The pairs also do not compound: of 21 factor pairs in their simulated benchmark, 16 landed within plus or minus 6 percent of the harder single factor.

LIBERO-Plus (Fei et al., revised December 2025) ran the same idea against modern VLA checkpoints in simulation across seven dimensions. It is the closest thing to a robustness datasheet for models people actually download.

Same ordering, only harsher. Note rows two and three: OpenVLA-OFT with a wrist camera holds 59.7 percent under viewpoint shift, the same model with only the third-person view holds 16.8. The authors credit close-range cues that do not move when the scene camera does.

The list that kills a policy

1. Moving the camera

First by a wide margin. Jiang et al. (2025) pinned the mechanism: policies without explicit camera extrinsics infer the viewpoint from static background cues, and that shortcut collapses as soon as the mount shifts. The policy was never viewpoint-invariant, it was reading the wall.

Xie et al. measured how the damage scales. Sampling camera and table position over radii of 0.025, 0.050 and 0.075 metres, the gap roughly doubles at each step: the gap at 2.5 cm is about half the gap at 5 cm and a quarter of the gap at 7.5 cm.

2. Where the object starts

LIBERO-Plus reports robot initial state as the most expensive factor for several models: Pi0 loses 87.6 points, OpenVLA-OFT 59.9. They split object layout into 'add confounders' and 'move the target'. Clutter barely hurt; moving the target hurt badly. The models learned where the target sits, not what it is.

If every one of your episodes starts with the cube in the same square of the mat, you trained a policy that reaches for that square. Randomising the start pose is cheap while recording and impossible afterwards. See the recording walkthrough and the SO-100 data collection page.

3. The surface the task happens on

Table texture is the factor people misfile as background: 38.9 points where the background cost 2.8. In simulation, table position was about as hard as table texture, and object texture about as hard as camera position. Anything inside the manipulation region is a foreground feature.

4. Motions the arm never performed

A ceiling, not a difficulty. The RT-2 authors state it in their limitations: pretraining boosts generalisation over semantic and visual concepts, but the robot acquires no new motions. Their failure figure shows the model reaching correctly for a pen and a banana, then failing on contact dynamics it had only seen with blocks.

So a vision-language-action model fine-tuned on pick-and-place data will not learn to unscrew a lid because the backbone read the internet. If the motion is not in your episodes it is not in the policy. That is the boundary of fine-tuning on 50 to 100 demonstrations.

The list a policy usually survives

• New backgrounds: 88.9 percent against a 91.7 baseline, and among the smallest drops in LIBERO-Plus.
• New distractors: 80.6 percent. Confounders barely moved Pi0, Pi0-FAST, RIPT-VLA, UniVLA or WorldVLA.
• New lighting, with a wrist camera. OpenVLA-OFT lost 11.3 points, the third-person-only variant 27.1.
• New instances of a familiar object. Pi0.5 scored seen categories with new instances separately from unseen categories, and the former improved faster with scale.
• Photometric sensor noise: 20.4 points off OpenVLA-OFT, 14.8 off Pi0.

Language does much less than the acronym implies

LIBERO-Plus found language perturbations produced one of the smallest average drops, which sounds like robustness. It is not. They replaced the instruction with an empty string and OpenVLA-OFT's object-suite score barely moved. Then they swapped the target inside the instruction, from 'pick up the alphabet soup' to 'pick up the tomato sauce' with both present: success fell close to zero and the model kept reaching for the original target.

The apparent robustness came from ignoring language: a nominal VLA can degenerate into a vision-action model. Worth knowing before you spend a week on task strings for one SmolVLA fine-tune.

BC-Z is the instructive contrast. Jang et al. trained on 25,877 demonstrations across 100 tasks, then gave the policy unseen sentences: non-zero success on 24 of 29 held-out tasks, averaging 44 percent on those 24. Single-task policies trained on 300 or more demonstrations of the four easiest held-out tasks scored 0 percent. Language generalisation is real, but a hundred tasks bought it. Our BC-Z walkthrough covers that dataset.

What actually buys generalisation

Diversity, not volume

RT-1 ran the cleanest version: cap episodes per task (same diversity, less data), then delete the smallest tasks (almost all the data, fewer tasks).

Removing 25 percent of the tasks while keeping 97 percent of the data cost about as much generalisation as deleting 49 percent of the data. Lin et al. (ICLR 2025) agreed from the other direction, over 40,000 demonstrations and 15,000 real rollouts: generalisation scales roughly as a power law with the count of objects, environments or environment-object pairs. Hold those fixed and add demonstrations and there is no power law.

For one SO-100 on one desk that is uncomfortable, because you have one environment. The translation: move the mat, change the lighting, shift the table and rotate the base between recording blocks. The minimum of 50 episodes for GR00T N1.7 or 30 for SmolVLA is easier to spend across six setups than one.

Augmentation, which is off by default

In lerobot (main branch checked 24 August 2026, version 0.6.2) image augmentation is opt-in. ImageTransformsConfig ships with enable False, max_num_transforms 3, random_order False, and six transforms: brightness and contrast over (0.8, 1.2), saturation (0.5, 1.5), hue (-0.05, 0.05), sharpness (0.5, 1.5), and a RandomAffine of plus or minus 5 degrees with 0.05 translation.

lerobot-train \
  --dataset.repo_id=youruser/so100_pick_place \
  --policy.type=act \
  --dataset.image_transforms.enable=true \
  --dataset.image_transforms.max_num_transforms=3 \
  --dataset.image_transforms.random_order=true \
  --output_dir=outputs/act_aug

Two caveats. The default set is photometric plus a weak affine, with no random crop, and random crop is exactly what Xie et al. found most valuable, helping most on new camera positions and table textures. It also helped non-spatial factors, which they called surprising.

1. 1
   Look at what the transforms do to your frames

   lerobot ships a visualiser that dumps the original frame plus per-transform minimum, mean and maximum examples. Run it before training.

   bash

   lerobot-imgtransform-viz \
     --repo_id=youruser/so100_pick_place \
     --episodes='[0]' \
     --image_transforms.enable=True
   # writes to outputs/image_transforms/<dataset>/

2. 2
   Hold out episodes so you get a number, not a vibe

   DatasetConfig carries eval_split, and the trainer refuses eval_steps unless eval_split is above zero. A held-out loss is not a success rate, but it is the cheapest signal there is.

   bash

   lerobot-train \
     --dataset.repo_id=youruser/so100_pick_place \
     --policy.type=act \
     --dataset.eval_split=0.1 \
     --eval_steps=1000 \
     --max_eval_samples=200

3. 3
   Drop episodes you know are broken

   exclude_episodes takes indices and applies on top of any episodes filter. An off-distribution episode is a generalisation problem in disguise.

   bash

   lerobot-train \
     --dataset.repo_id=youruser/so100_pick_place \
     --dataset.exclude_episodes='[7,19,42]' \
     --policy.type=act

For scene robustness on a small budget, GreenAug (Teoh et al., 2024) is the strongest published result: record against a green screen, chroma-key the background out, replace it with random textures. GreenAug-Rand beat no augmentation by roughly 65 percent across eight tasks, and generative augmentation by about 21 percent. Random backgrounds beat semantically plausible ones like kitchens and living rooms, so the policy is learning to stop caring what is behind the object.

Pretraining, which buys semantics and nothing else

RT-2's headline table is the clearest evidence that a large pretrained backbone changes generalisation without changing in-distribution skill. On seen tasks RT-1 scored 92, RT-2-PaLI-X-55B 91, RT-2-PaLM-E-12B 93.

Roughly 2x off-distribution, none on seen tasks. That is the argument for a vendor base checkpoint: GR00T N1.7, GR00T N1.5 or Pi0.5 over ACT. ACT has no base model at all. Our RT-2 explainer covers the co-fine-tuning recipe behind those numbers.

A generalisation test you can run in an afternoon

Because the factors are mostly additive you do not need a factorial grid. One column per factor, everything else at baseline.

Run each cell with lerobot's rollout CLI. Its episodic strategy mirrors teleoperated recording: the policy drives each episode, an optional leader arm resets the scene between them.

# One cell of the grid: 20 policy-driven episodes with manual resets
lerobot-rollout \
  --strategy.type=episodic \
  --policy.path=outputs/act_aug/checkpoints/last/pretrained_model \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
  --teleop.type=so100_leader \
  --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=youruser/rollout_gen_camera_nudge \
  --dataset.num_episodes=20 \
  --dataset.single_task="Grab the cube"

For slower policies add RTC inference. Pi0.5 runs at 485 ms per action step and SmolVLA at 245 ms on this platform, and one synchronous call per tick makes them look hesitant for reasons unrelated to generalisation. Use --inference.type=rtc so latency is not misread as a camera shift. See inference latency and action chunking.

Two ways to run this

Harden the policy, or lock the setup down?

Two legitimate strategies, and most people should pick the second first. Hardening means diverse data, augmentation, a wrist camera and a bigger backbone. Locking down means tape, one mat and a scripted reset pose.

Do both: lock down what you control, and spend the diversity budget on what you cannot. The policy only works in one setup and loss falls, policy does nothing pages cover the diagnosis order.

Where this platform does not help

• Nobody moves the camera for you. The factor grid is manual work with a real arm.
• There is no automatic success detector for a real rollout. You score by hand.
• The arena numbers are in-distribution scores. They say which model is stronger on a benchmark, not which survives a bumped tripod.
• Remote inference makes latency worse. The control loop is 20 to 485 ms per action step, and public-internet round trips on top turn a working policy into a hesitant one. Viable for slow pick-and-place, not reactive motion.
• A LeRobot v3.0 dataset crashes the GR00T loader and must be converted down to v2.1, so a rebuild is also a format decision. See dataset rejected v3.
• GR00T training is not seedable: its fine-tuning entry point exposes no seed, so its runs are not bit-for-bit reproducible. ACT, SmolVLA and Pi0.5 do take a seed, and lerobot's default is 1000.

None of that is a reason to skip the measurement, only to budget an afternoon before concluding a model is bad. Most reports of a policy that does not work describe one that works in exactly one configuration nobody wrote down. If you have recorded nothing, train your first policy and the data collection guide are the place to start; the training docs cover the form.