Data Augmentation for Robot Learning: What Helps, What Breaks

In image classification, augmentation is nearly free. A cat rotated by fifteen degrees is still a cat, so you flip, rotate, crop and jitter, and the label survives. Manipulation policies are not built that way. The label is an action, usually joint targets or an end effector delta, and it lives in the robot's frame. The image lives in the camera's frame. Augmentation assumes those frames are independent. They are not.

What follows is what the shipped stacks actually apply, read from published config or source on 23 August 2026: the numbers inside the GR00T N1.7 processor config, the augmentation code printed in the Pi0.5 appendix, lerobot's default transform set, and the ablations that say which of these earn their place. Then the part nobody writes down: which augmentations corrupt the action label on a single-arm setup like the SO-100.

Why a cat can be flipped and a grasp cannot

One test settles every candidate: after the transform, is the recorded action still correct for the transformed image? Colour jitter passes. A 20 percent brighter scene still needs the gripper to close in the same place at the same moment. Horizontal flip fails. In the flipped image the cube sits on the left, but the recorded action still reaches right.

Nothing crashes. You get label noise, and label noise in imitation learning has a signature on the real arm: training loss falls to a healthy-looking number, then the policy approaches the object and hesitates. A model trained on contradictory image-action pairs averages them, and the average of reaching left and reaching right is stopping in the middle. That is the symptom on loss falls, policy does nothing.

What the shipped stacks actually apply

Rather than argue from first principles, here is what five widely used stacks send to the model. Treat the version labels as load-bearing: this part of every repo moves.

Two things stand out. Brightness 0.3, contrast 0.4 and saturation 0.5 appear verbatim in both NVIDIA's and Physical Intelligence's stacks. That is an observation, not a causal claim, but when two labs that do not share code land on the same three numbers it is a place to start. The two models sit side by side on the GR00T N1.7 against Pi0.5 comparison.

# arXiv 2504.16054, pi-0.5 appendix, applied to all input images in this order
transforms = [
    augmax.RandomCrop(int(width * 0.95), int(height * 0.95)),
    augmax.Resize(width, height),
    augmax.Rotate((-5, 5)),
    augmax.ColorJitter(brightness=0.3, contrast=0.4, saturation=0.5),
]

GR00T publishes the same information in a less quotable place: the processor config inside the base checkpoint. Fine-tune GR00T N1.7 without overriding anything and these are the values that run.

// nvidia/GR00T-N1.7-3B, processor_config.json -> processor_kwargs
"crop_fraction": 0.95,
"shortest_image_edge": 256,
"image_target_size": [256, 256],   // legacy fallback, unused when crop_fraction is set
"image_crop_size": [230, 230],     // legacy fallback, unused when crop_fraction is set
"random_rotation_angle": 0,
"color_jitter_params": {
  "brightness": 0.3, "contrast": 0.4, "saturation": 0.5, "hue": 0.08
},
"use_albumentations": true,
"state_dropout_prob": 0.2

Colour jitter: the cheap win

Photometric augmentation is the one nobody argues about. It passes the invariance test trivially and targets the failure mode that bites on a desk robot: you record in the afternoon, run the checkpoint in the evening, the light has changed, and the policy behaves differently. A vision-language-action model with a pretrained backbone is fairly robust to that, but fine-tuning is where you teach it that your lighting is part of the task.

There is a conversion trap. GR00T and Pi0.5 quote jitter as torchvision-style magnitudes, where brightness 0.3 means a factor sampled uniformly from 0.7 to 1.3. lerobot's transform config wants explicit ranges. lerobot's own GR00T documentation translates between the two, and the translation is exact.

That is not a coincidence either. When lerobot added a GR00T training path, its documented recipe rebuilds NVIDIA's jitter by hand and throws away the rest of the default set while doing it.

# from the lerobot GR00T policy docs, the LIBERO training recipe
IMAGE_TRANSFORMS='{
  "brightness": {"weight": 1.0, "type": "ColorJitter", "kwargs": {"brightness": [0.7, 1.3]}},
  "contrast":   {"weight": 1.0, "type": "ColorJitter", "kwargs": {"contrast":   [0.6, 1.4]}},
  "saturation": {"weight": 1.0, "type": "ColorJitter", "kwargs": {"saturation": [0.5, 1.5]}},
  "hue":        {"weight": 1.0, "type": "ColorJitter", "kwargs": {"hue":        [-0.08, 0.08]}}
}'

lerobot-train \
  --dataset.repo_id=IPEC-COMMUNITY/libero_spatial_no_noops_1.0.0_lerobot \
  --dataset.image_transforms.enable=true \
  --dataset.image_transforms.max_num_transforms=4 \
  --dataset.image_transforms.tfs="$IMAGE_TRANSFORMS" \
  --policy.type=groot \
  --policy.base_model_path=nvidia/GR00T-N1.7-3B

Random crop: keep it small, and centre it at evaluation

Random crop is the exception in the geometric family and it has the strongest published evidence behind it. In the robomimic study, removing pixel shift randomisation caused a 47 percent relative drop on Square and 35 percent on Transport. The implementation is deliberately conservative: 76 from 84 pixels, 108 from 120, 216 from 240.

Diffusion Policy adopted the same approach and adds the detail that matters more than the crop size: at inference it takes a static centre crop of the same size. Across four independent stacks the crop lands in a narrow band.

• robomimic: 76 of 84, 108 of 120, 216 of 240 pixels, a consistent 90 percent.
• Diffusion Policy: 2x84x84 becomes 2x76x76 in simulation, 2x320x240 becomes 2x288x216 on the real robot. Also 90 percent.
• GR00T N1.7: shortest edge resized to 256, then crop_fraction 0.95, so a 256 by 256 view keeps a 243 by 243 window before being resized back.
• Pi0.5: 95 percent, then resized back to the original width and height.

Geometric augmentation and the action label

Now the part that costs people a weekend. Crop works because it is a small, zero-mean perturbation that keeps the object roughly where it was and is undone by a centre crop at test time. The rest of the geometric family lacks that property.

Flips

A horizontal flip looks like it doubles the dataset for free. It does not. The action in a LeRobot episode is a command that was actually sent to the arm, so mirroring the image without mirroring the action produces a frame whose correct answer is the mirror of its label. Mirroring the action too requires a symmetry a single arm does not have.

BC-Z is the interesting data point, because it uses reflections and is careful about where. Its policy input gets random crops, downsampling and standard photometric augmentation. Random reflections along the x and y axes go only to the human demonstration videos feeding the task embedding, sampled once per video, applied identically to all 20 frames, and none at inference. Longer version in the BC-Z write-up.

Rotation

Rotation about the optical axis means physically rolling the camera. For a fixed third-person camera that is a pose the robot will never see. For a wrist camera it is worse: the image really does rotate when the wrist rotates, and that rotation correlates with the joint state the policy is predicting, so fake rotation puts noise on top of a genuine cue. GR00T N1.7 ships random_rotation_angle at 0 while keeping the code path.

Translation and the wrist camera

Translation is random crop's less careful cousin. A few percent is the same regulariser: it stops the encoder keying on absolute pixel coordinates. Beyond that it breaks the mapping from image position to workspace position, which is what the policy uses in the final centimetres of a grasp. robomimic measured the wrist view too: removing the wrist camera caused a 43 percent relative drop on Transport.

One draw per sample, or one per camera?

This detail is invisible in the config and shows up in behaviour. GR00T applies augmentation through an albumentations ReplayCompose: the first image creates the random parameters and every later image replays them. The handle is created once and carried across the loop over camera views, so front and wrist views of the same moment get the same crop offset and the same colour shift.

# huggingface/lerobot, src/lerobot/datasets/dataset_reader.py, main, 23 Aug 2026
if self._image_transforms is not None:
    for cam in self._meta.camera_keys:
        if cam in self._meta.depth_keys:
            continue
        item[cam] = self._image_transforms(item[cam])

# NVIDIA/Isaac-GR00T, gr00t/model/gr00t_n1d7/processing_gr00t_n1d7.py, main, 23 Aug 2026
replay = None
for view in image_keys:
    transformed_images, replay = apply_with_replay(
        image_transform, view_images, view_masks, replay
    )

Which you want depends on what you are making the model robust to. Independent draws simulate cameras with different white balance and mounting. Shared draws keep multi-view geometry self-consistent, which matters when the policy fuses views to localise one object. Neither is wrong. What is wrong is not knowing which your stack does, then wondering why colour augmentation on a two-camera LeRobot dataset behaves differently from the single-camera run you tested.

Look at the frames before you spend a GPU hour

Both stacks ship a preview tool and both take under a minute. Augmentation bugs are obvious in a contact sheet and invisible in a loss curve.

1. 1
   Render lerobot's transforms on your own dataset

   Writes one PNG per example under outputs/image_transforms, once for the combined pipeline and once for each transform alone. Note the extra: the viewer scripts sit behind dataset_viz, not training.

   bash

   pip install "lerobot[dataset_viz]"
   
   lerobot-imgtransform-viz \
     --repo_id=lerobot/pusht \
     --episodes='[0]' \
     --image_transforms.enable=True

2. 2
   Preview GR00T's mask-guided augmentation

   Writes side by side comparisons of original, augmented and mask under output_dir/<view_name>/. It needs a dataset with masks, and the repo ships one.

   bash

   cd examples/mask-guided-background-suppression
   
   uv run python test_extra_augmentation.py \
     --dataset_path ../../demo_data/cube_to_bowl_5_with_mask \
     --embodiment_tag NEW_EMBODIMENT \
     --modality_config_path so101_config.py \
     --extra_augmentation_config '{"background_noise_transforms": [{"target_mask_values": [0], "p": 1.0}]}' \
     --output_dir /tmp/augmentation_vis --num_frames 5

3. 3
   Change the jitter on a real GR00T fine-tune

   examples/finetune.sh forwards to launch_finetune.py, a tyro CLI over FinetuneConfig. The colour jitter default is written out in the shell script, so you see what you override. crop_fraction and shortest_image_edge are documented as a pair.

   bash

   # examples/finetune.sh default, verbatim:
   # COLOR_JITTER_PARAMS="brightness 0.3 contrast 0.4 saturation 0.5 hue 0.08"
   
   CUDA_VISIBLE_DEVICES=0 NUM_GPUS=1 uv run bash examples/finetune.sh \
     --base-model-path nvidia/GR00T-N1.7-3B \
     --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich \
     --modality-config-path examples/SO100/so100_config.py \
     --embodiment-tag NEW_EMBODIMENT \
     --shortest-image-edge 256 \
     --crop-fraction 0.9 \
     --color-jitter-params "brightness 0.4 contrast 0.4 saturation 0.5 hue 0.08" \
     --output-dir /tmp/so100_finetune

Two ways to run the comparison

Whether augmentation helps on your task has a cheap answer: train the same dataset twice and drive the arm. What differs is how much plumbing sits between you and that answer.

Do it yourselfOn AY-Robots

You clone the trainer, mount the dataset, rent or own the GPU, and run the pair yourself. This is the only path if you want augmentation parameters the trainer form does not expose, which today is all of them.

1. 1
   Convert the dataset if you are training GR00T

   The GR00T loader wants LeRobot v2.x. The converter carries its own pyproject, so run it from its own directory.

   bash

   cd scripts/lerobot_conversion
   uv venv
   source .venv/bin/activate
   uv pip install -e .
   
   python convert_v3_to_v2.py --repo-id <your-user>/<your-dataset>

2. 2
   Run the baseline with augmentation off

   For ACT or SmolVLA this is the default, since image transforms are off unless enabled. Fix the seed so the comparison means something.

   bash

   lerobot-train \
     --dataset.repo_id=<your-user>/<your-dataset> \
     --policy.type=act \
     --seed=1000 \
     --output_dir=outputs/train/act_noaug

3. 3
   Run the same thing with colour only

   Supply your own tfs dict rather than only flipping enable to true, so the default RandomAffine does not join uninvited.

   bash

   lerobot-train \
     --dataset.repo_id=<your-user>/<your-dataset> \
     --dataset.image_transforms.enable=true \
     --dataset.image_transforms.max_num_transforms=4 \
     --dataset.image_transforms.tfs="$IMAGE_TRANSFORMS" \
     --policy.type=act \
     --seed=1000 \
     --output_dir=outputs/train/act_colour

4. 4
   Evaluate on the arm, not on the loss

   Augmentation raises training loss and is supposed to. Judge it by success rate under changed light or a moved lamp.

Budget it before you start

Two runs, not one. If the difference matters it is worth the second GPU hour; if it does not, spend the time recording more episodes.

What augmentation will not fix

Augmentation operates on pixels that already exist. It cannot create parallax, it cannot create a grasp you never demonstrated, and it cannot resolve a contradiction inside your dataset. The training documentation lists what each trainer runs.

• A new camera angle. Cropping a fixed view does not produce the occlusion pattern of a different viewpoint. Move the camera and record again.
• A new object or background. GreenAug is the sharpest data point: record in front of a green screen, then chroma-key real backgrounds in. Its best variant improved on no augmentation by about 65 percent, on standard computer vision augmentation by about 29 percent and on generative augmentation by about 21 percent, all relative, across over 850 demonstrations and 8.2k evaluation episodes.
• Inconsistent camera assignment. Swapped front and wrist streams are a device identification problem, the same family as a camera that is not detected.
• Too few episodes. GR00T and Pi0.5 want at least 50 episodes here, SmolVLA at least 30.
• A policy that only works in one setup. Start at that failure mode page.

Latency deserves plain speech, because it is the limit of remote inference generally. The per-step inference latency of these models ranges from 20 ms for ACT to 485 ms for Pi0.5, and public internet round trips on top turn a working policy into a hesitant one. Remote inference is viable for slow pick and place, not fast reactive motion.

RT-1 is a useful sanity check. Its paper describes the input pipeline in detail, six images at 300 by 300 into an ImageNet-pretrained EfficientNet-B3, and never describes an augmentation stack at all. What it does ablate is the pretrained initialisation: removing ImageNet pre-training decreased unseen task performance by 33 percent. Backbone and data first, pixel transforms second. See how to collect high quality VLA training data and the broader VLA overview.

Defaults worth starting from

For a single arm with one fixed camera and one wrist camera, this is the position supported by everything above.