Lighting Robustness for Robot Policies: Why It Breaks at Night

Why a policy stops working at night

You record 60 episodes in the afternoon. Daylight through a window, two overhead LEDs, a cube on a mat. You fine-tune a GR00T N1.7 or SmolVLA checkpoint and it works. That evening the sun is gone, and the same checkpoint hesitates over the cube, closes the gripper on air, or drifts two centimetres and stalls. Nothing about the robot changed, and nothing about the policy weights changed. Only the photons changed.

This is the most-reported entry on policy only works in one setup, and three things happen at once. Optical: shadow direction flips, colour temperature drops from about 6500 K daylight to 2700-4000 K indoor. Electronic: the camera driver reacts and hands the network an image it never saw during fine-tuning. Statistical: the network learned 'the mat is this shade of grey' as a stand-in for 'the mat'. Only the electronic one is free to fix.

What actually changes in the image

Turning the lights down is not one transform. It is at least four separate changes to the tensor reaching the vision encoder, and only two of them resemble a brightness slider.

The camera fights you before the policy does

V4L2 on Linux exposes the relevant controls directly. V4L2_CID_EXPOSURE_ABSOLUTE is in 100 microsecond units, so 1 means 1/10000 s. V4L2_CID_WHITE_BALANCE_TEMPERATURE is in Kelvin, and the kernel docs say a driver should cover at least 2800 (incandescent) to 6500 (daylight). V4L2_CID_POWER_LINE_FREQUENCY takes DISABLED (0), 50HZ (1), 60HZ (2) or AUTO (3), and exists to suppress flicker.

# what does this camera actually expose?
v4l2-ctl --device=/dev/video0 --list-ctrls-menus

# typical uvcvideo names on a recent kernel
# auto_exposure: 1 = manual, 3 = aperture priority (auto)
v4l2-ctl -d /dev/video0 -c auto_exposure=1
v4l2-ctl -d /dev/video0 -c exposure_time_absolute=250      # 25 ms
v4l2-ctl -d /dev/video0 -c white_balance_automatic=0
v4l2-ctl -d /dev/video0 -c white_balance_temperature=4600
v4l2-ctl -d /dev/video0 -c power_line_frequency=1          # 50 Hz mains

# confirm the writes stuck; some drivers ignore them mid-stream
v4l2-ctl -d /dev/video0 --list-ctrls | grep -E 'exposure|white_balance|power_line'

What the published ablations actually say

There is more folklore than data here, so it is worth naming the studies. The most useful is Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation (Xie, Lee, Xiao and Finn, July 2023). They varied one environment factor at a time on language-conditioned imitation learning policies, in simulation and on a real robot, and released the benchmark (Factor World: 19 tasks, 11 factors, Apache-2.0). Their conclusion: new backgrounds, distractors and lighting are the easier factors, new table textures and camera positions the harder ones. The real-robot numbers behind that ordering, read from Figure 6 for the no-augmentation policy:

RT-1 (Brohan et al., December 2022) is the second data point. Its L1 tier is defined as a new counter-top layout and new lighting, bundled; L2 adds unseen distractors, L3 drastically new settings. That the authors did not isolate lighting is itself informative. On the separate robustness slices RT-1 held 83% with distractors and 59% against changed backgrounds, against 47% and 41% for BC-Z, on roughly 130k episodes over 700-plus tasks, 13 robots, 17 months. BC-Z shows the mechanism more cleanly: its protocol deliberately included background variation from recording in multiple locations, and it reached 44% on held-out tasks with no robot demonstrations at all. Diversity was engineered in, not bolted on as a transform.

What augmentation fixes

Photometric augmentation simulates the sensor-level part of a lighting change: a global brightness shift, a colour cast, a contrast change, a gamma curve. It is applied on the fly at training time and costs nothing at inference. In LeRobot it is also off unless you switch it on.

The transform registry in that file is larger than the documentation page suggests. The docs list ColorJitter, SharpnessJitter and Identity; the source registers nine custom transforms, several written specifically for lighting. Defaults below are read from main on 2026-08-24, so check them against the version you have installed.

PlanckianJitter is the interesting one. Rather than jittering hue arbitrarily it samples a black-body temperature between 3000 K and 15000 K and applies the matching red and blue channel scaling while preserving green, following Zini et al. (2022). That output actually resembles swapping a daylight lamp for a tungsten one. Generic hue jitter produces colours no real illuminant can, which is that paper's argument against plain colour jitter.

IMAGE_TRANSFORMS='{
  "brightness": {"weight": 1.0, "type": "ColorJitter", "kwargs": {"brightness": [0.7, 1.3]}},
  "contrast":   {"weight": 1.0, "type": "ColorJitter", "kwargs": {"contrast":   [0.6, 1.4]}},
  "gamma":      {"weight": 1.0, "type": "GammaCorrection", "kwargs": {"gamma": [0.5, 2.0]}},
  "planckian":  {"weight": 1.0, "type": "PlanckianJitter", "kwargs": {"temperature": [3000, 9000]}},
  "patches":    {"weight": 1.0, "type": "GaussianPatchBrightness", "kwargs": {}},
  "shadow":     {"weight": 1.0, "type": "RandomShadow", "kwargs": {"opacity": [0.2, 0.5]}}
}'

lerobot-train \
  --dataset.repo_id=$HF_USER/$DATASET_NAME \
  --dataset.image_transforms.enable=true \
  --dataset.image_transforms.max_num_transforms=3 \
  --dataset.image_transforms.tfs="$IMAGE_TRANSFORMS" \
  --policy.type=smolvla --policy.device=cuda \
  --batch_size=2 --steps=20000 --seed=1000 \
  --output_dir=outputs/train/$DATASET_NAME

# look at the frames before you spend GPU hours on them
lerobot-imgtransform-viz \
  --repo-id=$HF_USER/$DATASET_NAME \
  --output-dir=./transform_examples --n-examples=5

The same knob on the GR00T side

Isaac-GR00T has its own fine-tuning entry point rather than going through lerobot-train, and exposes colour jitter as a tyro CLI flag. The SO-100 walkthrough in getting_started/finetune_new_embodiment.md passes it explicitly.

1. 1
   Convert the dataset if it is v3.0

   GR00T reads a flavour of LeRobot v2 with an extra meta/modality.json, so a v3.0 dataset will not load. The repo ships the converter in its own environment. See dataset rejected: v3 if the loader complains.

   bash

   cd scripts/lerobot_conversion
   uv venv && source .venv/bin/activate
   uv pip install -e . --verbose
   python convert_v3_to_v2.py --repo-id <DATASET_REPO_ID>

2. 2
   Fine-tune with explicit colour jitter

   The flag takes alternating key-value pairs, not JSON. These values come from the repository's SO-100 example. Omitting the flag is not the same as disabling augmentation: the field defaults to None, which the docstring says falls back to the colour jitter baked into the pretrained model config.

   bash

   CUDA_VISIBLE_DEVICES=0 uv run python \
       gr00t/experiment/launch_finetune.py \
       --base-model-path nvidia/GR00T-N1.7-3B \
       --dataset-path ./demo_data/cube_to_bowl_5 \
       --embodiment-tag NEW_EMBODIMENT \
       --modality-config-path examples/SO100/so100_config.py \
       --num-gpus 1 --output-dir /tmp/so100 \
       --max-steps 2000 --global-batch-size 32 \
       --color-jitter-params brightness 0.3 contrast 0.4 saturation 0.5 hue 0.08 \
       --dataloader-num-workers 4

3. 3
   Decide whether to unfreeze the vision tower

   In gr00t/configs/finetune_config.py, tune_visual and tune_llm both default to False; only tune_projector and tune_diffusion_model are on. Colour jitter therefore regularises the projector and action head, not the encoder. Unfreezing it is a real option and a real way to overfit 50 episodes.

   bash

   --tune-visual True   # adds the visual encoder to the trainable set

4. 4
   Or suppress the background instead of jittering it

   The repo carries a mask-guided background suppression example that applies noise or tint only to masked regions, described as useful for sim-to-real transfer or for stopping the model overfitting to static backgrounds. It needs per-frame masks, which is real extra work.

   bash

   --extra_augmentation_config '{"background_noise_transforms": [{"target_mask_values": [0], "p": 1.0}]}'

What augmentation does not fix

The honest boundary is global against spatial. Photometric transforms apply one function to every pixel, or in the case of GaussianPatchBrightness and RandomShadow, one crude mask unrelated to your actual scene geometry. Real lighting changes are geometric. What stays broken after every colour knob is turned:

• Cast shadows tied to the arm. Under one lamp the gripper throws a hard shadow onto the target that moves with joint angles and reads as a second object. No global transform generates it.
• Specular highlights. A shiny jaw or laminated table produces a blown-out patch whose position depends on lamp, surface and pose. Brightness jitter dims the frame, it does not add a highlight.
• Motion blur correlated with speed. A long exposure blurs fast joints more than slow ones, so blur carries action information; MotionBlur samples independently of the action, the wrong correlation.
• A different background. That is what GreenAug was built for; its authors found scene replacement beat the standard baseline of random photometric distortion plus random shift.
• Colour that carries task meaning. If the task is 'pick the red block' and your hue range turns it orange, you have taught the network to ignore the feature the task depends on.

There is a fourth case that is not a lighting problem at all and constantly gets diagnosed as one: the policy was never good. One that only worked by memorising a fixed approach looks lighting-sensitive because nothing makes it robust. Move the object 5 cm under the same lighting you trained in. If it fails there too, read loss falls but the policy does nothing and fix that before touching augmentation.

What only more data fixes

The Pi0.5 ablation should reset expectations: roughly 100 distinct environments was the point where a model generalised to a held-out home about as well as one trained inside it. You will not record 100 rooms with a single teleoperated SO-100. But that curve is steep early, and the practical reading is that going from one lighting condition to four captures most of the accessible gain.

1. 1
   Freeze everything that is not light

   Same camera position, mount, object set and task phrasing. If the tripod moves between blocks you have contaminated the experiment with the worst factor in the Xie table.

   bash

   v4l2-ctl -d /dev/video0 --list-ctrls | grep -E 'exposure|white_balance'
   # write these down, reuse identical values in every block

2. 2
   Record blocks, not a shuffle

   Four conditions: daylight only, daylight plus overhead, overhead only, one desk lamp at an angle. Fifteen episodes each rather than 60 in one. The desktop client writes LeRobot format straight from the teleop session; walkthrough at record your first dataset.

3. 3
   Hold one condition out

   Train on three, evaluate on the fourth. Without a held-out condition you cannot tell whether augmentation helped or whether you drew a better seed, and with GR00T there is no seed to set.

4. 4
   Then add augmentation on top

   Augmentation and diversity are complementary, not alternatives. Two checkpoints on the same three-condition dataset, one flag apart, both evaluated on the fourth. That is the comparison that answers the question for your setup.

   bash

   lerobot-train --dataset.repo_id=$D --dataset.image_transforms.enable=false --seed=1000 ...
   lerobot-train --dataset.repo_id=$D --dataset.image_transforms.enable=true  --seed=1000 ...

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=$HF_USER/lighting-blocks \
  --dataset.num_episodes=15 \
  --dataset.single_task="Grab the black cube"

Deciding which failure you actually have

Before spending money, spend ten minutes. Each row is a test you can run in the room.

The short version

Lock the camera first: free, and often the whole answer. Turn on photometric augmentation second: one flag, off by default. Record a second and third lighting condition third, because no transform substitutes for it. Only then reach for a different model. Numbers for the five trainable policies are on the policies page, benchmark data in the arena. Starting from nothing, read train your first policy, the GR00T N1.7 on SO-100 guide and the data collection guide before recording anything you intend to keep.