Occlusion: What a Robot Policy Does When It Cannot See

Occlusion is not one problem

Every demo of a vision-language-action model picking up a cube hits the same moment: the gripper closes over the object and the object stops being visible. On a tabletop arm that is not an edge case, it is the geometry of grasping. The end effector has to get between the camera and the target in order to touch it. With one webcam on a tripod, the last few centimetres of every approach happen behind the arm.

The interesting question is not whether occlusion happens but what a trained policy does during the seconds it cannot see. It keeps going. It does not slow down or report low confidence. It produces a smooth trajectory out of the parts of the observation that are still informative, and on an SO-100 that is almost always the joint angles it already knows.

LIBERO-Occ, published in June 2026, formalises three of these. It extends the LIBERO suite with what the authors call scene-induced occlusion, placing physical occluders in the 3D scene rather than masking pixels, and sorts 2000 occluded task instances into manipulated-object occlusion (900), receptacle occlusion (750) and dual occlusion (350). Severity is the fraction of the target rendered invisible, (A_full minus A_visible) over A_full, split by per-suite quartiles into light, medium and heavy. Worth keeping in mind when you review your own recordings, because receptacle occlusion is the one people forget: the bin disappears more often than the cube.

What the policy does when the object disappears

Behaviour cloning fits a map from observation to action, and nothing in that procedure knows which part of the observation caused the human's action. De Haan, Jayaraman and Levine called the result causal confusion in 2019 and stated the counter-intuitive part plainly: access to more information can yield worse performance. The model latches onto whichever input predicts the label best on the training set, and in a teleoperated LeRobot dataset that is rarely the image. It is the previous action, which is almost perfectly encoded in the current joint positions.

Wen and colleagues named the mechanism in 2020: the copycat problem. When expert actions are strongly correlated over time, and at 30 Hz on a leader-follower rig they are nearly identical frame to frame, the imitator learns to predict the expert's previous action instead of the next one. That shortcut exists in every imitation learning dataset recorded by teleoperation, and it is exactly the shortcut that survives when the view goes dark.

• Proprioceptive extrapolation. The trajectory implied by the current joint configuration continues. A smooth reach to the wrong place.
• Collapse to the training prior. It goes to the average object position in the dataset. On a dataset recorded with the cube in one spot, indistinguishable from working.
• Gripper timing failure. Open and close are the action dimensions most tightly coupled to what the wrist sees, so they go early or late. See gripper does not close.
• Stall, the rarer case. Some checkpoints emit near-zero deltas far out of distribution, which reads as policy freezes mid-motion.

What the published ablations measured

RoboMimic, 2021: remove the wrist camera and watch

Mandlekar and colleagues ran six offline learning algorithms across five simulated and three real tasks. Their observation-space ablation is still the cleanest number on this: not including wrist camera observations can reduce performance by 10 to 45 percent. On the real Can task the effect is easy to state.

ICLR 2022: the wrist view generalises, the room view memorises

Hsu, Kim, Rafailov, Wu and Finn asked the sharper question: which view survives a change in the world. Their real setup was a Franka Emika Panda grasping a sponge among distractors, trained by behaviour cloning on 360 teleoperated demonstrations. Both the hand-centric and the third-person policy reached 85 percent on the training distribution. Then they changed the table height, the distractors and the table texture.

The caveat is theirs and it matters: this holds only when hand-centric observability is sufficient. If the wrist camera cannot see the target at the start of the episode, the third-person view is not optional. Their fix was to keep both and regularise the third-person stream with a variational information bottleneck, cutting aggregate out-of-distribution failure by 64 percent relative to using both views naively. You will not implement that on a Tuesday evening, but the framing is the useful part: the room camera is where overfitting enters, the wrist camera is where robustness comes from.

LIBERO-Occ, 2026: what current VLAs do under occlusion

The most direct measurement is the newest. LIBERO-Occ evaluated five models on the original suites and on the occluded variants, with the standard third-person view as the primary observation. Simulation numbers, so treat them as a direction and a magnitude.

Why the wrist view helps, and where it does not

A wrist camera does not remove occlusion. The fingers still block the object at contact. What it changes is framing: the target appears in a coordinate frame that moves with the gripper, so the mapping from pixels to the correction the arm should make is close to invariant, and the background, the part of the image that differs between your kitchen table and your desk, mostly falls out of frame.

Two views is the setup everyone converged on

Nobody arrived here by theory. The stacks that work all landed on roughly the same shape, one view of the scene and one from the hand, and the differences are mostly budget.

{
    "video": {
        "front": {
            "original_key": "observation.images.front"
        },
        "wrist": {
            "original_key": "observation.images.wrist"
        }
    }
}

The matching modality config says what each one is for, and the inline comment is the clearest one-line statement of the idea. Note the second detail: the action horizon is 16 steps, and the open-loop evaluation in the same example runs an execution horizon of 16. That is a very different posture from ACT here, which defaults to chunkSize 100 and nActionSteps 100.

so100_config = {
    # Video: current frame only; keys must match "video" entries in meta/modality.json
    "video": ModalityConfig(
        delta_indices=[0],
        modality_keys=["front", "wrist"],  # front third-person view + wrist egocentric
    ),
    ...
    "action": ModalityConfig(
        delta_indices=list(range(0, 16)),  # predict 16 future steps
        ...
    ),
}

Zhao and colleagues define chunking precisely in the ACT paper: every k steps the agent receives an observation, generates the next k actions, and executes them in sequence. Their ablation shows why you want k large, success climbing from 1 percent at k=1 to 44 percent at k=100, and why not too large, with a dip at k=200 and k=400 they attribute to the lack of reactive behaviour. Read that through the occlusion lens: action chunking means the arm is deliberately blind for the length of a chunk. At 30 frames per second, 100 actions is over three seconds of committed motion.

Recording an SO-100 dataset that survives occlusion

The observation space is decided at record time and frozen there. You cannot add a wrist camera at training time because the frames do not exist, and you cannot drop a camera at inference time because the policy was fitted to specific keys. Decide before episode one. Below is the manual path with the LeRobot CLI; the desktop client records the same format if you would rather not manage ports and indices by hand.

1. 1
   Find your cameras and write the indices down

   LeRobot ships auto-discovery. Run it with both cameras plugged in, in the order you intend to keep.

   bash

   lerobot-find-cameras opencv
   # or: lerobot-find-cameras realsense

2. 2
   Mount the wrist camera so it sees the fingers

   The useful framing includes both jaws and roughly 10 cm in front of them. If the fingertips are out of frame, the view cannot supply grasp timing, which is the main reason you added it. Aim the scene camera so the workspace and the receptacle stay visible at full arm extension.

3. 3
   Run the human test

   The LeRobot recording guide states the rule that matters: you should be able to do the task yourself by only looking at the camera images. Open both streams, cover your view of the real table, and talk through the grasp. If you cannot tell when to close the gripper, neither can the policy.

4. 4
   Record with both cameras declared

   Camera names become dataset keys, so pick them once. front and wrist match NVIDIA's SO-100 example, which saves a conversion later.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower \
     --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader \
     --display_data=true \
     --dataset.repo_id=${HF_USER}/occlusion-pick \
     --dataset.num_episodes=50 \
     --dataset.single_task="Pick the red cube out of the bin"

5. 5
   Vary what gets occluded, not just where the object sits

   The LeRobot guide suggests at least 50 episodes with 10 per location for a basic pick and place. For an occlusion-heavy task, spend that budget on approach angles that put the arm between camera and target in different ways, and on starts where the object begins partly behind something.

6. 6
   Read the cadence report before you train

   lerobot-record prints what the loop achieved. Two streams usually push the observe step to the largest share of the loop, and a session that ran at 25 Hz still produces a dataset that claims 30, because timestamps come from the frame index. That mislabelled speed is a training problem you inherit for free.

Two ways to get there

# NVIDIA's own SO-100 closed-loop eval client, two cameras
WRIST_CAM_IDX=2
FRONT_CAM_IDX=6

uv run --no-sync python eval_so100.py \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM2 \
  --robot.id=orange_follower \
  --robot.cameras="{ front: {type: opencv, index_or_path: $FRONT_CAM_IDX, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: $WRIST_CAM_IDX, width: 640, height: 480, fps: 30}}"

The blackout test: which view is actually driving the policy

Before concluding anything about occlusion, find out whether your checkpoint uses the images at all. This is offline. Nothing moves, and it takes about a minute per episode. Replay a recorded episode through the policy twice: once intact, once with one camera stream replaced by zeros. Same frames, same state, one difference.

import numpy as np
import torch

# policy is a LeRobot PreTrainedPolicy; episode yields frames from a
# recorded LeRobotDataset. Nothing here touches the robot.
deltas = {"front": [], "wrist": []}

for step in episode:
    batch = {
        "observation.images.front": step["front"],
        "observation.images.wrist": step["wrist"],
        "observation.state": step["state"],
    }
    a_full = policy.select_action(dict(batch))

    for view in ("front", "wrist"):
        key = f"observation.images.{view}"
        blinded = dict(batch)
        blinded[key] = torch.zeros_like(batch[key])
        a_blind = policy.select_action(blinded)
        deltas[view].append((a_full - a_blind).abs().mean().item())

for view, d in deltas.items():
    print(view, "mean |delta action| =", np.mean(d))

Read it like this. A mean absolute action delta near zero for a view means that view is not driving the policy, and better camera placement will not help until the training data gives it a reason to. A large delta on wrist concentrated in the gripper dimension is the healthy pattern for a pick task. A large delta on front and near-zero on wrist usually means the wrist camera was mounted too close or too far and never showed the fingers. Rerun after every fine-tuning run; it is the cheapest regression check in the stack.

What the second camera costs at inference time

More views means more vision tokens, and vision tokens are most of the forward pass in a VLA. The published numbers are not always measured the way you will run the model: the GR00T N1.7 model card's timing table is explicitly labelled as 4 denoising steps and 1 camera. Your two-camera deployment is not that configuration. Here is what this platform states per action step, which is the number to plan the control loop around.

What none of this fixes

A wrist camera and a well-recorded dataset move the failure rate. They do not remove the problem, and four things stay out of reach.

• Total occlusion is still total. Fully inside a closed gripper or fully behind a wall, no camera arrangement recovers it. The best you get is a correct prior about where it was, which is a data problem.
• Nothing here gives the policy memory. The GR00T SO-100 example uses delta_indices=[0] for video, the current frame only. A single-frame policy cannot remember that the cube was there two seconds ago.
• The platform cannot repair an observation space you already froze. Recording without a wrist camera and then wanting one means recording again. There is no upgrade path in the dataset format.
• Simulation results do not transfer as numbers. The LIBERO-Occ drops are a shape, not a forecast for your table and your lighting.

For the wider data argument, how to collect high-quality VLA training data covers what else goes wrong in recording, and the DROID walkthrough is the closest thing to a reference rig at scale. For hardware, the complete SO-100 setup guide covers mounting and calibration, the SO-100 data collection page is the short version, and every named failure mode lives on the fix index.

Frequently asked questions