Camera Placement for Robot Manipulation Data

Two USB webcams on a bench set the ceiling on what a manipulation policy can learn. One watches the scene, the other rides on the gripper. A camera that cannot resolve the gap between the fingers and the object cannot teach a policy when to close them, no matter how many episodes you record.

This page covers the physical question when recording a LeRobot dataset: overhead, front, wrist, how many, and what changes in the trained model when one is missing. Commands are from lerobot 0.6.1, published to PyPI on 3 August 2026. Every number below is attributed to the paper, model card or repository it was read from.

What each viewpoint can and cannot resolve

A camera is a projection, and each projection destroys an axis. Ask which axis the task needs at the moment it is hardest, usually when the end effector touches something, then make sure one camera keeps it.

That is why the standard answer is not one camera. A single third-person view encodes coarse layout and fine contact in the same pixels and is weakest at the fine part, because the arm occludes the object exactly when precision matters. A single wrist view is the opposite: good at contact, useless at where do I go next.

What the published rigs actually use

Worth looking at what real imitation learning systems shipped. Every row was read from the paper, model card or repository in the sources.

The pattern across all six: at least one third-person view, and a wrist view wherever there is a grasp. The exception is instructive. DROID records a wrist camera, but the diffusion policy baseline in the paper defines observation keys for the two exterior cameras, the gripper position and the gripper state only, each image downsampled to 128 x 128. The wrist stream is there for whoever wants it; the paper's own baseline did not use it.

{
  "state":  { "single_arm": {"start": 0, "end": 5}, "gripper": {"start": 5, "end": 6} },
  "action": { "single_arm": {"start": 0, "end": 5}, "gripper": {"start": 5, "end": 6} },
  "video": {
    "front": { "original_key": "observation.images.front" },
    "wrist": { "original_key": "observation.images.wrist" }
  },
  "annotation": {
    "human.task_description": { "original_key": "task_index" }
  }
}

The wrist camera, and the failure you get without it

People skip this one because mounting a webcam on a moving link is fiddly and the cable has to survive rotation. Three independent measurements say it is the highest-value camera on the arm.

Read the Hsu row carefully. In distribution the two rigs were indistinguishable at 85 percent. The gap only opened once the table height, the distractor objects or the table texture changed. A third-person policy that scores well on your own bench is not evidence of anything until you move something.

The failure signature is recognisable. The arm travels to roughly the right place, the trajectory looks smooth, then the grasp is wrong: the gripper closes 2 cm early, or on air, or catches an edge and lifts nothing. From a fixed camera 60 cm away the last 2 cm of approach are a handful of pixels, and the gripper is in front of them. With no signal there the network falls back on proprioception and produces the same closing motion every time.

Overhead or front for the third-person view

Given a wrist camera, the remaining question is where the scene camera goes. There is no universal answer, but there is a defensible rule, and two of the sources state it almost identically.

• The LeRobot recording guide states it as a test: "A good rule of thumb is you should be able to do the task yourself by only looking at the camera images." If you cannot teleoperate from the streams alone, the policy cannot either.
• The robomimic study used the third-person view that matched what the operator saw during collection. Same rule from the other side.
• Overhead wins when the task is about which object and where it goes: sorting, bin placement, picking from a cluttered tray. The arm is thin from above, so it occludes less.
• Front or side wins when height must be visible: stacking, inserting, shelving, pouring. Height is the axis an overhead camera cannot recover.
• ALOHA kept both and rotated the front camera 90 degrees for more vertical coverage out of a 480 x 640 sensor. lerobot exposes that as the rotation field on OpenCVCameraConfig.
• The Hugging Face guidance is blunt about what must not be in frame: the leader arm should not appear, and the only moving objects should be the follower arm and the manipulated items.

Wiring two cameras into a recording session

The manual path end to end on an SO-100 or SO-101. It assumes the arm is calibrated; if not, do that first, because calibration and framing interact. A differently calibrated arm sits at a different place in frame.

1. 1
   Find the camera identifiers

   lerobot auto-discovers OpenCV and RealSense devices. The docs warn that these identifiers may change after rebooting or re-plugging, which is the most common cause of a silently ruined dataset.

   bash

   lerobot-find-cameras opencv
   # or, for Intel RealSense:
   lerobot-find-cameras realsense

2. 2
   Name the keys by location, not by device

   The Hugging Face dataset guidance asks for <modality>.<location>: images.top, images.front, images.left, images.right, and images.wrist.left / .right / .top / .bottom for wrist mounts. It explicitly asks you to avoid images.laptop and images.phone. The key becomes the dataset feature name and follows the data into the trainer.

   text

   good:  front, wrist, top, side
   bad:   laptop, phone, cam0, usb2

3. 3
   Record with both streams

   Cameras are passed as one inline dict on --robot.cameras, each entry mapping a key to a device config. Mixing camera types in one rig is supported.

   bash

   lerobot-record \
     --robot.type=so101_follower \
     --robot.port=/dev/tty.usbmodem585A0076841 \
     --robot.id=my_follower \
     --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
     --teleop.type=so101_leader \
     --teleop.port=/dev/tty.usbmodem58760431551 \
     --teleop.id=my_leader \
     --display_data=true \
     --dataset.repo_id=${HF_USER}/pick-place-two-cams \
     --dataset.num_episodes=50 \
     --dataset.single_task="Pick the red cube and put it in the box"

4. 4
   Watch the streams while you record

   --display_data=true shows the feeds live. Use the first episode as a check: is the object in frame for the whole approach, is the wrist camera looking past the fingers rather than at them, is the leader out of shot, does the exposure hunt when the arm casts a shadow.

   text

   DatasetRecordConfig defaults you accept unless you override them:
     --dataset.fps=30
     --dataset.episode_time_s=60
     --dataset.reset_time_s=60
     --dataset.num_episodes=50

5. 5
   Verify the schema before you spend GPU money

   Read meta/info.json and confirm both video features exist with the shape you expect. A camera that failed to open does not always stop the run; it can leave a one-camera dataset you notice only after training.

   bash

   python -c "import json; d=json.load(open('meta/info.json')); \
   [print(k, v['dtype'], v['shape']) for k,v in d['features'].items()]"

6. 6
   If the target is GR00T, convert and map the keys

   GR00T needs a LeRobot v2.1 dataset plus a meta/modality.json naming the video keys. Both the downgrade script and the SO-100 modality file live in NVIDIA's Isaac-GR00T repository, not in lerobot. Config, modality.json and recorded keys must use the same strings.

   bash

   # from your Isaac-GR00T checkout
   uv run --project scripts/lerobot_conversion \
     python scripts/lerobot_conversion/convert_v3_to_v2.py \
     --repo-id ${HF_USER}/pick-place-two-cams \
     --root examples/SO100/pick_place_lerobot
   
   cp examples/SO100/modality.json \
      examples/SO100/pick_place_lerobot/${HF_USER}/pick-place-two-cams/meta/modality.json

Two routes to the same two-camera dataset

pip install lerobot
lerobot-find-cameras opencv
lerobot-calibrate --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.id=my_follower
lerobot-record --robot.cameras="{ front: {...}, wrist: {...} }" ...

# lerobot 0.6.1 writes LeRobot v3.0; GR00T wants v2.1. The downgrade script ships
# with Isaac-GR00T, not with lerobot:
uv run --project scripts/lerobot_conversion \
  python scripts/lerobot_conversion/convert_v3_to_v2.py --repo-id <your-repo-id>

Camera keys are part of the model contract

Once the data exists the camera names stop being labels and become an interface. Each vision-language-action model handles multiple views differently, and each has a rule you can violate silently. GR00T N1.7 and SmolVLA are where the mistake is easiest to make.

# lerobot/policies/smolvla/configuration_smolvla.py
# Add empty images. Used by smolvla_aloha_sim which adds the empty
# left and right wrist cameras in addition to the top camera.
empty_cameras: int = 0

def validate_features(self) -> None:
    for i in range(self.empty_cameras):
        key = f"{OBS_IMAGES}.empty_camera_{i}"
        empty_camera = PolicyFeature(type=FeatureType.VISUAL, shape=(3, 480, 640))
        self.input_features[key] = empty_camera

Understand that padding before relying on it. It makes a one-camera rig loadable by a checkpoint trained on three. It adds no information. If the missing view is the wrist view, the policy expects contact information and receives a black rectangle.

The same discipline applies between sessions. If the front camera sits 5 cm to the left on day two, that is a new viewpoint as far as the network is concerned. RoVi-Aug, a CoRL 2024 paper on synthesising extra viewpoints, states the problem: imbalances in the distribution of robot types and camera angles in many datasets make policies prone to overfit. The policy only works in one setup page covers the symptom.

What a second camera costs

Views are not free. The cost lands in the recording loop, the training run and the control loop. The last one surprises people.

Fixed rig, or move the cameras on purpose

The LeRobot guide says to keep the cameras fixed, and for a first dataset that is right: hold everything constant except what you want the policy to generalise over, which early on is object position. The same guide then says that once grasping is reliable you can introduce variations, including altering camera positions, and warns against adding too much too quickly.

DROID is the far end of that spectrum. Its exterior cameras sit on adjustable tripods repositioned for every new scene, the collection GUI periodically prompts the operator to perform sampled scene augmentations including moving and re-calibrating the third-person cameras, and the released dataset covers 1417 camera viewpoints across 564 scenes. The protocol was built to prevent two mistakes worth a sticky note: "camera cannot see robot" and "teleoperator in camera view".

Where this platform helps, and where it does not

Camera placement is a physical problem on your bench and no web service solves it. The desktop client records LeRobot-format datasets from a teleop session, the directory lists public datasets to compare against, and the training form removes the format and GPU plumbing between recording and a fine-tuned checkpoint. None of that tells you your overhead camera cannot see height.

The second honest limit is where inference runs. The control loop is 20 ms per action step for ACT up to 485 ms for Pi0.5, and a cloud pod adds public-internet round trips on top. For slow pick-and-place that is workable; for fast reactive motion it is not. Action chunking helps by executing a chunk open loop between queries, and the three ways to start page lists what each GPU tier costs and what inference latency you are buying.

Resolution, frame rate and the things that ruin frames

Placement is the big decision, but a well-placed camera can still produce unusable data. Three settings are worth deciding deliberately, and the recording walkthrough assumes you already have.

A placement checklist you can run in five minutes

1. Try the task yourself watching only the camera streams. If you cannot, re-aim.
2. Move the arm to the exact moment of grasp. Are the fingers and the object visible together in at least one view?
3. Move through the full trajectory and watch for the object leaving frame in the third-person view. Occlusion by the arm at the critical moment is the normal failure.
4. Check that the leader arm, your hands and your torso are out of every frame.
5. Cast a shadow over the workspace with your hand and watch whether the exposure jumps.
6. Wiggle each USB cable at the wrist through the joint's full rotation. A stream that drops when the wrist turns will drop in half your episodes.
7. Record one episode, read meta/info.json, confirm both video features are present with the expected shapes.
8. Reboot, re-run lerobot-find-cameras, confirm the indices did not move before recording the other forty-nine.

None of it is glamorous, and all of it is cheaper than a training run on a dataset with a swapped camera key. The guide on collecting high-quality VLA training data covers diversity and episode counts, the DROID dataset walkthrough goes deeper into viewpoint-diverse collection at scale, and the SO-100 setup guide plus SO-100 data collection cover the arm and the recording side.

Once the rig is settled the decisions are about model and budget, not optics: which of the five trainable policies fits the task, whether you can stay on a 24 GB card, how many episodes you need. The GR00T N1.7 on SO-100 guide and the dataset documentation pick up from there, and the CLI exposes the same operations to a terminal.