Choosing USB Cameras for Robot Data Collection

Resolution: most of it never reaches the model

The instinct is to buy resolution. It is the number on the box, it is cheap, and it matters least, because almost none of it survives the trip into the network. Every policy you can train on an SO-100 resamples its camera input to a fixed grid first.

The ACT paper is explicit: each 480x640x3 image goes through a ResNet18 and comes back as a 15x20x512 feature map, one vector per 32x32 block of pixels. SmolVLA resizes to 512x512 and keeps 64 visual tokens per frame. BridgeData V2 recorded 640x480 and trained most baselines on 128x128.

640x480 is where the reference rigs above converged, and none of them fed the network more than 640 pixels on the long side. What it reliably buys is four times the bus traffic and a rig that breaks on the third camera. Spend the savings on manual controls instead.

The bandwidth ceiling is arithmetic, not bad luck

This is the part that eats a day. Two cameras work. You mount the third, and either it refuses to open or all three drop frames. The cable is fine and the camera works alone. USB 2.0 allocates isochronous bandwidth up front, per endpoint, regardless of what the cameras send.

The uvcvideo FAQ, written by the driver's maintainers, states it plainly. High speed USB carries at most 480 Mb/s including protocol overhead, and the standard caps periodic transfers at 80 percent of that. That 384 Mbit/s is shared by every webcam, microphone, keyboard and mouse on the controller, and the cap applies to allocation, not usage: bandwidth reserved and never used is still gone.

That is width x height x 2 bytes x fps, what YUYV costs uncompressed. It is worse than the table suggests, because cameras over-report. Some always request the single-endpoint maximum of 198.608 Mb/s; two of those ask 397 Mbit/s and blow the cap before a frame moves. The Good Penguin measured one requesting 195 Mbit/s for a stream needing 46.

# Which controller is each camera on?
# Devices under the same root hub share one 384 Mbit/s budget.
lsusb -t

# What formats and frame rates does this camera really offer?
# The fps lists for MJPG and YUYV are usually different.
v4l2-ctl -d /dev/video0 --list-formats-ext

# What is it streaming right now?
v4l2-ctl -d /dev/video0 --get-fmt-video

MJPEG or YUYV: the setting that decides whether the rig works

Most UVC webcams offer the same resolutions twice, as uncompressed YUYV and as MJPEG. YUYV is the default on many systems and the one that will not scale. MJPEG compresses on the camera, before the data crosses the wire, the only place compression helps.

Arducam's documentation for its OV9281 UVC module lists MJPG at 640x480 running 50, 30, 15, 10 and 5 fps, while YUV2 on the same device appears only at 1600x1200, 1280x720 and 800x600, all capped at 5 fps. Same sensor, same cable. The pixel format decides which frame rates exist at all.

from lerobot.cameras.opencv import OpenCVCamera, OpenCVCameraConfig
from lerobot.cameras import ColorMode, Cv2Rotation

config = OpenCVCameraConfig(
    index_or_path="/dev/v4l/by-id/usb-Acme_HD_Webcam_0001-video-index0",
    fps=30,
    width=640,
    height=480,
    fourcc="MJPG",              # compress on the camera, not on the bus
    color_mode=ColorMode.RGB,
    rotation=Cv2Rotation.NO_ROTATION,
    warmup_s=1,
)

with OpenCVCamera(config) as camera:
    frame = camera.read()
    print(frame.shape)          # (480, 640, 3)

First, fourcc must be a four character string, validated in __post_init__. Second, and this is the trap, setting it is best effort: LeRobot calls the OpenCV setter, reads the value back, and if the camera refused, logs 'failed to set fourcc' and 'Continuing with default format'. It does not raise. Your rig keeps running on YUYV and you find out when the third camera will not open. Resolution and frame rate do raise a RuntimeError.

Separately, index_or_path accepts a path, not just an integer, and LeRobot's docs warn that indices can change after a reboot or re-plug. On Linux, udev keeps stable symlinks under /dev/v4l/by-id/ keyed to vendor, product and serial. Passing one instead of 0 removes the 'wrist and front cameras swapped overnight' class of bugs. If they already swapped inside a recorded dataset, the symptom is a policy that only works in one setup.

Frame rate: match the loop, not the box

A camera advertised at 60 fps is no better than one at 30 if you record at 30. What matters is whether it sustains your rate under your lighting with the other cameras attached. ALOHA records at 50 Hz while its appendix notes all four cameras stream 480x640 at 30 fps; BridgeData V2 ran its loop at 5 Hz.

LeRobot measures this for you. Recent versions of lerobot-record pace the loop at the configured dataset fps and print a one-line digest for every episode you keep, plus a summary when recording ends that breaks the loop down by step. In that documented summary the observe step, the camera read, averages 13.65 ms and peaks at 40.20 ms against a 33.3 ms budget, and is 74.4 percent of the loop's work. Pacing headroom near zero means one slow frame pushes you over.

Cadence (episode 0): 29.88 Hz vs 30 Hz target, 600 ticks, 20.0 s measured,
  8/599 ticks over the 33.3 ms budget (work mean 18.4 ms, worst 45.1 ms)
Cadence summary, whole run, 2 episodes, target 30 Hz (33.3 ms budget per tick):
  effective cadence: 29.88 Hz over 40.1 s measured
  ticks over the 33.3 ms work budget: 16/1198 (1.3%), work mean 18.4 ms, worst 45.1 ms
  loop-body steps (share of measured work):
    observe      mean  13.65 ms  worst  40.20 ms   74.4% of work   1200 calls
    process_obs  mean   0.30 ms  worst   0.30 ms    1.6% of work   1200 calls
    teleop       mean   1.80 ms  worst   1.80 ms    9.8% of work   1200 calls
    send         mean   0.90 ms  worst   0.90 ms    4.9% of work   1200 calls
    record       mean   1.70 ms  worst   1.90 ms    9.3% of work   1200 calls
  pacing headroom: 15.1 ms slept per tick on average (max 16.7 ms)

Exposure is the control nobody sets

Exposure time and frame rate are the same budget: a camera cannot expose for 40 ms and also deliver 30 frames per second. The uvcvideo FAQ answers 'why does my webcam only produce 15 fps when it advertises 30' with exactly this. In a dim room, auto exposure raises exposure and lowers the frame rate to pay.

A rig that held 30 Hz in the afternoon can drop to 15 Hz in the evening, no error anywhere, still writing timestamps that claim 30. V4L2 has a control for it: V4L2_CID_EXPOSURE_AUTO_PRIORITY, named 'Exposure, Dynamic Framerate' in the kernel and exposed by v4l2-ctl as exposure_dynamic_framerate. The kernel documentation scopes it: when auto exposure is set to Auto or Aperture Priority, this control decides whether the device may vary the frame rate, and by default it is disabled and the rate must stay constant. Cameras do not always honour that default. Set it to 0 explicitly whenever you leave auto exposure on.

1. 1
   See what the camera exposes

   List the video nodes, then dump every control with its range and current value. Controls marked inactive are held by an auto mode.

   bash

   v4l2-ctl --list-devices
   v4l2-ctl -d /dev/video0 --list-ctrls

2. 2
   Switch off dynamic frame rate first

   It only bites while auto exposure is on: the kernel scopes it to the Auto and Aperture Priority modes. Set it to 0 before anything else, so the camera cannot trade frame rate for exposure without telling you. Harmless to set in manual mode too.

   bash

   v4l2-ctl -d /dev/video0 --set-ctrl=exposure_dynamic_framerate=0

3. 3
   Set manual mode, then the exposure time

   Order matters. In uvcvideo, V4L2_CID_EXPOSURE_ABSOLUTE is a slave of V4L2_CID_EXPOSURE_AUTO with master_manual = V4L2_EXPOSURE_MANUAL, so exposure time stays inactive until auto exposure is manual. Writing it first does nothing. Menu value 1 is Manual Mode, 3 is Aperture Priority Mode. White balance uses the same master/slave mechanism and needs the same two steps, but its manual value is 0: set white_balance_automatic=0 first, then the temperature.

   bash

   v4l2-ctl -d /dev/video0 --set-ctrl=auto_exposure=1
   v4l2-ctl -d /dev/video0 --set-ctrl=exposure_time_absolute=157

4. 4
   Read the values back

   Every one can fail silently. Read them back from the device while it is streaming, because some drivers reset controls when the stream starts.

   bash

   v4l2-ctl -d /dev/video0 --get-ctrl=auto_exposure,exposure_time_absolute,exposure_dynamic_framerate

The unit for exposure_time_absolute is 100 microsecond steps, so 10000 is one second. 157 is roughly 15.7 ms, about as long as you can expose and still clear a 30 fps budget. If that looks dark, add light rather than exposure time. Light costs no frame rate.

The rest of the automatic stack

Autofocus is the automatic with no upside on a fixed rig. A wrist camera sitting close to a gripper hunts every time the arm moves, dropping blurred frames into the dataset as if they were sharp. Focus changes also alter apparent scale, so the same object looks different between episodes. ALOHA fixed focal length on all four cameras. Every control below resets on a re-plug, so keep them in a script beside your recording command.

Rolling shutter, global shutter, and whether you care

Almost every USB webcam has a rolling shutter: the sensor reads row by row, so the top of the frame is captured milliseconds before the bottom, and fast motion makes straight edges lean. NVIDIA's Isaac-GR00T SO-100 configuration feeds the current frame only, delta_indices [0], so no temporal context averages that away. Even so, at SO-100 speeds it is second-order next to motion blur, and the fix for both is the same: shorten the exposure, add light. Global shutter UVC modules exist, such as Arducam's OV9281, but cost more and are usually monochrome. Prefer a manual-focus lens over more pixels. The SO-100 setup guide covers the arm side, and the data collection page covers what a usable session looks like end to end.

Verify the rig in ten minutes, before you record 50 episodes

Every policy here needs 30 to 50 episodes before it is worth training on, so a fault found at episode 48 costs a whole session. This is the cheapest insurance in the pipeline. What makes those episodes worth keeping, beyond the cameras, is covered in the guide to collecting high-quality VLA training data.

1. 1
   Enumerate what is attached

   LeRobot ships a discovery command. It prints identifier, backend and default stream profile per camera, the fastest way to spot one defaulting to 15 fps at 1920x1080.

   bash

   lerobot-find-cameras opencv

2. 2
   Confirm the bus topology

   Cameras under the same root hub share one 384 Mbit/s allocation. If all three sit on one controller and you plan to run uncompressed, fix that first.

   bash

   lsusb -t
   v4l2-ctl -d /dev/video0 --list-formats-ext | grep -A3 MJPG

3. 3
   Open every camera at once, not one at a time

   The whole point: a camera that works alone tells you nothing. Start the recorder with the full camera set at the real resolution and frame rate, and watch for the fourcc warning as it connects.

   bash

   lerobot-record \
       --robot.type=so101_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_follower \
       --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30}}" \
       --teleop.type=so101_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_leader \
       --dataset.repo_id=${HF_USER}/camera-check \
       --dataset.num_episodes=2 \
       --dataset.single_task="Pick up the cube" 

4. 4
   Record two throwaway episodes, then read the cadence report

   Two is enough. Check effective cadence against target, observe as a share of work, and pacing headroom. If headroom is near zero, lower the frame rate or resolution now, then check the kernel log for what the recorder swallowed.

   bash

   dmesg | grep -i uvcvideo | tail -20

5. 5
   Look at the frames, not only the numbers

   With --display_data=true, watch the wrist view while the arm moves fast. Blur bursts mean exposure is too long. Tilted vertical edges are rolling shutter. A brightness step between episodes means auto exposure is live.

   bash

   lerobot-record ... --display_data=true

Two ways to get a working camera rig

# 1. install the tools
pip install lerobot
sudo apt install v4l-utils           # Linux camera controls

# 2. discover cameras and their default profiles
lerobot-find-cameras opencv

# 3. pin the controls, per camera, every session
for D in /dev/video0 /dev/video2; do
  v4l2-ctl -d $D --set-ctrl=exposure_dynamic_framerate=0
  v4l2-ctl -d $D --set-ctrl=auto_exposure=1
  v4l2-ctl -d $D --set-ctrl=exposure_time_absolute=157
  v4l2-ctl -d $D --set-ctrl=white_balance_automatic=0
  v4l2-ctl -d $D --set-ctrl=focus_automatic_continuous=0
done

# 4. record, with MJPG and stable device paths
lerobot-record --robot.type=so101_follower ... --display_data=true

Where this does not help

Three honest limits. First, no platform fixes your USB bus. If three cameras will not fit on one controller, that holds whether you record locally or through a hosted client.

Second, camera choice does not rescue remote inference latency. The control loop here runs 20 to 485 ms per action step depending on the model, and public-internet round trips turn a working policy into a hesitant one. Remote inference is viable for slow pick and place, not fast reactive motion. A better camera does not change that.

• Third: a decision made at recording time can bite at training time. Fine-tuning GR00T needs a LeRobot v2.0 or v2.1 dataset; a v3.0 dataset crashes the GR00T loader and must be converted down to v2.1.
• That is a dataset problem rather than a camera one, but it lands in the same place: a session to redo. The dataset-rejected-v3 page covers the conversion.

Questions people actually ask