The Cost of a Second Camera: Bandwidth, Latency, Money

A second camera on an SO-100 is a cheap decision at the hardware shop and an expensive one everywhere else. What you actually spend is USB isochronous bandwidth on the recording machine, extra vision tokens in every forward pass, milliseconds of inference latency on every control step, and a full re-record of your LeRobot dataset, because you cannot retrofit a view into episodes captured without it.

Below is each bill, priced from upstream code and published ablations. Short version: on a tabletop pick-and-place with one fixed third-person view, a wrist camera is usually the highest-return change available. Where the existing views already cover the workspace it can make things worse, and there is a 2026 SO-101 result where it did.

Bill one: USB bandwidth, and the error that pretends to be a disk problem

A UVC webcam does not negotiate a bitrate. It asks the host for an isochronous endpoint with a fixed payload per microframe, and the host either grants the whole reservation up front or refuses the stream. Two separate ceilings apply, and conflating them is why the arithmetic often looks like it should work. USB 2.0 high speed signals at 480 Mbit/s in 125 microsecond microframes, and the specification caps periodic transfers at 80 percent of each microframe, leaving about 384 Mbit/s for every camera on the bus to share. Separately, one high-bandwidth isochronous endpoint tops out at three 1024-byte transactions per microframe, about 196 Mbit/s, and that is the most a single camera can ever reserve.

The 384 Mbit/s figure is per bus, not per port. A hub adds no bandwidth; it multiplexes devices onto the same upstream reservation. So the number that decides whether your second camera opens is what each stream reserves, not what it sends, and for uncompressed formats the payload at least is exact arithmetic.

1. 1
   Ask what the host already sees

   LeRobot ships auto-discovery. The identifiers it prints are not stable across reboots.

   bash

   lerobot-find-cameras opencv
   # use 'realsense' instead of 'opencv' for Intel RealSense devices

2. 2
   Find which controller each camera hangs off

   Separate root controllers get separate isochronous budgets. Two cameras behind one hub share one.

   bash

   lsusb -t   # look for Driver=uvcvideo, and whether both sit under the same Bus

3. 3
   Check which formats the camera really offers

   No MJPEG in the list means an uncompressed stream, and the arithmetic above applies.

   bash

   v4l2-ctl --list-formats-ext -d /dev/video0

4. 4
   Stream both at the resolution you intend to record

   The test that matters. Two minutes here, or an evening of episodes with a camera that drops out.

   bash

   v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=480,pixelformat=MJPG \
     --stream-mmap --stream-count=300 &
   v4l2-ctl -d /dev/video2 --set-fmt-video=width=640,height=480,pixelformat=MJPG \
     --stream-mmap --stream-count=300

5. 5
   If it still fails, reload uvcvideo with the quirk

   Uncompressed formats only. Close everything holding a camera or the module will not unload.

   bash

   sudo modprobe -r uvcvideo
   sudo modprobe uvcvideo quirks=128   # UVC_QUIRK_FIX_BANDWIDTH = 0x80

Bill two: what each policy does with the second view

There is no single answer: the five trainable policies ingest images in four different ways. The table below is read out of upstream source in August 2026.

# NVIDIA/Isaac-GR00T, examples/SO100/so100_config.py (main branch, 24 Aug 2026)
so100_config = {
    # Video: current frame only; keys must match "video" entries in meta/modality.json
    "video": ModalityConfig(
        delta_indices=[0],
        modality_keys=["front", "wrist"],  # front third-person view + wrist egocentric
    ),
    "state": ModalityConfig(
        delta_indices=[0],
        modality_keys=["single_arm", "gripper"],
    ),
    ...
}

// NVIDIA/Isaac-GR00T, examples/SO100/modality.json
"video": {
    "front": { "original_key": "observation.images.front" },
    "wrist": { "original_key": "observation.images.wrist" }
}

Bill three: latency, paid on every control step

The cleanest published measurement of what an extra view costs at inference time is the OpenVLA-OFT study by Kim, Finn and Liang: throughput and latency for 7-dimensional actions on an NVIDIA A100, averaged over 100 queries, holding the recipe fixed and changing only the inputs.

The last row is the price tag: latency up about 54 percent, throughput down a third, LIBERO-Long success up 3.8 points. Note what is bundled there. That row adds the wrist image and the proprioceptive state together, so the delta is an upper bound on what the camera alone costs, not a camera-only measurement. Per-step figures here span an order of magnitude, from 20 ms for ACT through 152 ms for GR00T N1.7 and 245 ms for SmolVLA to 485 ms for Pi0.5. Half again on 20 ms is invisible. Half again on 485 ms is a different robot.

This is also where action chunking earns its keep. ACT predicts 100 steps per forward pass by default, Pi0 uses a horizon of 50, GR00T N1 uses 16. A long chunk amortises the second camera's encoder cost over many actuator commands.

Bill four: time and money, smaller than people expect

This bill surprises people in the good direction. A second camera changes the token count, but not the GPU tier, and the tier is what money is attached to.

The real cost is recording, paid in arm time. LeRobot's episode defaults are 50 episodes, 60 seconds of recording and 60 of reset: 100 minutes at the leader arm and 90,000 frames per camera. A second camera adds none of that, provided it was mounted before you started. If it was not, you pay those 100 minutes again. That asymmetry is the whole argument for settling the layout first.

What the second view buys, according to people who measured

The literature agrees on one thing and splits on another. Agreed: a wrist view helps most where a third-person view is weakest, in the last few centimetres before contact. Split: whether it helps once your existing views already cover that region.

Where to put the second camera

Placement decides most of the payoff. The points below come from the Hugging Face dataset guidance and the LeRobot imitation-learning tutorial, and every one of them is something people get wrong on a first imitation learning run.

1. Two views is the recommended default. More is more tokens, more bandwidth, more ways to disagree.
2. Record at 480x640 or better, around 30 fps. Below that the wrist view stops resolving what it is for.
3. Keep both cameras mechanically fixed. One nudged between sessions trains a policy that only works in one setup.
4. Keep the leader arm out of frame. The policy will happily learn to watch the human instead of the scene.
5. The only moving things in frame should be the follower arm and the objects it moves.
6. The LeRobot rule of thumb: you should be able to do the task yourself by only looking at the camera images. If you cannot, neither can the policy.

Recording a two-camera dataset that will train

The manual path end to end, against upstream LeRobot 0.6.2. Substitute your own ports and device paths. The platform walkthrough is at record your first dataset.

1. 1
   Teleoperate with both streams visible first

   Ten minutes here saves discovering after 50 episodes that the wrist camera is upside down.

   bash

   lerobot-teleoperate \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower_arm \
     --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30} }" \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader_arm \
     --display_data=true

2. 2
   Record the episodes

   The camera keys here become the dataset feature names. num_episodes defaults to 50, episode_time_s and reset_time_s to 60 s each.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower_arm \
     --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30} }" \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader_arm \
     --dataset.repo_id=${HF_USER}/so100_twocam \
     --dataset.num_episodes=50 \
     --dataset.single_task="Pick up the cube and drop it in the bin" \
     --dataset.streaming_encoding=true \
     --dataset.encoder_threads=2 \
     --display_data=true

3. 3
   Check that both streams landed

   Encoder settings land in meta/info.json from the first episode. You want two video features with matching fps and resolution.

   bash

   python - <<'PY'
   import json, os, pathlib
   repo = f"{os.environ['HF_USER']}/so100_twocam"
   root = pathlib.Path.home() / ".cache/huggingface/lerobot" / repo
   info = json.loads((root / "meta/info.json").read_text())
   for key, feat in info["features"].items():
       if feat["dtype"] == "video":
           i = feat["info"]
           print(key, feat["shape"], i["video.codec"], i["video.fps"], "crf", i["video.crf"])
   PY

4. 4
   Train

   ACT adapts to the camera count stored in the dataset, so there is no camera flag. That convenience is the trap: a broken second stream trains quietly and badly.

   bash

   lerobot-train \
     --dataset.repo_id=${HF_USER}/so100_twocam \
     --policy.type=act \
     --output_dir=outputs/train/act_so100_twocam \
     --job_name=act_so100_twocam \
     --policy.device=cuda \
     --policy.repo_id=${HF_USER}/act_so100_twocam

5. 5
   Roll out with the same camera keys

   front and wrist must be spelled exactly as at recording time. The most common way a two-camera setup fails on demo day.

   bash

   lerobot-rollout \
     --strategy.type=base \
     --policy.path=${HF_USER}/act_so100_twocam \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30} }" \
     --task="Pick up the cube and drop it in the bin" \
     --duration=60

Two routes to the same two-camera policy

A decision rule that survives contact

Put the camera where the current view is blind. If the front camera loses the gripper behind the object at the moment of the grasp, a wrist view is worth the bandwidth and the latency. If the existing views already show the contact clearly, the second camera buys tokens and little else.

Unsure? Record two views and train both ways. A fine-tuning run on the 24 GB tier costs 1 to 3 USD, so training ACT once with both views and once with the front view alone is cheaper than a considered opinion. You can drop a camera from a dataset; you cannot add one, and the dataset, not the checkpoint, is the expensive artefact.

• Episode design: collecting high-quality VLA training data.
• Hardware: the SO-100 complete guide.
• The model class: vision-language-action models, plus the glossary entry.
• The two cheap options: ACT against SmolVLA, and the arena, 85 models and 332 benchmark results.
• No arm yet: drive a real one, or browse the dataset directory.