Depth Cameras vs RGB for Manipulation: Do You Need Depth?

The short answer, and why it keeps surprising people

Depth is almost certainly not what is holding your policy back. Every model you can fine-tune on AY-Robots takes the same three things: RGB frames, a proprioceptive state vector, a string of text. Bolt a depth camera onto your SO-100 tomorrow and none of them will read the stream. The tensor has nowhere to go.

That is an argument about sequencing, not against depth. The question is what an RGB-only policy already knows about your geometry, where that runs out, and what would have to change before a depth channel earns its calibration work. What follows reads the observation spaces out of the model cards and repositories, and dates each one.

What an RGB-only policy actually learns about geometry

RGB policies are not geometry-blind, they are metric-blind, which is much less serious. A policy trained by imitation learning never answers "how far away is that cube", only "given this image and these joint angles, what did the human do next". Occlusion, shading, relative size and the gripper's apparent motion carry that, and parallax under the robot's own motion is a depth signal, free on every episode. The schema below is the whole surface a GR00T dataset exposes:

// Isaac-GR00T, getting_started/data_preparation.md, meta/modality.json schema
{
    "state": {
        "<state_key>": {
            "start": <int>,         // Starting index in the state array
            "end": <int>            // Ending index in the state array
        }
    },
    "action": {
        "<action_key>": {
            "start": <int>,
            "end": <int>
        }
    },
    "video": {
        "<new_key>": {
            "original_key": "<original_video_key>"
        }
    },
    "annotation": {
        "<annotation_key>": {}
    }
}

The wrist camera does most of the geometry work

On an SO-100 the wrist camera sits centimetres from the object at grasp time. Parallax is large there, occlusion boundaries are crisp, and the gripper fingers are in frame as a known-size ruler next to the end effector. That two-view layout is what published SO-100 work assumes: SmolVLA uses top and wrist cameras for its SO-100 tasks, and normalises community datasets onto a top, wrist, side ordering because inconsistent naming hurt pretraining.

• A wrist camera buys relative depth near the gripper, contact timing, and a strong signal for when to close the fingers.
• It does not buy metric distance outside its frame, the scale of a novel object, or usable geometry on glass and chrome.
• Moving it breaks things. Even DP3, with real 3D input, manually transforms its point clouds and adjusts the crop box when the view changes, and warns that significant changes might be hard to handle. A 2D policy has no such repair step.

That is the honest weakness of RGB geometry: a policy that works at your desk and dies when you nudge the tripod, which has its own page, policy only works in one setup. Camera placement is decided at recording time and expensive to change later, covered at length in collecting high-quality VLA training data.

Where depth genuinely pays off, according to published ablations

There is real evidence that 3D input helps, plus a pattern the headline numbers hide: raw depth is usually the worst 3D option, and gains concentrate in generalization rather than trained tasks. Four results, each read from the paper:

Read the DP3 row again before you buy anything. Raw depth and RGB-D both scored below plain RGB; only as a cropped, downsampled point cloud did it reach 78.3. The authors are plain: RGB-D and depth images are "not comparable to point clouds, indicating that the proper usage of depth information is essential". Dropping only the crop step costs 33 points. Both sensorless entries are on the model arena: MolmoAct and SpatialVLA.

The camera you would actually buy, and the number that decides it

If you do add depth, the spec that decides everything on a small arm is minimum depth distance, min-Z: the closest range at which the stereo pair can still triangulate. Below it you get holes, not numbers. On most models a wrist camera spends the interesting part of every episode inside that range, which is how people end up owning an expensive RGB camera. Hardware walkthrough in the SO-100 setup guide.

Recording depth with LeRobot today, and what still does not work

Depth recording landed through PR #3644, merged 27 June 2026 and shipped in LeRobot v0.6.0 on 6 July 2026. In 0.5.1 and earlier there is no depth module at all: src/lerobot/datasets/depth_utils.py returns 404 on the v0.5.1 tag and 200 on v0.6.0. That matters because the Pi0.5 trainer here runs lerobot 0.5.1, whose dataset path predates depth entirely.

In v0.6.x the SO follower checks each camera for a use_depth attribute. When set, the robot advertises an extra feature named after that camera, wrist_depth for a camera called wrist, of shape (height, width, 1), filled from read_latest_depth(). The writer quantizes those maps to 12-bit codes as HEVC Main 12 in gray12le, logarithmically by default (depth_min 0.01 m, depth_max 10.0 m, shift 3.5 m) so more quanta land near the camera. Lossless TIFF is available for raw maps. Format details in the dataset docs.

1. 1
   Install LeRobot with the RealSense extra

   pyrealsense2 sits behind one extra, the recording CLI behind another. You need both, and Python 3.12 or newer as of v0.6.1.

   bash

   pip install 'lerobot[intelrealsense,core_scripts]'
   python -c "import lerobot; print(lerobot.__version__)"
   # expect 0.6.0 or newer; depth does not exist below that

2. 2
   Find the camera and note its serial number

   RealSense devices also enumerate as generic OpenCV cameras, which is how people record only the colour stream and wonder where the depth went. The upstream example runs the recorder under sudo: access is a permissions problem first.

   bash

   lerobot-find-cameras realsense

3. 3
   Record with use_depth enabled

   use_depth defaults to False, and the config rejects a camera with neither use_rgb nor use_depth. Depth and RGB encoders are configured separately, which is why v0.6.0 renamed --dataset.vcodec to --dataset.rgb_encoder.vcodec.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_so100 \
     --robot.cameras='{
       wrist: {type: intelrealsense, serial_number_or_name: "0123456789",
               width: 640, height: 480, fps: 30, use_depth: true},
       front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}
     }' \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader \
     --dataset.repo_id=${HF_USER}/so100-depth-pick \
     --dataset.num_episodes=50 \
     --dataset.single_task="Pick up the cube and drop it in the bowl" \
     --dataset.depth_encoder.vcodec=hevc \
     --dataset.depth_encoder.depth_min=0.05 \
     --dataset.depth_encoder.depth_max=1.0 \
     --dataset.streaming_encoding=true

4. 4
   Verify the depth feature actually landed

   Depth is flagged in the feature metadata. If the flag is missing, the stream went out as a plain single-channel video and the quantization never applied.

   python

   import json, os
   from pathlib import Path
   
   repo_id = "your-user/so100-depth-pick"
   root = Path(os.environ.get(
       "HF_LEROBOT_HOME",
       Path.home() / ".cache" / "huggingface" / "lerobot",
   )) / repo_id
   
   info = json.loads((root / "meta" / "info.json").read_text())
   for key, ft in info["features"].items():
       is_depth = ft.get("info", {}).get("is_depth_map", False)
       print(f"{key:44s} {str(ft.get('dtype')):8s} {ft.get('shape')}  depth={is_depth}")

5. 5
   Decide what reads it

   Here the manual path stops being a config change. No shipped policy reads that feature, so step five is writing code.

Do it yourself, or settle it empirically

# start from the repo that has a depth ablation to reproduce
git clone https://github.com/YanjieZe/3D-Diffusion-Policy

# or extend the LeRobot trainer you already run
git clone https://github.com/huggingface/lerobot
cd lerobot && pip install -e '.[intelrealsense,core_scripts,training]'

If you do not own a depth camera, predict one

The notable development of the last two years is that the strongest 3D-aware vision-language-action models mostly do not use depth sensors. They estimate depth from RGB and use the estimate as structure. MolmoAct predicts 100 depth tokens per frame from RGB alone before any trajectory or action. SpatialVLA runs ZoeDepth over the frame and back-projects into an egocentric 3D position encoding, keeping that model frozen at 8.6% of parameters and 0.06 s per action. The broader arc is in how VLA models are developing.

• Neither needs a depth camera at inference, only a depth model at training time, on the GPU you already rented.
• Predicted depth is metrically wrong and geometrically plausible. SpatialVLA argue that captures relative layout and makes precise scale unnecessary: 72.7 with ZoeDepth against 70.5 with the sensor.
• It sidesteps the failure that kills stereo: a monocular model produces a confident surface for a glass jar, where a stereo pair produces a hole.
• It costs tokens. In MolmoAct the depth tokens sit ahead of everything else in the autoregressive path, so every action waits on them.

# Depth Anything V2, README as of 24 Aug 2026
git clone https://github.com/DepthAnything/Depth-Anything-V2
cd Depth-Anything-V2
pip install -r requirements.txt
# then place depth_anything_v2_vits.pth (Apache-2.0) under checkpoints/

A decision rule that survives contact with a real task

People underrate this: every extra stream is extra tokens and milliseconds per action step, and depth is not free. Here, inference latency runs from 20 ms per action step for ACT to 485 ms for Pi0.5, the difference between a smooth policy and a hesitant one; GR00T N1.7 against Pi0.5 lays those out. Experiments are cheap next to hardware, and the pricing page carries the full figures behind this table.