Synchronising Multiple Cameras in a LeRobot Dataset

A single camera gives you one timing question: how old is the picture the policy sees. Add a second and you have three streams that never sample the same instant. Wrist view, scene view and the joint positions that become the action label are each fixed by different hardware, and nothing forces them to agree. This article is about how far apart they drift in a LeRobot dataset and what a two-frame skew does to a policy trained on it.

Everything below about upstream behaviour comes from the lerobot source on main at commit 7427f318 (22 August 2026; latest tagged release v0.6.1, 3 August 2026). New to recording? Start with recording your first dataset and come back when you add the second camera.

Three clocks and one row

Recording an episode on an SO-100 during teleoperation means running a loop at a fixed rate, usually 30 Hz, and writing one row per iteration: an image per camera, the follower arm's measured joint positions, and the action, which is the leader arm's position. Each value is fixed at a different moment inside the same 33.3 ms tick.

The ordering matters. The action in row n was sampled later than the images in row n, and the images were captured before the loop asked for them. The dataset stores none of this.

What lerobot-record actually does per tick

The recording loop is a plain sequence with a pacing sleep at the end. Condensed from the upstream script (what it writes is covered in the dataset docs), one iteration looks like this:

# lerobot/scripts/lerobot_record.py, record_loop(), condensed
timer.tick()

with timer.section("observe"):
    obs = robot.get_observation()        # state first, then every camera

with timer.section("teleop"):
    act = teleop.get_action()            # leader arm, read after the images

with timer.section("send"):
    robot.send_action(robot_action_to_send)

with timer.section("record"):
    frame = {**observation_frame, **action_frame, "task": single_task}
    dataset.add_frame(frame)

timer.wait()                             # sleep until this tick's 1/fps deadline

Inside get_observation() the follower does the same thing for every camera you declared, in dictionary order:

# lerobot/robots/so_follower/so_follower.py, condensed:
# the timing logs and the optional depth branch are left out
def get_observation(self) -> RobotObservation:
    obs_dict = self.bus.sync_read("Present_Position", num_retry=self.config.num_read_retries)
    obs_dict = {f"{motor}.pos": val for motor, val in obs_dict.items()}

    for cam_key, cam in self.cameras.items():
        if getattr(cam, "use_rgb", True):
            obs_dict[cam_key] = cam.read_latest()   # non-blocking peek
    return obs_dict

The timestamp column is a grid, not a measurement

This is the part that catches people. When a frame is appended to the episode buffer, its timestamp is not read from any clock:

# lerobot/datasets/dataset_writer.py, DatasetWriter.add_frame()
frame_index = self.episode_buffer["size"]
timestamp = frame_index / self._meta.fps
self.episode_buffer["frame_index"].append(frame_index)
self.episode_buffer["timestamp"].append(timestamp)

Row 90 of an episode recorded at 30 fps is stamped 3.0 s whether that tick took 33 ms or 61 ms. Each camera's MP4 is encoded at the same declared rate, so video and parquet timestamps agree by construction rather than by measurement. Two consequences follow.

• Real skew is invisible. A wrist camera consistently two frames behind the scene camera produces timestamps that are perfectly regular and perfectly wrong.
• A slow loop is recorded as if it had been fast. At 22.4 Hz, 90 rows are still stamped 3.0 s end to end when they really spanned about 4.0 s.

Where the skew comes from

Six mechanisms contribute, and they do not add up in a fixed way. The sizes below are what the mechanism can produce, expressed in frames at 30 fps, where one frame is 33.3 ms.

What a two-frame skew does to a trained policy

A constant skew is a relabelling

Suppose the wrist image is always exactly two frames behind the action in the same row. At 30 fps that is 66.7 ms. The policy does not learn a wrong function, it learns a shifted one: emit the action that followed this scene by 66.7 ms. That is perfectly learnable by imitation learning, and it costs reaction time and nothing else, on one condition: the same offset has to exist when the policy runs.

That condition is what the Universal Manipulation Interface work formalises. UMI measures each observation stream's latency separately, then, in their words, aligns all observations with respect to the stream with the highest latency, usually the camera, using each image's capture timestamp to linearly interpolate the gripper and proprioception streams onto it. Its bimanual cameras are soft-synchronised by nearest-neighbour matching, off by at most 1/60 s. The standard is not zero skew but known and reproduced skew.

A skew that changes every tick is label noise

The peek-based loop does not give you a constant offset. It gives you one redrawn every tick, between zero and one camera frame interval, plus whatever the bus and the decoder add. The same visual scene now appears alongside several different actions, and gradient descent does the only thing it can with contradictory labels: it predicts near their mean. On the arm that reads as a policy which approaches confidently and hedges in the last few centimetres, exactly where a 66.7 ms visual error changes the correct action most.

Nothing in the architecture averages the jitter away. Chunking policies predict from the observation in front of them: ACT emits a chunk of 100 actions with nActionSteps 100 on this platform, with no image history for the jitter to cancel against. The averaging happens in the loss instead. If gripper timing is what fails, the gripper failure page walks the same symptom from the other end.

And the skew at deployment is a different number

Recording peeks inside a 30 Hz loop; running the policy adds inference on top. On this platform the per-action-step cost is 20 ms for ACT, 152 ms for GR00T N1.7, 165 ms for GR00T N1.5, 245 ms for SmolVLA and 485 ms for Pi0.5, all five on the policies page. Action chunking amortises that, since one forward pass yields a chunk. It does not repair training data, and it adds a seam at chunk boundaries, which is what the real-time chunking work addresses and what lerobot-rollout implements behind --inference.type=rtc.

Measure it before you record 50 episodes

None of this needs special hardware. Four steps, about ten minutes, and you will know whether your rig has a one-frame spread or a five-frame spread.

1. 1
   Confirm what the driver actually negotiated

   Auto-discovery reports the default profile, which is often not the one you asked for. On Linux, check the negotiated format and frame interval rather than trusting the config you passed.

   bash

   lerobot-find-cameras opencv
   
   # Linux: what the device really offers, and what it is currently set to
   v4l2-ctl -d /dev/video0 --list-formats-ext
   v4l2-ctl -d /dev/video0 --get-parm

2. 2
   Log how old each frame is at the instant the loop would peek it

   Each camera object exposes latest_timestamp, taken with time.perf_counter() right after post-processing. Against the tick time it gives the frame's age; the difference between cameras is the skew.

   python

   import time
   from lerobot.cameras.opencv import OpenCVCamera, OpenCVCameraConfig
   
   cams = {
       "wrist": OpenCVCamera(OpenCVCameraConfig(index_or_path=0, width=640, height=480, fps=30)),
       "front": OpenCVCamera(OpenCVCameraConfig(index_or_path=2, width=640, height=480, fps=30)),
   }
   for c in cams.values():
       c.connect()
   
   try:
       for _ in range(300):                       # 10 s at 30 Hz
           tick = time.perf_counter()
           ages = {}
           for name, c in cams.items():
               c.read_latest()                    # the same call the record loop makes
               with c.frame_lock:
                   ages[name] = (tick - c.latest_timestamp) * 1e3
           spread = max(ages.values()) - min(ages.values())
           print(" ".join(f"{n}={a:5.1f}ms" for n, a in ages.items()), f"spread={spread:5.1f}ms")
           time.sleep(1 / 30)
   finally:
       for c in cams.values():
           c.disconnect()

3. 3
   Record ten seconds with a hard visual edge

   Point both cameras at the same light switch or the same book being clapped shut, and record one short episode. This measures the whole chain end to end.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower \
     --robot.cameras='{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}' \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader \
     --dataset.repo_id=${HF_USER}/sync-check \
     --dataset.num_episodes=1 \
     --dataset.episode_time_s=10 \
     --dataset.single_task="sync check" \
     --dataset.push_to_hub=false

4. 4
   Find the flash in each video and subtract

   Find the first frame in each MP4 whose mean brightness jumps. The difference in frame indices is the residual skew after exposure, bus, decode and peek phase have had their say.

   python

   import cv2
   
   def first_flash(path, thresh=8.0):
       cap, prev, idx = cv2.VideoCapture(path), None, 0
       try:
           while True:
               ok, frame = cap.read()
               if not ok:
                   return None
               y = float(frame.mean())
               if prev is not None and y - prev > thresh:
                   return idx
               prev, idx = y, idx + 1
       finally:
           cap.release()
   
   f, w = first_flash("front.mp4"), first_flash("wrist.mp4")
   print(f"front={f} wrist={w} skew={None if None in (f, w) else w - f} frames")

The fifth check is free: read what the recorder tells you. lerobot-record keeps a cadence timer and prints a per-episode line plus a full block at the end, broken down by loop-body section. The section named observe covers all of get_observation(), the Feetech state read plus every camera peek, so its mean and worst milliseconds bound what the cameras cost. The block also reports effective cadence, ticks over the work budget, and pacing headroom, where near zero means the loop is saturated. If it cannot hold the rate at all, you get this, verbatim:

Control loop is running slower (22.4 Hz) than the target FPS (30 Hz). Dataset frames
might be dropped and robot control might be unstable. Common causes are:
1) Camera FPS not keeping up 2) Policy inference (action or text) taking too long
3) CPU starvation

Fixes, in the order that pays off

The last row is the one people skip. Placement and timing are separate properties of the same rig, and both are baked into the dataset. More on placement in the SO-100 data collection guide and in the guide to collecting high-quality VLA training data.

Two routes to a synchronised two-camera dataset

# install and check the rig
pip install 'lerobot[core_scripts]'
lerobot-find-cameras opencv

# measure the spread with the age logger from the previous section,
# then record with both cameras declared in one map
lerobot-record \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower \
  --robot.cameras='{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}' \
  --teleop.type=so100_leader \
  --teleop.port=/dev/ttyACM1 \
  --teleop.id=my_leader \
  --dataset.repo_id=${HF_USER}/two-cam-task \
  --dataset.num_episodes=50 \
  --dataset.episode_time_s=30 \
  --dataset.single_task="Pick up the cube and drop it in the bin"

What the careful rigs do

The pattern is consistent: serious rigs either capture a real timestamp per image and align afterwards, or use hardware that fixes the instant for several sensors at once. The low-cost route does neither, a defensible trade for a 110 to 150 EUR arm as long as you know it and keep the rig stable. More in the DROID dataset write-up and the complete SO-100 guide.

Where this does not help, including on this platform

• No amount of training fixes a skewed dataset. A loss curve that falls while the arm does nothing useful is a data problem, and that page exists because it is common.
• AY-Robots adds no hardware synchronisation. The recording path is LeRobot's, with the same peek and the same synthetic timestamps.
• A public dataset carries the same blind spot as yours. Treat downloaded timing as unknown, not as fine.
• Cloud inference makes deployment worse. The control loop is 20 to 485 ms per action step depending on the model, and public-internet round trips on top turn a working policy into a hesitant one. Remote inference suits slow pick-and-place, not fast reactive motion.
• If the policy works on your desk and fails on another, timing is one candidate; the setup-specific page lists the rest.

The honest summary: a two-camera SO-100 rig has a frame-scale timing uncertainty that neither LeRobot nor this platform measures for you. Bound it once, keep the rig frozen, and stop treating inference latency and recording latency as separate topics. They are the same budget, spent at different ends of the pipeline.