Building a Small Robot Data Collection Team: 2 to 5 Seats

One person and one arm can record a dataset. This page is about what changes when two, three or five people record the same task in parallel: what each seat costs in hardware, how to structure a shift so the episodes stay comparable, what to check before an operator goes home, and how many usable episodes an hour of teleoperation actually produces.

The short version from the published record is uncomfortable. More operators reliably buy you more episodes. They do not reliably buy you a better policy. The one study that shipped both versions of the same dataset found the six-operator version worse than the one-operator version, despite having 50 percent more demonstrations in it. So a team is worth building only if you also build the thing that measures it.

A seat is two arms, one host and two cameras

The unit of planning is the seat, not the operator. A seat is everything one person needs to produce one episode without touching anyone else's equipment: a follower arm, a leader arm, a controller board and USB cable for each, two cameras, a host machine, and a calibration file per arm. Two operators sharing one seat in shifts is a different operation from two seats running at once, and the second is what people usually mean by team.

The other thing that changes at more than one seat is calibration. LeRobot keys the calibration file to the id you pass on the command line, not to the serial port and not to the arm. That is convenient until two seats use the same id, or somebody swaps a follower between benches after a repair. Give every arm a unique id at build time, write it on the arm in marker, and never reuse it.

# find the port, one arm at a time (unplug when prompted)
lerobot-find-port

# write motor ids and baudrate into EEPROM, once per arm
lerobot-setup-motors \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0

# calibrate, and use a seat-unique id
lerobot-calibrate \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=seat2_follower

lerobot-calibrate \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=seat2_leader

The throughput ceiling, and where it actually lands

Start from the recording tool, because it sets the ceiling. In LeRobot 0.6.1 (released 3 August 2026), lerobot-record defaults to --dataset.episode_time_s=60, --dataset.reset_time_s=60 and --dataset.num_episodes=50. At those defaults one episode costs two minutes of wall clock, so one seat tops out at 30 episodes an hour and the default 50-episode run takes 100 minutes.

pip install 'lerobot[core_scripts,feetech]'

lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=seat2_follower \
    --robot.cameras="{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30} }" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=seat2_leader \
    --dataset.repo_id=${HF_USER}/cube2bin-seat2-day03 \
    --dataset.single_task="Pick up the red cube and place it in the bin" \
    --dataset.num_episodes=25 \
    --dataset.episode_time_s=20 \
    --dataset.reset_time_s=15 \
    --dataset.push_to_hub=false \
    --dataset.no_stamp=true

A tabletop pick-and-place on an SO-100 does not need 60 seconds. Set the episode window to 20 s and the reset to 15 s and the cycle drops to 35 s, which puts the ceiling near 100 episodes an hour. That number is arithmetic, not experience. Everything below the ceiling is the interesting part, and the published collections, from RoboTurk to DROID, tell you roughly how much gets taken off it.

Working down from the ceiling with the published discard rate gives a planning number rather than a promise. DROID recorded roughly 92,000 trajectories and counted 76,000 of them, so about 17 percent were marked not successful by the collector at the moment of recording. Take a 35 s cycle, remove 17 percent for re-records, and remove the block boundaries where the scene gets rebuilt, and a short tabletop task lands somewhere around 40 to 60 usable episodes per operator-hour. Measure your own first session before you plan a month on that figure.

What actually changes at two, three and five seats

The jump that surprises people is two to three, not one to two. With two seats you can still hold both benches in your head. With three you cannot, and every difference you have not written down becomes a difference in the data. This is the point where the AgiBot World approach becomes worth copying: before formal collection, they ran preliminary acquisition to validate each task and establish written collection standards, and only then let teleoperators start.

Structuring a shift

A shift is not a block of hours, it is a sequence of blocks with a defined reset between them. Borrow the block size from BridgeData V2, which re-randomised camera poses, objects and workspace position every 50 trajectories. On a small team, 25 is easier to review and still large enough that a block is a meaningful unit.

1. 1
   Pre-flight the seat, every time

   Ports move across reboots and a nudged camera is invisible until you look at the frame. Ten minutes here is cheaper than a discarded block. Confirm the calibration id matches the arm, then teleoperate for a minute with the camera view up.

   bash

   lerobot-teleoperate \
       --robot.type=so101_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=seat2_follower \
       --robot.cameras="{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \
       --teleop.type=so101_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=seat2_leader \
       --display_data=true

2. 2
   Record the reference episode against yesterday's

   One episode, then play back the same episode index from the previous day's dataset and compare frames. If the camera moved or the object start pose drifted, you find out in two minutes instead of at merge time.

   bash

   lerobot-dataset-viz \
       --repo-id ${HF_USER}/cube2bin-seat2-day02 \
       --root ~/.cache/huggingface/lerobot \
       --mode local \
       --episode-index 0

3. 3
   Record a block of 25 with a fixed task string

   The task string is a feature of the dataset, not a comment. Keep it identical across all seats: the LeRobot dataset guidance asks for a specific description of 25 to 50 characters, not task1 or demo2. Different wording per seat splits your task index and silently turns one task into three.

   bash

   --dataset.single_task="Pick up the red cube and place it in the bin"
   --dataset.num_episodes=25

4. 4
   Re-randomise between blocks

   Move the object start region, swap distractors, adjust the workspace position relative to the arm. Do not move the cameras unless the whole team moves them together and you record the change. Camera pose is the variation that most often turns into a policy that only works in one setup.

5. 5
   Review the block before the operator leaves

   Re-recording a bad block on the spot costs 15 minutes. Discovering it a week later costs the block plus everything you concluded from it. This is the step small teams skip and then regret.

   bash

   lerobot-edit-dataset \
       --repo_id ${HF_USER}/cube2bin-seat2-day03 \
       --operation.type info \
       --operation.show_features true

Shift length should follow the task, not the clock. RoboTurk had every participant fill in a NASA-TLX form, and the total workload varied by task from 51.5 for the Laundry Layout task to 76.9 for the open-ended Tower Stacking task, with frustration nearly doubling between the two. An open-ended task with no clear success point exhausts operators far faster than a bounded pick-and-place, and the exhaustion shows up as longer, sloppier episodes before anybody says they are tired.

• Bounded task with a crisp success condition: two-hour blocks with a short break between them.
• Open-ended or long-horizon task: shorten the blocks, and expect the last block of the day to be your worst.
• Rotate operators across seats only if every seat is calibrated and the camera geometry is genuinely identical. Otherwise rotation mixes two variables.
• Have the operator mark success or failure at the moment of recording, the way DROID's GUI did. Nobody reconstructs it from video a week later.
• New operators get one supervised session first. RoboTurk gave its 54 users a 5 minute tutorial plus one supervised hour, then let them collect alone.

Quality control, or why the six-operator dataset lost

The single most useful experiment for anyone building a team is in robomimic. The authors collected each simulated task twice through RoboTurk. The Proficient-Human set is 200 demonstrations from one experienced operator. The Multi-Human set is 300 demonstrations from six operators: two rated better, two okay, two worse, 50 each. Same tasks, same interface, same pipeline.

Policies trained on the 300-demonstration multi-operator sets performed worse than policies trained on the 200-demonstration single-operator sets. The paper attributes it to suboptimal and multimodal data: six people solve a task six ways, and behaviour cloning has to fit all six. When the authors split the multi-operator data by proficiency, plain behaviour cloning showed a gap of roughly 27 and 35 percentage points between the better and worse 100-demonstration subsets on two of the tasks, and adding 100 worse demonstrations to a good 100 made most algorithms decline rather than improve.

You do not need a rating panel to get a proficiency signal. robomimic's own appendix says it plainly: trajectory length is a proxy for quality, because less proficient operators take longer. Their numbers, averaged per demonstration in simulator steps, show what that looks like. Lift, Can and Square are single-arm tasks, where the proficient set is 200 demonstrations from one operator and each MH subset is 100 demonstrations from two. Transport needs two arms, so its proficient set is 300 demonstrations recorded jointly by two proficient operators and its MH subsets are 50-demonstration operator pairings. Read that column as a trend, not as the same comparison.

Read the standard deviations, not just the means. On the Can task the better group averaged 143 steps with a spread of 29; the worse group averaged 304 with a spread of 148. Twice as slow, five times as inconsistent. Both signals are computable from a LeRobot dataset without watching a single frame, which makes them the right first gate for a small team.

import numpy as np
from lerobot.datasets import LeRobotDataset

SEATS = ["seat1", "seat2", "seat3"]
DAY = "day03"

for seat in SEATS:
    ds = LeRobotDataset(f"local/cube2bin-{seat}-{DAY}")
    meta = ds.meta
    lengths = np.array(
        [meta.episodes[i]["length"] for i in range(meta.total_episodes)],
        dtype=float,
    )
    secs = lengths / meta.fps
    print(
        f"{seat}: n={len(secs):3d}  "
        f"mean={secs.mean():5.1f}s  sd={secs.std():5.1f}s  "
        f"cv={secs.std() / secs.mean():4.2f}  "
        f"p05={np.percentile(secs, 5):5.1f}s  p95={np.percentile(secs, 95):5.1f}s"
    )

# What to look at:
#   mean far above the team mean  -> that operator is struggling with the task
#   cv above ~0.35                -> that operator is not repeating one strategy
#   p95 near the episode_time_s   -> episodes are being truncated, not completed

Above that gate sits a checklist. The LeRobot dataset guidance is specific about what belongs in frame and what does not, and it is worth turning into a one-page card taped to each bench, because these are the errors that survive a merge and only surface during fine-tuning.

• Two camera views, steady mounts, at least 480x640 at about 30 fps.
• Neutral stable lighting with no colour cast, and consistent focus and exposure across the whole block.
• The leader arm must not be visible in frame. Neither should hands, arms or anybody walking past.
• Nothing else moving in frame except the follower arm and the manipulated object.
• Camera keys named by location, not by device: images.top, images.front, images.wrist.left, never images.laptop.
• One task string, 25 to 50 characters, identical on every seat.
• A last check that costs nothing: can you do the task yourself from the camera images alone? If not, neither can the policy.

AgiBot World runs the same idea at industrial scale, and the shape of their pipeline transfers down. Phase one is preliminary collection to validate each task and write the standards. Phase two is collection against those standards, with a local validity check that catches mechanical problems like missing frames before upload. Phase three is per-episode review against the phase-one standards. They also keep a human-in-the-loop cycle: record a small set, train, deploy, and let the failure tell you what to change about the recording. One concrete example from that paper is a model that paused at the start of every action, traced back to inconsistent transitions and idle time in the recordings, and fixed by changing the protocol and stripping idle frames in post-processing.

Merging seats into one dataset

Record per seat and per day, merge at the end. A single shared repo id across five operators means one bad block cannot be removed without disturbing the rest, and it means two people writing to the same dataset directory. The dataset docs cover the format side; the operational rule is that the merge is the last step, never the first.

# 1. per seat: remove the episodes review rejected, into a new repo id
lerobot-edit-dataset \
    --repo_id ${HF_USER}/cube2bin-seat2-day03 \
    --new_repo_id ${HF_USER}/cube2bin-seat2-day03-clean \
    --operation.type delete_episodes \
    --operation.episode_indices "[4, 11, 12, 19]"

# 2. merge the cleaned per-seat datasets, in a fixed order
lerobot-edit-dataset \
    --repo_id ${HF_USER}/cube2bin-day03-merged \
    --operation.type merge \
    --operation.repo_ids "['${HF_USER}/cube2bin-seat1-day03-clean', '${HF_USER}/cube2bin-seat2-day03-clean', '${HF_USER}/cube2bin-seat3-day03-clean']"

# 3. confirm what came out the other side
lerobot-edit-dataset \
    --repo_id ${HF_USER}/cube2bin-day03-merged \
    --operation.type info \
    --operation.show_features true

# optional: unify wording that drifted between seats
lerobot-edit-dataset \
    --repo_id ${HF_USER}/cube2bin-day03-merged \
    --operation.type modify_tasks \
    --operation.task_replacements '{"Put the red cube in the bin": "Pick up the red cube and place it in the bin"}'

One version detail worth writing on the wall: GR00T N1.7 and GR00T N1.5 need LeRobot v2.0 or v2.1 datasets and will crash on a v3.0 one, while ACT, SmolVLA and Pi0.5 expect v3.0. If your team is heading for a GR00T run, the merged dataset has to be converted down to v2.1 before it goes near the trainer. Decide that before you have 5,000 episodes to convert.

Two ways to run this

Where more operators do not help

Some problems look like throughput problems and are not. Adding seats to any of these makes the situation worse, because you produce more of whatever was already wrong.

• The task is not learnable from your camera views. More episodes of an unobservable task just produce a policy that guesses.
• The policy fails on a variation you never recorded. That is coverage, not volume: you need different scenes, objects and start poses, and one seat can produce those by changing the setup.
• You are below the trainer minimum anyway. The floors are 30 episodes for SmolVLA and 50 for ACT, GR00T N1.7, GR00T N1.5 and Pi0.5, and one operator clears those in an afternoon.
• The bottleneck is the training loop, not the data. If you are waiting on results rather than on episodes, more seats change nothing.
• Operators are recording rescue attempts. An episode where the grasp failed and the operator recovered teaches the model that retreating is the correct response to a good approach. AgiBot World keeps such trajectories deliberately, annotated with failure reasons, and they are about one percent of that dataset. Deliberate recovery data is useful; accidental recovery data mixed into ordinary episodes is not.

What a month actually costs

Hardware is a one-off, and small. Five leader-follower seats at the upstream parts price is roughly 1,150 USD of arms, plus cameras, hosts and mounting. The recurring costs are people and GPU time, and GPU time is the smaller one by a wide margin.

That asymmetry is the planning insight. A five-seat team can afford to throw away a week of recording and start over on a better protocol, and can afford to run a cheap ACT training run after every single day of collection just to see whether the curve is moving. Teams that do not do this discover on day 20 that days 1 through 19 shared a fixable defect. The data quality guide goes deeper on what those defects look like.