Fleet Data Collection: One Task, Several Locations

One arm on one desk produces a policy that works on that desk. That is the most repeatable result in low-cost manipulation learning, and it is why teams start looking for a second room. Recording the same task at three sites is not three times the data. It is a different kind of data, and it only pays off if the three recordings pour into one LeRobot dataset without anyone rewriting metadata by hand.

This page covers the mechanics: which fields the merge refuses to reconcile, which differences it accepts in silence, what the DROID team standardised across 13 institutions, and the commands that turn three site folders into one training set. Every LeRobot detail comes from lerobot 0.6.2 on main, read on 24 August 2026. Upstream moves, so check the version before copying flags.

What recording in several places actually buys

The strongest public evidence comes from DROID, designed as a distributed collection effort rather than a single-lab dataset. The team built one hardware platform and replicated it 18 times: a Franka Emika Panda 7 DoF arm with a Robotiq 2F-85 gripper, two adjustable Zed 2 stereo cameras on tripods, a wrist-mounted Zed Mini, all on a height-adjustable standing desk with wheels so the station could be pushed into a different room. Actions were recorded in both joint space and end-effector space at 15 Hz.

The number that matters is not the headline size, it is the ablation in section V-C, "How important is the scene diversity in DROID?". The authors built two subsets of equal size: 7362 trajectories from the 20 scenes with the most demonstrations each, against 7362 successful demonstrations sampled uniformly at random. Same trajectory count, very different scene coverage, and the diverse subset won out of distribution. In the headline comparison, across 6 tasks in 4 locations, co-training with full DROID beat the next best method by 22 percent absolute success rate in distribution and 17 percent out of distribution.

Hold on to that when someone proposes a second arm in the same lab. Two arms in one room give you throughput. Two arms in two buildings give you throughput plus the thing that actually fixes a policy that only works in one setup. If the budget covers exactly one extra arm, the second room is usually the better spend.

The three fields that must be identical

LeRobot does not guess. Before it copies a frame, the merge loads the metadata of every source and compares three things against the first in the list. If any differ, it raises and stops. The check itself, from src/lerobot/datasets/aggregate.py in lerobot 0.6.2:

fps = all_metadata[0].fps
robot_type = all_metadata[0].robot_type
features = all_metadata[0].features

for meta in tqdm.tqdm(all_metadata, desc="Validate all meta data"):
    if fps != meta.fps:
        raise ValueError(f"Same fps is expected, but got fps={meta.fps} instead of {fps}.")
    if robot_type != meta.robot_type:
        raise ValueError(
            f"Same robot_type is expected, but got robot_type={meta.robot_type} instead of {robot_type}."
        )
    if not features_equal_for_merge(features, meta.features):
        raise ValueError(
            f"Same features is expected, but got features={meta.features} instead of {features}."
        )

The encoder exemption is narrower than it looks. features_equal_for_merge strips exactly six keys from a video feature before comparing: video.g, video.crf, video.preset, video.fast_decode, video.extra_options and video.video_backend. Two keys people assume are in that set are not. video.codec and video.pix_fmt are derived from the stored stream, so they stay in the comparison, and the recorder's vcodec resolves per machine: left on auto it takes the first available hardware encoder and falls back to libsvtav1 when there is none. Two sites running the same script on different machines can write two different codec strings, and that is a ValueError at merge time, not a harmless difference. Resolution is not exempt either, since it lives in the feature shape.

// lerobot/svla_so100_pickplace  meta/info.json (condensed)
"robot_type": "so100",  "codebase_version": "v3.0",  "fps": 30,
"features": {
  "action": { "names": ["main_shoulder_pan", "main_shoulder_lift", "main_elbow_flex",
                        "main_wrist_flex", "main_wrist_roll", "main_gripper"] },
  "observation.images.top":   { "shape": [480, 640, 3], "info": { "video.codec": "av1" } },
  "observation.images.wrist": { "shape": [480, 640, 3], "info": { "video.codec": "av1" } }
}

// lerobot/svla_so101_pickplace meta/info.json (condensed)
"robot_type": "so100_follower",  "codebase_version": "v3.0",  "fps": 30,
"features": {
  "action": { "names": ["shoulder_pan.pos", "shoulder_lift.pos", "elbow_flex.pos",
                        "wrist_flex.pos", "wrist_roll.pos", "gripper.pos"] },
  "observation.images.up":   { "shape": [480, 640, 3], "info": { "video.codec": "av1" } },
  "observation.images.side": { "shape": [480, 640, 3], "info": { "video.codec": "av1" } }
}

What should vary, and by how much

Everything the validator does not look at is fair game, and most of it should move. DROID put this in the tooling rather than the guidelines: the collection GUI periodically prompted the operator to perform a randomly sampled scene augmentation, drawn from nudges to the mobile base, moving and re-calibrating the third-person cameras, changing the room lighting, and adding or removing items. Diversity was a scheduled interruption, not a hope.

• Room, table, and background. This is the whole point of a second site and costs nothing extra.
• Lighting. Window light at different times of day beats one ring light in three rooms.
• Third-person camera pose. DROID ended up with 1417 distinct viewpoints and treated that as a headline feature.
• Object instances. Same task, different mug, different cube colour.
• Clutter. DROID collectors were told to pick scenes with a healthy amount of it.
• Operator. Different hands produce different approach angles and pause patterns.
• Start pose randomisation, within whatever range your gripper can recover from.

One variation looks harmless and is not: wrist camera mounting. The wrist view is what most policies use to decide when to close the gripper, so a rotated wrist camera at one site is not scene diversity, it is a conflicting sensor definition wearing the same feature key. Nothing in the merge catches it, because the shape and the key name are unchanged. Fix the mount with a printed part and a photograph in the contract sheet.

Write the site contract before anyone records

The merge validator is a late, blunt check: it tells you the datasets are incompatible after three people have each spent a week recording. The cheap version is a one-page contract every site signs off before episode zero, plus a five-minute verification run per site that produces a metadata dump you can diff.

Merging: the actual commands

The tool is lerobot-edit-dataset, a console script in the lerobot package. It exposes nine operations: delete_episodes, split, merge, remove_feature, modify_tasks, convert_image_to_video, recompute_stats, reencode_videos and info. Merge runs aggregate_datasets underneath. Note the ordering: fix the strings and drop the bad episodes first, because the merge rewrites episode indices and afterwards you no longer know which came from where.

1. 1
   Install and read all three datasets before touching anything

   The info operation prints repo id, total episodes, total tasks, total frames, average frames per episode, average episode time, fps and size on disk. With show_features it dumps the feature dictionary as JSON, which is what you diff between sites. It does not print robot_type, so read that from meta/info.json separately.

   bash

   pip install 'lerobot[dataset]'
   
   for SITE in berlin lisbon tallinn; do
     echo "===== $SITE"
     lerobot-edit-dataset \
       --repo_id myorg/pickplace_$SITE \
       --operation.type info \
       --operation.show_features true
     jq -r '.robot_type, .fps' ~/.cache/huggingface/lerobot/myorg/pickplace_$SITE/meta/info.json
   done

2. 2
   Normalise the task strings

   The merge builds its task table with pd.concat(...).index.unique(), so two spellings of one instruction survive as two tasks and a task-conditioned policy treats them as two goals. This operation modifies in place, so copy first if you care.

   bash

   lerobot-edit-dataset \
     --repo_id myorg/pickplace_lisbon \
     --operation.type modify_tasks \
     --operation.task_replacements '{"pick up the cube and put it in the box": "Pick up the cube and place it in the box."}'

3. 3
   Delete the episodes you already know are broken

   Blocked cameras, a servo that sagged, a run where the operator gave up. Do it per site, while the episode indices still mean something locally.

   bash

   lerobot-edit-dataset \
     --repo_id myorg/pickplace_tallinn \
     --new_repo_id myorg/pickplace_tallinn_clean \
     --operation.type delete_episodes \
     --operation.episode_indices "[7, 19, 20, 41]"

4. 4
   Merge

   new_repo_id is required for merge, and repo_id is the one argument that is not. Pass roots when the datasets sit at explicit local paths rather than under $HF_LEROBOT_HOME. Episode and frame indices get constant offsets in the order you list the sources, task indices are recomputed from the stable task strings, and video timestamps are shifted per source file.

   bash

   lerobot-edit-dataset \
     --new_repo_id myorg/pickplace_fleet \
     --new_root /data/pickplace_fleet \
     --operation.type merge \
     --operation.repo_ids "['myorg/pickplace_berlin', 'myorg/pickplace_lisbon', 'myorg/pickplace_tallinn_clean']" \
     --operation.roots "['/data/berlin', '/data/lisbon', '/data/tallinn_clean']"

5. 5
   Check the totals and the task table

   Total episodes should equal the sum of the three sources. Total tasks should equal the number of distinct instructions you intended, not the number of sites. If it came out as three when you meant one, step two did not take.

   bash

   lerobot-edit-dataset \
     --repo_id myorg/pickplace_fleet \
     --root /data/pickplace_fleet \
     --operation.type info \
     --operation.show_features false

6. 6
   Push it, or keep it local

   push_to_hub is a top-level flag and every operation honours it, merge included, so add it to the merge rather than running a second command. Do not use recompute_stats as a way to push: with neither --new_repo_id nor --new_root set it copies the dataset to <repo_id>_recomputed_stats and pushes that instead, and in-place recomputation refuses to run without --operation.overwrite true.

   bash

   # push straight from the merge, rather than as a second step
   lerobot-edit-dataset \
     --new_repo_id myorg/pickplace_fleet \
     --new_root /data/pickplace_fleet \
     --operation.type merge \
     --operation.repo_ids "['myorg/pickplace_berlin', 'myorg/pickplace_lisbon', 'myorg/pickplace_tallinn_clean']" \
     --operation.roots "['/data/berlin', '/data/lisbon', '/data/tallinn_clean']" \
     --push_to_hub true

Weighting the sites, and why you usually cannot

Once merged, the sites are indistinguishable. There is no site column and no sampler that draws evenly from three unequal contributions. If Berlin recorded 300 episodes and Tallinn managed 60, the merged set is 83 percent Berlin and your policy learns Berlin's lighting as the default world. There are two honest ways around it and both have a cost.

That last row is the one real escape hatch, and it belongs to NVIDIA rather than LeRobot. The GR00T N1.7 fine-tune entry point takes dataset_path, documented as "Path to one dataset root, or an os.pathsep-separated list of dataset roots", and ds_weights_alpha, a power-law exponent where each dataset's sampling weight is len(dataset) to the power alpha and per-dataset mix_ratio values are ignored once it is set. Alpha of 1 samples in proportion to size, which is what merging gives you anyway. Alpha of 0 makes every weight 1, so every site is sampled equally. The default is None, so the flag is off unless you set it.

# Isaac-GR00T, gr00t/experiment/launch_finetune.py
# dataset_path is split on os.pathsep (":" on Linux)
python -m gr00t.experiment.launch_finetune \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path /data/berlin:/data/lisbon:/data/tallinn \
  --embodiment-tag new_embodiment \
  --ds-weights-alpha 0.0 \
  --output-dir /data/out

For every other policy, merging is mandatory. In lerobot 0.6.2 make_dataset raises NotImplementedError when repo_id is not a string, with the MultiLeRobotDataset branch sitting unreachable behind it. Passing a list of datasets to SmolVLA or ACT training is not a supported path today, whatever older tutorials suggest.

Do it yourself, or do it here

# Run this identical command at every site.
# Only --robot.port, --robot.id and the camera indices may differ.
lerobot-record \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=site_berlin_follower \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
  --teleop.type=so100_leader \
  --teleop.port=/dev/ttyACM1 \
  --teleop.id=site_berlin_leader \
  --dataset.repo_id=myorg/pickplace_berlin \
  --dataset.num_episodes=100 \
  --dataset.single_task="Pick up the cube and place it in the box." \
  --dataset.rgb_encoder.vcodec=h264

Where multi-site collection does not help

Three sites will not rescue a task a single site cannot demonstrate cleanly. If the policy fails because the gripper cannot hold the object or the wrist camera cannot see the contact point, adding rooms multiplies the same flaw. Fix the task at one site, get a working checkpoint, and only then scale. The data quality guide is the prerequisite for this page.

Three further limits are worth stating plainly, because none are fixed by adding sites. The third catches teams who assume a distributed dataset implies a distributed deployment.

• No held-out site. The merged dataset has no site column, and lerobot's eval_split holds out the last ceil(n * eval_split) episodes per task rather than a sample per location. Because the merge appends sources in the order you list them and offsets episode indices by a constant, that tail is mostly whichever site you merged last: an accidental held-out site, not a chosen one. For a real cross-environment test, leave one site unmerged and evaluate on its arm.
• No seed in GR00T. Its fine-tune entry point exposes none, so two runs on two site mixtures are not bit-for-bit comparable and a small success-rate gap may be run-to-run noise. lerobot's default seed is 1000, so ACT, SmolVLA and Pi0.5 comparisons are on firmer ground.
• Unchanged inference latency. The control loop is 20 to 485 ms per action step depending on the model, and running the policy on a cloud GPU while the arm sits in another country adds public-internet round trips on top. Workable for slow pick-and-place, not for fast reactive motion, whatever the training data looked like.

There is also a format trap on the way in. GR00T N1.7 and N1.5 want LeRobot v2.0 or v2.1, and a v3.0 dataset crashes the GR00T loader, so a freshly merged v3.0 fleet dataset has to be converted down before a GR00T run. Pi0.5, SmolVLA and ACT take v3.0 directly, and the dataset rejected as v3 page covers the conversion. How heterogeneous data behaves once pooled is the subject of Open X-Embodiment, which pooled 60 datasets from 34 labs into 22 embodiments and had to coarsely align action spaces to do it, accepting that camera observations still varied substantially across sources.