Using DROID, BridgeData V2 and Open X on an SO-100

There is a million-trajectory public dataset on a Google Cloud bucket and a SO-100 on the desk that cost 110 to 150 EUR in parts. Why can the first not teach the second? It partly can, but almost none of the transfer happens where people expect, and the part that looks easiest does not work at all.

What follows: what is inside DROID, BridgeData V2 and Open X-Embodiment, where each collides with a low-cost 5-DoF arm, and what to do instead. Every number below came from the paper, dataset card or source file it belongs to.

What the three datasets actually contain

DROID

The most standardised of the three. One rig everywhere: a Franka Panda with a Robotiq 2F-85 gripper, two adjustable ZED 2 stereo cameras and a wrist ZED Mini, teleoperated with Meta Quest 2 controllers, recorded through Polymetis at 15 Hz in both joint and end-effector space. Language labels came later via tasq.ai, up to three per episode.

• 76k trajectories, 350 hours, 564 scenes, 86 tasks, 50 collectors on three continents.
• The headline result is co-training, not standalone training: batches mixed 50/50 with in-domain demonstrations beat the next best method by 22 percent absolute success in distribution, 17 percent out of it.
• IPEC-COMMUNITY/droid_lerobot: 92,233 episodes, 27,044,326 frames, franka, 15 fps, codebase_version v2.0, three AV1 streams at 180x320, 392 GB.
• A 2 GB, 100-episode debugging sample sits at gs://gresearch/robotics/droid_100. Start there.

BridgeData V2

The closest to a hobby setup: a WidowX 250 6-DoF arm, 60,096 trajectories across 24 environments and 13 skills at 5 Hz. Note the composition: 50,365 expert teleoperated demonstrations plus 9,731 from a randomised scripted pick-and-place policy, so about 16 percent is not human demonstration, which matters for imitation learning quality. The usual download, IPEC-COMMUNITY/bridge_orig_lerobot, reports 53,192 episodes and 1,893,026 frames at 5 fps, robot_type widowx: fewer than the paper's 60,096, so read the count from meta/info.json rather than quoting either.

Open X-Embodiment

Not a dataset in the same sense: 60 existing robot datasets from 34 labs pooled into one RLDS collection covering 22 embodiments and over a million trajectories. BridgeData V2 sits inside it as bridge_orig; the google_robot slice, fractal20220817_data, converts to 87,212 episodes at 3 fps.

The pooling carries a caveat the paper states outright. For the RT-X experiments the authors convert each source into a 7-DoF end-effector action, but do not align coordinate frames across datasets, and allow action values to be absolute or relative positions or velocities, per each robot's original control scheme. Their conclusion: the same action vector may induce very different motions for different robots.

The mismatch, in four parts

Embodiment mismatch is usually treated as one vague problem. It is four, they fail differently, and two are not fixable by scripting.

1. Degrees of freedom

An SO-100 has five arm joints plus a gripper. Counted as motors that is a 6-DoF arm, and the SmolVLA paper calls it that; counted as a positioning mechanism it is 5-DoF, and LeRobot calls it that in its inverse-kinematics docstring, which describes soft-orientation IK on the 5-DOF SO-101 where the wrist tracks orientation only partially. A Franka has seven positioning joints. That gap decides which poses exist: a 5-DoF arm cannot generally reach an arbitrary position and orientation at once, so the solver returns the closest it can, a different motion from the one demonstrated. Background: degrees of freedom.

# src/lerobot/robots/so_follower/so_follower.py
motors = {
    "shoulder_pan":  Motor(1, "sts3215", norm_mode_body),
    "shoulder_lift": Motor(2, "sts3215", norm_mode_body),
    "elbow_flex":    Motor(3, "sts3215", norm_mode_body),
    "wrist_flex":    Motor(4, "sts3215", norm_mode_body),
    "wrist_roll":    Motor(5, "sts3215", norm_mode_body),
    "gripper":       Motor(6, "sts3215", MotorNormMode.RANGE_0_100),
}
# action keys are "<motor>.pos"; send_action sync_writes them to
# "Goal_Position"  ->  a 6-D ABSOLUTE JOINT POSITION command


# openpi/src/openpi/policies/droid_policy.py
def make_droid_example() -> dict:
    return {
        "observation/exterior_image_1_left": np.random.randint(256, size=(224, 224, 3), dtype=np.uint8),
        "observation/wrist_image_left":      np.random.randint(256, size=(224, 224, 3), dtype=np.uint8),
        "observation/joint_position":        np.random.rand(7),   # seven Franka joints
        "observation/gripper_position":      np.random.rand(1),
        "prompt": "do something",
    }
# state = concat(joint_position, gripper_pos)  ->  8-D

So a ready-made DROID checkpoint is no shortcut. Physical Intelligence ships pi05_droid at gs://openpi-assets/checkpoints/pi05_droid, and the openpi README is plain about what those expert checkpoints are: fine-tuned from a base model, intended to run directly on the target robot, and they may or may not work on your particular robot. Its state is seven Franka joint positions plus a gripper value, eight numbers in all, and its image keys are exterior_image_1_left and wrist_image_left. No flag turns that into a six-motor SO-100 command.

2. What the action vector actually says

Deeper than dimensionality. In the LeRobot conversions all three say where the gripper should go, in Cartesian space. An SO-100 says where six servos should go. Converting needs a kinematic model and a solver, not a reshape.

LeRobot does ship the missing piece: the SO follower has a kinematics processor with InverseKinematicsEEToJoints and ForwardKinematicsJointsToEE steps. Its keys are ee.x, ee.y, ee.z plus a rotation vector ee.wx, ee.wy, ee.wz and ee.gripper_pos, so even the orientation encoding differs from the roll-pitch-yaw in the files. The IK step takes an orientation_weight, default 0.01, whose docstring says to set 0.0 for position-only IK on under-actuated arms. You can build the bridge, but the orientation half of every borrowed action stays approximated.

3. Control rate

DROID is 15 Hz, BridgeData V2 5 Hz, the google_robot slice 3 fps; pi0's authors describe the open-source part of their mixture as low-frequency control between 2 and 10 Hz. LeRobot's DatasetRecordConfig defaults to fps 30, episode_time_s 60, reset_time_s 60, num_episodes 50. A policy trained on 5 Hz data learned that one action covers 200 ms. Replay it at 30 Hz and the arm crawls; resample naively and you smear the frame where the gripper closes. It also interacts badly with action chunking: a 100-step chunk is 20 seconds at 5 Hz, 3.3 at 30 Hz.

4. Cameras

BridgeData V2 randomised two camera poses every 50 trajectories, and its project page notes most of the data carries only the fixed view anyway. DROID used adjustable ZED 2 mounts plus a wrist ZED Mini. You have two USB webcams positioned by eye. Camera pose is not a nuisance variable for a vision-language-action model; it is much of what the visual encoder keyed on, and nothing in the file format tells you the poses differ.

What happens when you try to merge the data anyway

The obvious plan is to concatenate: a few thousand DROID episodes plus your 50. LeRobot refuses, and the refusal names the three things that differ.

1. 1
   Pull the 100-episode sample, not the full 1.7 TB

   2 GB is enough to see the structure.

   bash

   pip install gsutil tensorflow tensorflow-datasets
   gsutil -m cp -r gs://gresearch/robotics/droid_100 ~/tensorflow_datasets/

2. 2
   Convert RLDS to LeRobot form

   openx2lerobot wraps the OXE standard transformations and annotates robot type and control frequency. The README puts this in convert.sh.

   bash

   git clone https://github.com/Tavish9/any4lerobot.git
   cd any4lerobot/openx2lerobot
   
   python openx_rlds.py \
       --raw-dir ~/tensorflow_datasets/droid_100/1.0.0 \
       --local-dir ~/lerobot_droid100 \
       --repo-id you/droid100_lerobot \
       --use-videos

3. 3
   Read meta/info.json before anything else

   This file decides whether the rest of your day works.

   bash

   python -c "import json;d=json.load(open('meta/info.json'));\
   print(d['codebase_version'], d['robot_type'], d['fps']);\
   print(d['features']['action']['shape'], d['features']['observation.state']['shape'])"

4. 4
   Try the merge and read the error

   merge loads every dataset, then validate_all_metadata checks fps, robot_type and features against the first in the list, raising on the first mismatch.

   bash

   lerobot-edit-dataset \
       --new_repo_id you/mixed \
       --operation.type merge \
       --operation.repo_ids "['you/droid100_lerobot', 'you/my_so100_task']"
   
   # ValueError: Same fps is expected, but got fps=30 instead of 15.

The reference values come from whichever dataset you listed first, which is why the message complains about your 30 fps rather than DROID's 15. Fix the fps and you hit the robot_type check; fix that and you hit the feature check, 7 against 6 for the action. No ordering gets through, and the same guard runs at record time via sanity_check_dataset_robot_compatibility.

So what actually transfers?

Weights, not episodes. Every modern generalist policy absorbed some of it in pretraining, and when you fine-tune from a released checkpoint you inherit it already reconciled by people with the compute to do it properly. pi0's paper is candid about the proportion: 9.1 percent of its pre-training mixture, counted in timesteps, is open-source data including OXE, Bridge v2 and DROID. That figure is pi0's; each vendor's mixture differs.

This is why SmolVLA behaves differently on a low-cost arm. Its paper selects 481 community datasets from Hugging Face, filtered by embodiment type, episode count, data quality and frame coverage: 22.9K episodes, 10.6M frames, evaluated on real SO-100 and SO-101 arms. In the paper's own ablation that pretraining is worth a jump from 51.7 to 78.3 percent average success across three real SO-100 tasks, against the same model trained without it. Compare on ACT against SmolVLA.

Three paths worth taking

Path A: fine-tune from a checkpoint that already ate the data

Most people should take this one. You never touch DROID or Open X-Embodiment: pick a policy whose pretraining already absorbed cross-embodiment data, record your own episodes, fine-tune.

The platform names the vendors' own base checkpoints for GR00T N1.7, GR00T N1.5 and Pi0.5; SmolVLA fine-tunes from lerobot/smolvla_base on the Hub. ACT is the honest edge case: no base model at all, so none of the public data ever reaches it. Not automatically a disadvantage, since at 20 ms per action step it is the only one of the five that can close a fast loop, as the ACT page sets out. Pick by task using all five compared, GR00T N1.7 against Pi0.5, and the 332 benchmark results across 85 models in the arena.

Path B: use DROID as a test fixture

The 100-episode sample is the best 2 GB you will download this month, and not for training. It is a dataset you know is correct. Run your converter, loader and a short GPU job on it; anything that fails is an infrastructure bug found while it was cheap. NVIDIA does the same at scale: the GR00T N1.7 card lists four post-trained variants, for Bridge and Fractal in SimplerEnv, DROID and LIBERO.

Path C: record your own, deliberately

Thirty to fifty episodes sounds small next to 76,000 until you remember yours are the only ones with your arm, your cameras and your table. At LeRobot's defaults, 50 episodes is 100 minutes of wall clock. See record your first dataset, SO-100 data collection and the data quality guide. Public LeRobot datasets that already match a supported arm are listed in the dataset directory.

Two ways to get from public data to a working policy

What each path costs

The asymmetry is the point: the path that borrows the most data is the most expensive and least likely to move your arm. Under two hours of your own teleoperation beats a terabyte of somebody else's Franka. No arm yet? /live streams a physical SO-100 with no signup. Then train your first policy, and SmolVLA on SO-100 for the specific guide.