The Data Bottleneck in Robot Learning

Where is the ImageNet for robots, the multi-terabyte dump you download once and fine-tune against forever? It does not exist, it is not close, and a bigger crawl will not fix it. Here is what the largest robot datasets contain, why they stay small, and what that means for one arm on a desk.

What internet-scale means, and what robotics has instead

The cleanest comparison is vision, not language. LAION-5B was crawled from the web and CLIP-filtered, ending at 5.85 billion image-text pairs, 2.32 billion of them English. Nobody staged those photographs; they existed already, as a by-product of people publishing things. Robot episodes are a by-product of nothing. Somebody sits down and produces each one.

Read the right-hand column, not the middle one. Every robot row describes a physical operation with a payroll attached; the vision row describes a crawler. One million trajectories against 5.85 billion image-text pairs is nearly four orders of magnitude. Longer write-ups of the two biggest entries: Open X-Embodiment and the DROID dataset.

Why the corpus cannot simply be scraped

The instinct is to point at YouTube, where millions of hours of people manipulating objects are sitting right there. Six reasons that is not the corpus, all of which show up as engineering work.

1. The action labels are not in the frame. Video shows what a hand ended up doing, not what joint command produced it. Imitation learning needs the command, not the outcome.
2. Embodiments disagree. Open X-Embodiment pools 22 embodiments, but the RT-X models in that paper trained on data from 9 manipulators. Merging action spaces is not free.
3. Control rates disagree. RT-1 commands actions at 3 Hz, other datasets record at 30 Hz or higher, and resampling changes what a single action means.
4. Proprioception is missing. Joint angles, gripper width and load live in the state vector, nowhere in the pixels. A LeRobot dataset stores them as a separate parquet stream for that reason.
5. There are almost no negatives. Teleoperated demonstrations are curated successes, so the corpus says little about what failure looks like.
6. Data ages with the hardware. Recalibrate, move a camera 3 cm, swap a gripper finger, and your old episodes describe a robot that no longer exists.

What collection actually costs in wall-clock time

These papers are unusually honest about their logistics, which lets you do throughput arithmetic. The rightmost column is our division, not a figure they state.

The DROID row is the one worth staring at. Fifty people, thirteen institutions, twelve months, 350 hours. That is seven hours per person per year, not because the people were slow but because teleoperation has an irreducible cost: reset the scene, drive the arm, notice the grasp slipped, discard, reset. The RoboTurk row is the counterexample: its throughput scales with people rather than robots. That is the bet behind remote operator work and behind browser-based driving on the live arm.

Five things people are doing about it

1. Pool what already exists

Open X-Embodiment is the flagship attempt: 60 datasets from 34 labs, converted to RLDS and published as one union. It works well enough that RT-X showed positive transfer across platforms, and it also shows the limits of pooling. The paper's own figure notes that Franka has the largest diversity of distinct scenes, while xArm and Google Robot contribute the most trajectories because of a few large datasets. The union is not a uniform sample of robotics. It is a stack of lab-shaped lumps.

# Open X-Embodiment lives in a public GCS bucket, stored as RLDS episodes.
# Pull one sub-dataset rather than the whole thing:
gsutil -m cp -r gs://gdm-robotics-open-x-embodiment/bridge ~/tensorflow_datasets/

# DROID ships four variants. Start with the 100-episode debug slice (2 GB),
# not the 1.7 TB RLDS build and definitely not the 8.7 TB raw stereo build.
gsutil -m cp -r gs://gresearch/robotics/droid_100 ./droid_100

# Full RLDS DROID, 1.7 TB, only when you know you need it:
# gsutil -m cp -r gs://gresearch/robotics/droid ./droid

2. Make the robot cheap enough that data comes from everywhere

The second strategy: drop the price of a data-producing rig until hobbyists outnumber labs. Hence the SO-100 at roughly 110 to 150 EUR in parts, and LeRobot as a common format. Narrowly it works: on 24 August 2026 the Hugging Face Hub listed 78,214 datasets under the robotics task category. Broadly it does not, because most are a few episodes of one task on one desk, with no language annotation and no consistent camera convention. Repository count is not usable data. The data quality guide is the checklist for landing in the usable half.

3. Borrow supervision from the web

If you cannot get more robot data, lean on data that is not robot data. This is the co-training argument behind every current vision-language-action model. The clearest published number is Pi0.5: about 400 hours of mobile manipulator data from about 100 home environments, but 97.6% of the first-phase training examples come from elsewhere, including cross-embodiment lab data, non-mobile robot data and web data such as captioning and visual question answering. The robot data steers a model the internet mostly taught. Mechanism in the VLA overview, model-by-model numbers on the policies page.

4. Manufacture the data you cannot collect

NVIDIA's GR00T N1 paper calls this a data pyramid: web and human video at the base, synthetic data in the middle, real teleoperation at the peak. The ratios are the interesting part. The in-house teleoperated set was 88 hours; video generation models expanded it to 827 hours of neural trajectories. Separately, DexMimicGen produced 780,000 simulation trajectories, about 6,500 hours equivalent, in 11 hours. The base is action-less human video from Ego4D, EPIC-KITCHENS, HOI4D and similar, with pseudo-actions attached by latent action models and inverse dynamics.

5. Curate instead of collect

The cheapest lever left is using the data you already have better. Re-Mix optimises the mixture weights over Open X-Embodiment sub-datasets rather than guessing them, and reports 38% over uniform weights and 32% over the human-selected weights used for the RT-X models, with no new robot hours collected. A small team can copy the idea: before recording another 200 episodes, check whether half of them are duplicates of one lighting condition.

What the scaling analyses actually say

The most useful paper here is Data Scaling Laws in Imitation Learning for Robotic Manipulation (Lin et al., posted October 2024, revised through June 2026): over 40,000 demonstrations with handheld UMI grippers and over 15,000 real-world rollouts, to answer one question: what do you get for each additional unit of data?

That does not remove the corpus problem, it reframes it. What is scarce is not trajectory count, it is covered conditions. A lab chasing a generalist policy across every kitchen on earth needs an enormous number of conditions. You, chasing one pick-and-place on one desk, need about a dozen.

The manual path: getting real data on disk today

1. 1
   Install LeRobot and check it runs

   The format most of this ecosystem standardised on. Dataset v3.0 ships in lerobot 0.4.0 and later.

   bash

   pip install lerobot
   lerobot-info

2. 2
   Pull a small public slice before you pull a large one

   The 100-episode DROID slice is 2 GB and tells you in minutes whether your loader and disk are up to it. The full RLDS build is 1.7 TB.

   bash

   gsutil -m cp -r gs://gresearch/robotics/droid_100 ./droid_100

3. 3
   Record your own episodes, because the public data is not your robot

   The command that actually creates new data. Vary the scene between batches rather than repeating one setup.

   bash

   lerobot-record \
     --robot.type=so101_follower \
     --robot.port=/dev/tty.usbmodem585A0076841 \
     --robot.id=my_awesome_follower_arm \
     --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \
     --teleop.type=so101_leader \
     --teleop.port=/dev/tty.usbmodem58760431551 \
     --teleop.id=my_awesome_leader_arm \
     --display_data=true \
     --dataset.repo_id=${HF_USER}/record-test \
     --dataset.num_episodes=5 \
     --dataset.single_task="Grab the black cube"

4. 4
   Know which format version you produced

   v3.0 packs many episodes per parquet and mp4 file; v2.1 was one file per episode. Some trainers accept only one, so check before renting a GPU.

   bash

   cat meta/info.json | python3 -m json.tool | head -20

5. 5
   Convert between format versions when the trainer demands it

   Mind the direction. The documented converter goes v2.1 to v3.0: it aggregates per-episode files into shards and rewrites the episode offsets. GR00T N1.7 and N1.5 want v2.0 or v2.1, so for those the job is to have the v2.1 dataset in the first place, not to run this script.

   bash

   python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=<HF_USER/DATASET_ID>

6. 6
   Train, then measure on the real arm rather than on the loss curve

   Falling loss says the model reproduces your demonstrations. It says nothing about whether the arm finishes the task.

   bash

   lerobot-train \
     --policy.type=act \
     --dataset.repo_id=${HF_USER}/record-test

Two ways to get from no data to a trained policy

What this means for a small team

The practical number is smaller than the papers make it sound. Because these policies start from vendor base checkpoints, not random weights, fine-tuning needs demonstrations in the tens, not the tens of thousands. The exception is ACT, which has no base model and trains from scratch on your task.

Thirty is the lowest bar there, which is why SmolVLA is usually the right first attempt: 30 episodes on a 24 GB card for one to three dollars tells you whether the data is any good before you commit to a GR00T N1.7 run on an A100. If the small model produces nothing, the problem is almost always the dataset. Full pricing is on its own page.

Where this platform does not help

Nothing here manufactures training data for you, and no proprietary corpus sits behind the login. The dataset directory holds what people chose to publish, which for your specific task is probably nothing. If the task is 'sort my particular bolts into my particular tray', the published record of robotics contains zero episodes of it, and you are going to record them.

A collection plan you can run this weekend

The scaling analysis translated into an afternoon: one arm, one task, one person, the case the papers never describe.

1. Pick one task and write its instruction string first. If it does not fit in one sentence, it is two tasks.
2. Define six to twelve conditions before recording anything: object position, object identity, lighting, background, table height, distractors. These are your environment-object pairs.
3. Record 5 to 10 episodes per condition. Six conditions at 8 episodes is 48, which clears the 30-episode SmolVLA bar.
4. Do not adjust the rig between conditions. Adjust it before condition one or after the last, never in the middle.
5. Discard failed episodes at record time. A half-successful grasp teaches half-successful grasps.
6. Train SmolVLA first, for one to three dollars, and evaluate on the real arm.
7. Only once the small model shows signs of life, spend 4 to 12 dollars on GR00T N1.7 or Pi0.5 with the same data.

Works in condition one, fails in condition six? That is a data answer: record more of six. Fails everywhere? Read loss falls but the policy does nothing before recording anything else. Never recorded a dataset at all? Record your first dataset walks the loop end to end, and SO-100 data collection covers the arm-specific parts.

Sources