Record your first imitation learning dataset

You have a calibrated arm. Now you need data. This guide takes you through recording a first dataset that is actually good enough to train on, which is a lower bar than people fear and a higher bar than people assume.

Step 2 of 4 · 20 min · beginner · Last updated 2026-08-07

Pick a task you can actually finish

Your first task should be boring. Pick up one rigid object and place it in one location. No stacking, no tools, no multi-step sequences. You are validating a pipeline, not demonstrating ambition.

Good first taskSave for later
Pick up a cube, put it in a bowlStack three blocks in order
Push a block to a marked spotOpen a drawer and retrieve an item
Pick up a marker, place it in a cupFold a cloth

Set up the scene once and leave it

Fix your cameras so they cannot be nudged. Everything you record before you bump the tripod becomes worthless the moment you bump it, because the policy learns pixel positions.

  • Scene camera: the whole workspace, arm and object both visible throughout
  • Wrist camera if you have one: pointed where the gripper closes
  • Lighting: constant. Close the blinds rather than fight the sun
  • Background: uncluttered, and not changing between episodes

Record the first episode

  1. 1
    Write the task string

    One sentence, and use exactly the same sentence for every episode of this task. "pick up the red cube and place it in the bowl".

  2. 2
    Reset

    Arm to home, object placed. Vary the object position between episodes, but keep it inside the region you want the policy to handle.

  3. 3
    Record

    Three-second countdown, then teleoperate. Move deliberately. Stop recording the moment the task is done.

    bash
    ay-robots record --task "pick up the red cube and place it in the bowl"
  4. 4
    Review before you keep it

    Play it back. Did the grasp succeed? Was the motion smooth? Was the object visible throughout? If any answer is no, discard it.

Discard failures, always

A failed grasp in your training set is not a neutral data point, it actively teaches the policy to fail. Reviewing costs seconds. Training on bad data costs hours and then lies to you about why the policy does not work.

Get to fifty

Repeat until you have fifty kept episodes. Vary the object position across the reachable area. Do not vary your strategy: approach the same way every time, or you are building two datasets that disagree with each other.

bash
# Record a batch without re-typing the task each time
ay-robots record --task "pick up the red cube and place it in the bowl" --episodes 10

# See where you stand
ay-robots datasets list --local
Take breaks

Demonstration quality drops measurably when you are bored. Three sessions of twenty episodes produce better data than one session of sixty, even though the count is the same.

Upload

Recordings are written locally first. When you are satisfied with the set, upload it. Nothing leaves your machine before that point.

bash
ay-robots datasets upload --name "SO-100 red cube pick v1"
Record as LeRobot v2.1 if you plan to use GR00T

GR00T will not load a v3.0 dataset. The client defaults to v2.1 for exactly this reason, but if you changed the setting, change it back before you record fifty episodes you cannot train on.