Record Your First LeRobot Dataset With an SO-100

An SO-100 follower, a leader arm of the same design and two USB cameras can fine-tune a policy in an afternoon. The same bench can as easily produce sixty episodes that look healthy in a file browser and waste a six-hour GPU run. The difference is rarely the model; it is what happened between the servos and the parquet file.

Here is the manual route, then the shorter one. Every command is from lerobot 0.6.1, released 3 August 2026 and current on PyPI. It moved to console entry points, so tutorials that run python lerobot/scripts/control_robot.py describe a file that no longer exists.

What a recording session captures

A LeRobot dataset is not a folder of videos but a time-indexed table with video attached: every control-loop tick writes one row holding the commanded action, the state the follower reached, one frame per camera, a timestamp and indices. The policy sees only those columns. The schema of lerobot/svla_so100_pickplace, read from its meta/info.json.

The joints are main_shoulder_pan, main_shoulder_lift, main_elbow_flex, main_wrist_flex, main_wrist_roll and main_gripper: the SO-100's six degrees of freedom. Action and state share a shape because leader-follower teleoperation records a target and the position reached one step later. That gap is information: where the arm fought gravity or a stuck object. Those strings are that dataset's. A session recorded with 0.6.1 today writes shoulder_pan.pos through gripper.pos, ids 1 to 6 on the bus: same six joints, different keys, which matters the moment a config addresses a feature by name.

What you need on the bench

If the arm is not built yet, that is a separate evening: start at SO-100 getting started and the complete SO-100 setup guide. If you have not bought anything, read the SO-100 against SO-101 comparison first: the SO-101 is the newer revision with improved wiring and no gear-removal step, and the recording workflow is identical.

Install lerobot 0.6.1

conda create -y -n lerobot python=3.12
conda activate lerobot

# TorchCodec is the default video decoder and needs ffmpeg
conda install ffmpeg -c conda-forge

# core_scripts = dataset + hardware + viz extras (record, replay, calibrate)
# feetech     = SDK for the STS3215 bus servos in the SO-100
pip install 'lerobot[core_scripts,feetech]'

lerobot-info

Extras trip people up most. pip install lerobot installs core ML dependencies only, nothing that talks to a robot. Koch arms need dynamixel instead of feetech. If your shell has never heard of lerobot-record, this is why.

Ports, motor IDs and calibration

Three one-off steps stand between parts and a working teleop loop. Calibration makes a policy trained on your arm run on somebody else's, mapping raw encoder counts onto a shared joint convention.

1. 1
   Find the USB port of each arm

   Run it with both arms plugged in, unplug the one you are identifying when prompted, and note which port disappears. On Linux you may need sudo chmod 666 /dev/ttyACM0.

   bash

   lerobot-find-port
   # Finding all available ports for the MotorsBus.
   # Ports before disconnecting: ['/dev/ttyACM0', '/dev/ttyACM1']
   # Remove the USB cable from your MotorsBus and press Enter when done.
   # The port of this MotorsBus is '/dev/ttyACM1'
   # Reconnect the USB cable.

2. 2
   Write motor IDs and baudrates

   Ids are written one motor at a time, and the docs are strict about how: connect exactly one motor to the controller board, not yet daisy-chained to any other. The script walks the chain backwards, prompting for the gripper first and giving it id 6, then wrist_roll as 5, down to shoulder_pan as 1. Do it before assembly.

   bash

   lerobot-setup-motors \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0
   
   lerobot-setup-motors \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1

3. 3
   Calibrate both arms

   Move every joint to the middle of its range, press Enter, then sweep each through its full range. The id becomes the profile filename.

   bash

   lerobot-calibrate \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_so100_follower
   
   lerobot-calibrate \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_so100_leader

4. 4
   Teleoperate before recording anything

   The acceptance test for everything above. If teleoperation is jerky, mirrored, or one joint will not follow, recording preserves that in 50 episodes.

   bash

   lerobot-teleoperate \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_so100_follower \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_so100_leader \
       --display_data=true

Cameras decide what the policy sees

lerobot-find-cameras opencv   # or: lerobot-find-cameras realsense

# --- Detected Cameras ---
# Camera #0:
#   Name: OpenCV Camera @ 0
#   Type: OpenCV
#   Id: 0
#   Backend api: AVFOUNDATION
#   Default stream profile:
#     Format: 16.0
#     Width: 1920
#     Height: 1080
#     Fps: 15.0

Two views, and where they sit matters: a fixed scene camera covering the workspace, and a wrist camera near the end-effector showing what the gripper is about to touch. The LeRobot community-datasets checklist asks for preferably two views at 480x640 / 720p or better, a static background, neutral stable lighting, and the leader arm and human limbs out of frame. The recording guide adds the rule of thumb: you should be able to do the task yourself by only looking at the camera images.

The record command and every flag

HF_USER=$(NO_COLOR=1 hf auth whoami | awk -F': *' 'NR==1 {print $2}')

lerobot-record \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_so100_follower \
    --robot.cameras="{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
    --teleop.type=so100_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=my_so100_leader \
    --display_data=true \
    --dataset.repo_id=${HF_USER}/so100_pick_cube \
    --dataset.single_task="Pick the red cube and drop it in the box" \
    --dataset.num_episodes=50 \
    --dataset.fps=30 \
    --dataset.episode_time_s=25 \
    --dataset.reset_time_s=10 \
    --dataset.streaming_encoding=true \
    --dataset.encoder_threads=2

The defaults below come from src/lerobot/configs/dataset.py on main, not a tutorial. Several are not what people assume.

Keyboard control during a session

• Right arrow or n: end the episode or reset phase early. The key you use most, because a clean grasp rarely needs 25 seconds.
• Left arrow or r: discard the episode and redo it. A bad take costs nothing now and a lot later.
• Escape or q: stop the session, finish encoding, upload.
• These work on X11, Wayland and headless SSH: with no global key backend, lerobot-record reads the same keys from the controlling terminal. The letters survive laggy SSH links, where arrow sequences split.
• Keyboard teleoperation is different and does need a global backend: X11, Windows, or macOS with Accessibility.

How many episodes, and what a good one looks like

The recording guide suggests at least 50 episodes for a first task, about 10 per object location. The policy pages list a minimum per model, below which a run is not worth the GPU time.

The better question is how many of what. Data Scaling Laws in Imitation Learning for Robotic Manipulation (Lin et al., 2024) collected over 40,000 demonstrations and ran more than 15,000 real-world rollouts. Generalisation followed a roughly power-law relationship with the number of environments and objects, and past a threshold per environment or object, additional demonstrations had minimal effect. On one bench: move the object, change the lighting, swap the cube, rather than repeat one take.

A good episode is boring: repeatable home pose, one thing done, ended once the object is in the bin, task string in the 25 to 50 characters the checklist recommends. Pick the red cube and drop it in the box is a task string; task1 is the anti-pattern the checklist names explicitly. Vague annotations head its list of problems, and they matter most for vision-language-action models, where the string is a model input, not a filename.

Defects that quietly ruin a dataset

None throw an exception. All survive into training, surfacing as a loss curve that looks fine and a robot that does nothing. Check while the scene is set up.

One caveat: the streaming-encoding guide's table lists the default as True, while the dataclass on main reads streaming_encoding: bool = False. Docs and code disagree, so set it explicitly; lerobot logs a hint recommending it whenever it starts with the flag off.

Check the dataset before renting a GPU

The acceptance test from the docs: compare video duration against the episode duration the CLI reported, and confirm the row count equals fps times duration. Per episode, not on the total.

from lerobot.datasets import LeRobotDataset

ds = LeRobotDataset("your-user/so100_pick_cube_20260823_141530")
print("fps:", ds.fps, "frames:", ds.num_frames)

# In v3.0 the per-episode records live in meta/episodes/ as chunked parquet:
# lengths, tasks and offsets into the shared parquet and mp4 shards.
# They load through the datasets stack, so this is a datasets.Dataset --
# use .column_names and integer indexing, not pandas .columns / .head().
eps = ds.meta.episodes
print(eps.column_names)
print(eps[0])

# Global feature statistics, including per-joint min and max.
# A joint whose min equals its max never moved. A joint sitting at a
# hard limit for most of the run is the one that will stall the policy.
print(ds.meta.stats["observation.state"])

Then look at it. lerobot-dataset-viz replays an episode frame by frame with joint traces beside the camera views, in Rerun or Foxglove. Swapped cameras and frozen frames show up in ten seconds. People skip this step.

# Replay one episode with camera views and joint traces
lerobot-dataset-viz \
    --repo-id your-user/so100_pick_cube_20260823_141530 \
    --episode-index 0

# Foxglove instead, for a seekable, scrubbable timeline
lerobot-dataset-viz \
    --repo-id your-user/so100_pick_cube_20260823_141530 \
    --episode-index 0 \
    --display-mode foxglove

# Drop the episodes that did not survive review
lerobot-edit-dataset \
    --repo_id your-user/so100_pick_cube_20260823_141530 \
    --new_repo_id your-user/so100_pick_cube_clean \
    --operation.type delete_episodes \
    --operation.episode_indices "[3, 17, 41]"

v2.1 or v3.0: decide before you record

v2.1 wrote one parquet and one MP4 per episode. v3.0 concatenates many episodes into shared shards and rebuilds boundaries from metadata, so info.json carries path templates like data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet instead of an episode number. The upstream justification is fewer, larger files: faster initialisation and less file-system pressure at scale.

# Upgrade an older v2.1 dataset to v3.0
python -m lerobot.scripts.convert_dataset_v21_to_v30 \
    --repo-id=your-user/so100_pick_cube

# By default it pushes the converted dataset back to the hub and tags it v3.0.
# To convert a local copy and keep it off the hub:
python -m lerobot.scripts.convert_dataset_v21_to_v30 \
    --repo-id=your-user/so100_pick_cube \
    --root=/path/to/dataset/directory \
    --push-to-hub=false

Two routes to the same dataset

From dataset to policy

Fifty clean episodes feeds every imitation learning run here. ACT trains from scratch on your task alone, about 80 M parameters at roughly 20 ms per action step, the only one of the five comfortable with fast motion. SmolVLA is about 450 M parameters on a 24 GB card. GR00T N1.7 is a roughly 3 B parameter foundation model where fine-tuning touches about 40 M parameters, needs an A100 or H100, and wants that v2.1 dataset.

Next, the guide for your combination: ACT on SO-100, SmolVLA on SO-100 or GR00T N1.7 on SO-100; for a first run, train your first policy is shorter. When the policy works on the bench but collapses the moment you move the table, that is a data problem: policy only works in one setup and collecting high-quality VLA training data go deeper on diversity.