Teleoperation Ergonomics: Recording Sessions That Produce Clean Data

The dataset does not record the task, it records you

Every frame in a LeRobot dataset has an action column, and that column is the position your hand put the leader arm in at that instant. There is no filter between your wrist and the training target. Overshoot a grasp and correct back, and the correction is in the data. Pause because you lost track of which cube is next, and those sixty frames of stillness are in the data. Approach from a slightly higher arc than you did an hour ago, and the policy sees two approach distributions for what you call one task.

This is the whole premise of imitation learning: the model fits actions given observations, and it cannot know which parts of your behaviour were the task and which were you. The ACT paper notes that all human demonstrations are inherently stochastic even when a single person collects all of them. That stochasticity is why action chunking exists at all. Chunking absorbs some of it. It does not absorb a fatigued operator.

That table lists mechanisms, not measured effects on your arm. The direction is the part that is well supported: in the RoboTurk real-robot study covered further down, task completion times dropped steadily as users gained experience. Operators change over a session, and the dataset is the integral of that change.

The leader arm is ergonomic hardware, and it shows

A leader-follower rig is the cheapest good teleoperation interface there is, because the mapping is joint to joint and your proprioception does the inverse kinematics for free. The catch is that you hold the leader up against gravity for the whole episode. Every design in this space has spent effort on that one problem.

On the SO-101, the follower uses six STS3215 motors with 1/345 gearing, but the leader uses three different gear ratios. The LeRobot docs state the reason directly: the leader uses differently geared motors to make sure it can both sustain its own weight and be moved without requiring much force. On the older SO-100 the fix is cruder. The assembly guide has you remove the gears from all six leader motors, so that you only use the position encoding of the motor and reduce friction to more easily operate the leader arm.

The shoulder lift keeps the high 1/345 ratio on the SO-101 because it carries the most static load. The wrist joints drop to 1/147 because you move them constantly and the friction there is what wears your hand out. If you expect to record hundreds of episodes, read the SO-100 versus SO-101 comparison with that in mind, and the full SO-100 guide for the build itself. The parts cost difference is around 20 EUR. The difference in how your forearm feels after ninety minutes is larger than that.

The bimanual ALOHA setups went the same way. The original design added a 3D-printed handle and scissor mechanism to the leader specifically to reduce the force required from the operator to backdrive the motor, plus a rubber band mechanism to partially counteract gravity on the leader side. ALOHA 2 replaced the scissor gripper with a low-friction rail design and swapped the rubber bands for passive gravity compensation built from off-the-shelf parts. Two hardware generations, both spent on operator effort.

Bench setup before the first episode

Everything fiddly should be fiddly before you record, not between episodes. Ports first, because they change when you replug. Cameras second, because they change when you reboot. Calibration third, because it is the one step that silently poisons every later episode if you rush it.

1. 1
   Find and pin the serial ports

   Run this once per arm, unplugging when prompted. Write the two paths on tape and stick it on the USB hub. On Linux you may also need to relax permissions on the tty device.

   bash

   lerobot-find-port
   
   # example output
   # ['/dev/ttyACM0', '/dev/ttyACM1']
   # Remove the USB cable from your MotorsBus and press Enter when done.
   # The port of this MotorsBus is /dev/ttyACM1
   
   sudo chmod 666 /dev/ttyACM0
   sudo chmod 666 /dev/ttyACM1

2. 2
   Enumerate the cameras and note their real profile

   The identifiers can change after a reboot or a replug, and the default stream profile is often not what you want. A camera advertising 1920x1080 at 15 fps will not feed a 30 fps recording loop.

   bash

   lerobot-find-cameras opencv
   
   # --- Detected Cameras ---
   # Camera #0:
   #   Name: OpenCV Camera @ 0
   #   Id: 0
   #   Default stream profile:
   #     Width: 1920
   #     Height: 1080
   #     Fps: 15.0

3. 3
   Calibrate both arms in the same session

   Move every joint to the middle of its range, press Enter, then sweep each joint through its full range. Do the leader and the follower back to back so the two calibration files describe the same physical geometry. The id you pass here is the id you must reuse when recording.

   bash

   lerobot-calibrate \
     --robot.type=so101_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower_arm
   
   lerobot-calibrate \
     --teleop.type=so101_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader_arm

4. 4
   Drive it for five minutes with the visualiser on

   This is the dry run. Watch the camera feeds while you drive, not the arm. If you cannot do the task looking only at the camera images, the policy cannot either, and no amount of episodes will fix a view that does not show the contact.

   bash

   lerobot-teleoperate \
     --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower_arm \
     --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
     --teleop.type=so101_leader --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader_arm \
     --display_data=true

5. 5
   Set the bench so you are not reaching

   The leader base, your elbow and the work surface should let you drive with your upper arm close to your body. If you are recording standing with the leader at chest height, the shoulder lift joint carries your arm weight as well as the leader's, and that is the joint that decides how your approach trajectories drift.

What lerobot-record actually assumes about you

The defaults below are read from lerobot's own recording config, src/lerobot/configs/dataset.py on the main branch, on 23 August 2026. They are worth reading as a statement about pacing, because that is what they are. For how the format itself is laid out on disk, see the dataset documentation and the LeRobot dataset glossary entry.

Fifty episodes at one minute each plus a minute of reset is a hundred minutes of continuous fine motor work with no break structure at all. The defaults describe a budget; the protocol is yours to add. Fifty does line up with the minimum episode counts the trainers here want: 50 for ACT, GR00T N1.7 and Pi0.5, 30 for SmolVLA. That is a floor, not a target.

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --teleop.id=my_leader_arm \
  --display_data=true \
  --dataset.repo_id=${HF_USER}/cube_to_bin \
  --dataset.single_task="Pick the red cube and drop it in the bin" \
  --dataset.num_episodes=12 \
  --dataset.episode_time_s=25 \
  --dataset.reset_time_s=15 \
  --dataset.fps=30 \
  --dataset.push_to_hub=false

Two flags do the pacing work. episode_time_s=25 is roughly twice the time the task actually takes you, so a clean run finishes early and a fumbled one gets cut off instead of becoming a two-attempt episode. num_episodes=12 ends the block while you are still sharp. Resuming into the same dataset later is supported, and the flag semantics are easy to get wrong.

Pacing: the three keys that make a session survivable

Three keys control the flow during recording, and they are the difference between a session you can prune afterwards and one you cannot. On Linux they work on X11, Wayland and headless or SSH sessions with no $DISPLAY setup: when no global keyboard backend is available, lerobot-record reads the same keys from the terminal, so it has to run in an interactive TTY that has focus. That fallback covers the recording control flow only. Keyboard teleoperation, where the keyboard replaces the leader arm, still needs a global key backend.

• Right arrow or n: end the current episode or reset phase early and move to the next. This is your main tool. A clean run at 12 seconds should not sit there burning the remaining 13 seconds of episode_time_s with the arm parked.
• Left arrow or r: cancel the current episode and re-record it. The episode buffer is cleared and nothing is written. Use it the moment you know the attempt is bad, not at the end.
• Escape or q: stop the session, encode the videos and upload. This is the honest exit when you notice your hand is shaking.

The re-record key deserves a habit, not just knowledge. When an episode goes wrong at second 3 the instinct is to recover and finish, because finishing feels productive. That instinct is what fills a dataset with recovery behaviour. Press r; it costs ten seconds. Leaving the episode in costs a retraining run at 4 to 12 USD on the A100 tier or 1 to 3 USD on the 4090 tier, plus the hours.

ALOHA is the useful reference for session structure. Teleoperation and recording both run at 50 Hz there, episodes last 8 to 14 seconds depending on task complexity, which is 400 to 700 timesteps, and they recorded 50 demonstrations per task except Thread Velcro, which got 100. Short episodes, moderate counts. Five blocks of 12 episodes at a 25 s cap is about 8 minutes of recording each:

That schedule is assembled from the pieces above, not a measured protocol. What is not negotiable: a short episode cap, real breaks, and reviewing something between blocks. Five blocks of twelve clears the 50-episode floor for ACT on the SO-100 with a margin for the ones you will delete.

The reset is part of the data, even when it is not recorded

In lerobot-record the reset phase calls the same control loop as the episode, with the teleoperator live and the dataset argument left out. The arm still follows your leader; nothing is written. That is deliberate. You drive the follower back to its start pose with the same interface you just used, so the arm never gets yanked by hand and the servos never get back-driven against their gearing. The recorder also skips the reset after the last episode of a run, which is why a block that ends cleanly ends faster than the arithmetic suggests.

The effect on the dataset is indirect but strong. The object start pose at frame 0 of the next episode is entirely determined by how you reset. Place the cube by eye, let your eye drift over the session, and you have added a variable the dataset does not label. The LeRobot guidance is roughly 10 episodes per object location, and to avoid adding too much variation too quickly. That only works if the locations are actual locations.

What the evidence says about operator variance

The clearest experiment here is robomimic's dataset design. The Proficient-Human sets came from 1 operator on the RoboTurk platform, 200 successful trajectories each. The Multi-Human sets came from 6 operators, 50 trajectories each for 300 total, rated 2 worse, 2 okay, 2 better. The result worth carrying around: success rates are lower on the Multi-Human sets than on the Proficient-Human sets even though the Multi-Human sets contain 100 more demonstrations. The study attributes that to suboptimal and multimodal data, and separately reports that batch reinforcement learning methods are proficient on machine-generated data but poor on human data, while history-dependent models do markedly better on it.

That is usually read as a statement about multiple people. It is equally a statement about one person across a long session: the operator at episode 45 is not the one from episode 5. RoboTurk's real-robot deployment, which extended the platform from simulation onto three Sawyer arms, saw the drift in the other direction: 54 users in one week produced 2144 demonstrations and 111.25 hours across three tasks at a mean task length of 186 seconds, completion times dropping steadily with experience. There is a warm-up curve and a fatigue curve, and your dataset sits on both. The RoboTurk write-up covers that platform in more detail.

The DemInf result is the practical one: filtering a dataset by an automatic quality score, rather than collecting more of it, bought a 5 to 10 percent improvement on RoboMimic tasks plus better performance on real ALOHA and Franka setups. A 2026 framework called DQAF closes the loop during collection instead, scoring each episode on sub-task progress, motion smoothness, stalls and kinematic limits and turning that into natural-language feedback right after the episode. Read that one at its actual size: in a pilot with three novice operators, the one who got the automated feedback improved faster than those who did not, with no effect size reported. Neither tool ships with LeRobot today, so the manual equivalent is you, a notebook and the data quality checklist.

Two ways to run the session

pip install -e ".[feetech]"

# one block of 12 short episodes, stays local
lerobot-record \
  --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
  --teleop.type=so101_leader --teleop.port=/dev/ttyACM1 \
  --teleop.id=my_leader_arm \
  --dataset.repo_id=${HF_USER}/cube_to_bin \
  --dataset.root=./data/cube_to_bin \
  --dataset.single_task="Pick the red cube and drop it in the bin" \
  --dataset.num_episodes=12 \
  --dataset.episode_time_s=25 \
  --dataset.reset_time_s=15 \
  --dataset.push_to_hub=false \
  --display_data=true

# next block, same dataset: num_episodes is ADDITIONAL episodes
lerobot-record \
  ... same flags ... \
  --dataset.root=./data/cube_to_bin \
  --dataset.num_episodes=12 \
  --resume=true

# review a specific episode before you trust the block
lerobot-dataset-viz --repo-id ${HF_USER}/cube_to_bin \
  --root ./data --mode local --episode-index 7

Prune the episodes fatigue ruined

Pressing r during the session catches the obvious failures. It misses the slow ones, where the task succeeded but the trajectory wandered. Those you find afterwards: open the visualiser, step through the episodes you flagged, and delete the ones that are technically successful and behaviourally wrong.

# look at one episode: camera streams, joint states and actions on a timeline
lerobot-dataset-viz \
  --repo-id ${HF_USER}/cube_to_bin \
  --root ./data --mode local \
  --episode-index 41

# how many episodes and frames do I actually have
lerobot-edit-dataset \
  --repo_id ${HF_USER}/cube_to_bin \
  --operation.type info

# drop the bad ones into a NEW dataset, keeping the original intact
lerobot-edit-dataset \
  --repo_id ${HF_USER}/cube_to_bin \
  --new_repo_id ${HF_USER}/cube_to_bin_pruned \
  --operation.type delete_episodes \
  --operation.episode_indices "[7, 41, 42, 55]"

Two habits make this cheap. Keep a text file open with one line per episode index and a two-word note, because at review time you will not remember why episode 41 felt wrong. And keep one task string for the whole session: the guidance is 25 to 50 characters clearly describing the objective. To fix wording later, modify_tasks can rewrite it, but that operation edits in place and ignores new_repo_id.

Where none of this helps

Three honest limits. First, no amount of pacing discipline fixes a camera that cannot see the contact. The LeRobot rule of thumb is that you should be able to do the task yourself looking only at the camera images. If you cannot, stop recording and move a camera. Camera problems are cheaper to fix than dataset problems.

Second, this platform does not measure your fatigue. There is no posture check, no break timer, no per-episode smoothness score. The prototypes that score episodes on smoothness and stalls exist in papers, not in the tooling you have today. Your substitute is a short block, a notebook and the discipline to stop.

Third, distance. Driving an arm over the public internet works, and the no-hardware entry points exist for exactly that. But every round trip you add sits between your hand and the follower, and the same physics governs the return path at inference time. The control loop runs at 20 ms per action step for ACT, 152 ms for GR00T N1.7, 165 ms for GR00T N1.5, 245 ms for SmolVLA and 485 ms for Pi0.5. Adding public-internet round trips on top of inference latency turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place, not for fast reactive motion, and the same applies to remote recording of fast tasks.