How to Train ACT on the SO-100 from Scratch

ACT is the odd one out among the SO-100 policies. GR00T N1.7 and N1.5 start from nvidia/GR00T-N1.7-3B and nvidia/GR00T-N1.5-3B, Pi0.5 from lerobot/pi05_base. ACT starts from nothing: there is no base checkpoint, because it is not a foundation model. It is a roughly 80 million parameter transformer you train from scratch on one task, on your arm, under your lighting.

That is also why it runs a control step in 20 ms where a 3 billion parameter VLA needs 152 to 485 ms, and costs 1 to 3 USD per run instead of 4 to 12. This guide walks the manual route with lerobot on an SO-100: record, the act config, what chunk_size and n_action_steps control, the 100000 step schedule, the rollout. Then the same job on AY-Robots, including where the platform does not help.

What ACT actually is

Action Chunking with Transformers comes from the ALOHA paper, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, by Zhao, Kumar, Levine and Finn, arXiv, 23 April 2023. The rig records at 50 Hz with four webcams streaming 480x640 at 30 fps: two on the grippers, one top, one front. The abstract claims six skills at 80 to 90 percent success, among them opening a translucent condiment cup and slotting a battery, from 10 minutes of demonstrations.

The body is more useful when planning a recording session: 50 demonstrations per task, except Thread Velcro at 100, which is 10 to 20 minutes of data and 30 to 60 minutes of wall-clock time once resets are counted. Success is not uniform either: Thread Velcro ends at 20 percent, Put On Shoe at 92, Cup Open 84, Prep Tape 64.

Action chunking is the whole idea

Ordinary behavioural cloning maps one observation to one action, and errors compound: a deviation puts the arm off-distribution, producing a worse action, and thirty steps later the gripper is nowhere near the object. Action chunking predicts k actions at once and executes them, dropping the effective horizon by a factor of k. It also handles a nuisance specific to human data: teleoperators pause, and a single-step Markovian policy cannot model a pause that depends on what came before.

The paper ablates k rather than asserting it. With temporal ensembling off, averaged over four settings, success rises from 1 percent at k = 1 to 44 percent at k = 100, then tapers at 200 and 400 as the policy nears open-loop control. That curve is why the default is 100.

• chunk_size: how many future actions the decoder predicts per forward pass. Default 100.
• n_action_steps: how many of them you execute before querying again. Default 100, so lerobot runs the whole chunk open loop.
• lerobot validates n_action_steps <= chunk_size and raises a ValueError if you get it backwards.

What matters operationally is chunk_size divided by frame rate. At the 30 fps lerobot's SO-100 examples use, a chunk of 100 commits about 3.3 seconds from one observation. If the task needs a correction inside that window, lower n_action_steps, not chunk_size: you keep the long prediction and re-observe more often.

# predict 100 actions, re-query after 25 of them (about 0.8 s at 30 fps)
lerobot-train \
  --dataset.repo_id=${HF_USER}/so100_cube \
  --policy.type=act \
  --policy.chunk_size=100 \
  --policy.n_action_steps=25 \
  --policy.device=cuda

When ACT beats a foundation model

The five trainable policies side by side, with the numbers AY-Robots measures and uses to size the GPU it rents.

Pick ACT when task and scene are fixed and motion must be fast and precise. Pick a vision-language-action model when one checkpoint must cover several instructions. Two pages work the decision head to head: ACT vs SmolVLA and ACT vs GR00T N1.7. For published benchmarks, the ACT arena entry links every number to its source.

What you need before you start

One follower arm, one leader arm for teleoperation, at least one camera, a 24 GB GPU. ACT reads images and joint positions only. Two cameras is the sweet spot: a fixed front view for where things are, a wrist camera for what the end effector is about to touch, as on ALOHA.

From bare arm to recorded dataset

The flow below is lerobot 0.6.x. Skip it if you have a calibrated arm and a dataset. Otherwise the SO-100 getting started guide covers assembly, the recording walkthrough covers capture and the dataset docs the format.

1. 1
   Install lerobot with the right extras

   Recording needs core_scripts, training training, Feetech servos feetech. Python 3.12+.

   bash

   conda create -y -n lerobot python=3.12
   conda activate lerobot
   conda install ffmpeg -c conda-forge
   
   pip install 'lerobot[core_scripts,training,feetech]'
   lerobot-info

2. 2
   Find the USB port of each arm

   Run it with both arms plugged in, unplugging one when prompted. On Linux you may need to open the node permissions.

   bash

   lerobot-find-port
   # on Linux, if the port exists but is unreadable:
   sudo chmod 666 /dev/ttyACM0

3. 3
   Set the motor ids and baudrate

   On the SO-100 this happens before assembly: unlike the SO-101 the connectors are unreachable once built. The script walks the bus one motor at a time from the gripper, writing ids to EEPROM.

   bash

   lerobot-setup-motors \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0
   
   lerobot-setup-motors \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1

4. 4
   Calibrate both arms

   Set every joint to the middle of its range, press Enter, then sweep each through its full range. Calibration lets a policy trained on one arm run on another. Reuse the same id.

   bash

   lerobot-calibrate \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_follower
   
   lerobot-calibrate \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_leader

5. 5
   Teleoperate once with the cameras on

   The lerobot rule of thumb: you should be able to do the task looking only at the camera images. If you cannot, neither can ACT. This catches more bad datasets than later debugging.

   bash

   lerobot-teleoperate \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_follower \
       --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_leader \
       --display_data=true

6. 6
   Record 50 episodes

   50 is the AY-Robots minimum and what ALOHA used per task. lerobot advises 10 per object location, cameras fixed, grasp consistent. n ends an episode, r re-records, q stops and encodes.

   bash

   HF_USER=$(NO_COLOR=1 hf auth whoami | awk -F': *' 'NR==1 {print $2}')
   
   lerobot-record \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=my_follower \
       --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
       --teleop.type=so100_leader \
       --teleop.port=/dev/ttyACM1 \
       --teleop.id=my_leader \
       --dataset.repo_id=${HF_USER}/so100_cube \
       --dataset.num_episodes=50 \
       --dataset.single_task="Grab the black cube and put it in the bin" \
       --display_data=true

Before training, replay at least five episodes. The LeRobot dataset format stores camera streams, joint states and actions per episode, and replay pushes those actions back at the arm. If the replay does not do the task, the data does not contain it and training will not invent it.

lerobot-replay \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_follower \
    --dataset.repo_id=${HF_USER}/so100_cube \
    --dataset.episode=0

Training the ACT policy

This is the whole command. Everything ACT-specific is already a default, which is why the lerobot ACT page says to start with them.

lerobot-train \
  --dataset.repo_id=${HF_USER}/so100_cube \
  --policy.type=act \
  --output_dir=outputs/train/act_so100_cube \
  --job_name=act_so100_cube \
  --policy.device=cuda \
  --wandb.enable=true \
  --policy.repo_id=${HF_USER}/act_so100_cube

--policy.type=act loads ACTConfig, which adapts to however many motors and cameras your dataset recorded, so you never declare the observation shape. --wandb.enable=true is optional and worth it: the loss curve is the only cheap signal in a 100000 step run. The schedule comes from lerobot's train config, not ACTConfig: 100000 steps, batch 8, seed 1000, a checkpoint every 20000 steps, logging every 200.

A full run leaves five checkpoint directories, 020000 through 100000, plus a last symlink. Keep them all: the best policy is often not the last one.

Duration: around 5 hours on an 11 GB RTX 2080 Ti in the paper, a few hours for 100k steps per lerobot's ACT page, 2 to 5 hours on the AY-Robots 24 GB tier. Do not cut it short. The reference repo's README says a jerky or pausing policy usually just needs more training, because success and smoothness keep improving after the loss plateaus: for real-world data it wants at least 5000 epochs, or 3 to 4 times the length again after the plateau.

lerobot-train \
  --config_path=outputs/train/act_so100_cube/checkpoints/last/pretrained_model/train_config.json \
  --resume=true

Running the trained policy on the arm

Deployment uses lerobot-rollout. Camera keys must match the recorded ones: a policy trained on front and wrist will not accept cam0 and cam1, and rename_map does not help, since it needs a pretrained checkpoint. The task string can be omitted; lerobot's own example marks it skippable for ACT.

lerobot-rollout \
  --strategy.type=base \
  --policy.path=${HF_USER}/act_so100_cube \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
  --display_data=true \
  --duration=60

To pin a checkpoint rather than the final model, add --policy.pretrained_revision. That needs the run to have started with --save_checkpoint_to_hub=true, off by default: without it lerobot pushes the final model and nothing else. With it, each checkpoint is tagged with its zero-padded step, so --policy.pretrained_revision=060000 recovers the 60000 step one. Comparing it against 100000 on the real arm is the cheapest experiment available.

Two routes to the same checkpoint

Everything above is the manual route and it works. The platform route trades control for not owning a GPU or a Python environment.

# the two commands that matter, end to end
lerobot-record --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
  --teleop.type=so100_leader --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=${HF_USER}/so100_cube --dataset.num_episodes=50 \
  --dataset.single_task="Grab the black cube"

lerobot-train --dataset.repo_id=${HF_USER}/so100_cube --policy.type=act \
  --output_dir=outputs/train/act_so100_cube --policy.device=cuda

What actually goes wrong

Almost none of the pain is in the training command. It is in the things around it, ordered by how often they bite first time.

Two rows deserve emphasis. ACT trains on LeRobot v3.0 while GR00T's loader crashes on it and needs v2.1, so a dataset that trains ACT can fail a GR00T run. And the last row has two fixes: the ACT authors answer jerky motion with more training, while with n_action_steps at 100 a genuine stall lands at a chunk boundary, a visible pause every 3.3 seconds at 30 fps. Two more to know: a single dead joint is usually a servo id that was never written, and a gripper that approaches but never closes means too little gripper range in the demonstrations. Full index: the failure mode pages.

What a run costs

This is the argument for starting with ACT even if you want a VLA later. A failed ACT run costs the price of a coffee and tells you within hours whether your dataset contains the task. A failed GR00T run costs four times that for the same lesson. Moving up to GR00T N1.7 or SmolVLA afterwards is a form change, not a rebuild. Background: vision-language-action models, the complete SO-100 guide, train your first policy and imitation learning. No arm? The live page streams a real SO-100 to drive without signing up.