How to train ACT on the SO-100

What ACT actually is

ACT is the Action Chunking Transformer from the Stanford ALOHA work, about 80 million parameters, and it is trained from scratch on your dataset. There is no pretrained checkpoint underneath it. It has never seen a robot before it sees yours, and it never reads the task string, because it has no language conditioning at all. The inputs are camera frames plus the current joint positions; the output is joint targets.

That has one blunt consequence: one trained ACT is one task. If you want the SO-100 to pick up a cube and also to press a button, you train two policies and switch between them. What you get for accepting that limit is speed. At about 20 ms per action step ACT is the fastest model the platform trains, roughly seven times quicker than GR00T N1.7 and more than twenty times quicker than Pi0.5, and the difference is visible in the room: the arm moves instead of stepping between decisions.

Action chunking, the mechanism that makes it work

A plain behavior-cloning policy predicts one action per observation. Each prediction has a small error, that error puts the arm in a slightly different state than any demonstration ever visited, and the next prediction is made from that slightly wrong state. Over a few hundred steps the drift feeds itself. This is the compounding-error problem, and it is why naive behavior cloning falls apart on long motions even when the training loss looks fine.

ACT predicts a whole sequence of future actions in a single forward pass. That sequence is the chunk. The runtime then executes some number of actions from it before throwing the rest away and predicting again from a fresh observation.

Two things follow. First, the number of decision points across an episode drops by however many actions you execute per chunk, and every decision point is an opportunity to compound error, so fewer of them means less drift. Second, the actions inside one chunk were planned together, so they agree with each other. Consecutive commands from independent forward passes disagree slightly and the arm shows it as high-frequency correction. Commands from one plan do not, which is the practical reason ACT motion looks fluid.

chunkSize and nActionSteps

The training form exposes both of these for ACT and for no other model. chunkSize is how many future actions the model is trained to predict per pass. nActionSteps is how many of those the runtime actually executes before it replans. In lerobot both default to 100, and nActionSteps must be less than or equal to chunkSize, because you cannot execute actions the model was never asked to produce. With both at 100 and a 30 fps control loop, one prediction carries a little over three seconds of motion, entirely open loop.

Change one of the two at a time and evaluate between changes. Lowering nActionSteps costs you nothing in training, it is a runtime decision, so it is the cheap knob. Raising chunkSize is not cheap: it asks the model to predict further into the future from the same single observation, which only works if your demonstrations are consistent that far ahead. Lowering both together turns ACT back into a per-step policy and throws away the reason you picked it.

Before you start

• An assembled and calibrated SO-100 with Feetech STS3215 servos, six actuators in total
• The desktop client installed and paired with your account, which is where the driver lives, so there is no Python environment to build
• At least one camera, mounted so it cannot be nudged. A wrist camera in addition to the scene camera is worth more than any hyperparameter on this page
• At least 50 recorded episodes of a single task, in LeRobot v2.1
• A funded account, because the training run rents a GPU by the hour

curl -fsSL https://www.ay-robots.com/install.sh | sh
ay-robots --version

# Confirm the arm and the cameras are seen before you record anything
ay-robots devices

# Calibration teaches the software where each joint physically sits.
# Skip it and every recorded joint value means something else on your arm.
ay-robots calibrate

The dataset decides the outcome

Because ACT is trained from scratch, every regularity it exploits has to be present in your episodes. A pretrained vision-language-action model already knows roughly what a grasp looks like, so it can absorb a certain amount of inconsistency in your demonstrations. ACT cannot. Demonstration consistency matters more for this model than for anything else the platform trains, and it matters far more than episode count.

1. One task per dataset. ACT ignores the task string entirely, so two tasks in one dataset are not two tasks to it, they are contradictory demonstrations of one task.
2. One strategy per task. Approach from the same side, grasp at the same height, lift before you translate. If you changed your mind about technique halfway through the session, you now own two half-datasets.
3. Vary the object placement, never the technique. Placement variation is what buys you generalization inside the workspace. Technique variation is what buys you an averaged, useless policy.
4. Discard every failed attempt. A missed grasp in the training set is a demonstration of missing, and the model has no prior knowledge telling it that was not the point.
5. Do not touch the cameras. The policy is reading pixels, so a nudged tripod invalidates every episode recorded before the nudge.
6. 50 episodes is the working floor for ACT. 100 to 200 is what covers a table rather than a spot on a table.

Start the training run

1. 1
   Upload the dataset and look at it

   Open Datasets, upload the recording, and wait for it to finish processing. Then play back two or three episodes in the preview. Thirty seconds of watching your own data catches problems that no loss curve will ever tell you about.

2. 2
   Open Training and select the dataset

   The training form pulls the robot type and the feature schema from the dataset, so what you pick here determines what the trainer sees. An SO-100 dataset gives it six action dimensions and however many camera streams you recorded.

3. 3
   Choose ACT as the model

   The form switches to the lerobot knob set: batch size, learning rate, steps, gradient accumulation, seed and log frequency. Chunk size and nActionSteps appear too, and only for ACT.

4. 4
   Leave the defaults alone on the first run

   Batch 8, learning rate 1e-5, 100 000 steps, gradient accumulation 1, chunk size 100, nActionSteps 100. Do set a seed, though. A first run without one cannot be compared against anything you do next.

5. 5
   Name the policy and press Start

   The pool rents a card on the spot market, selected by VRAM rather than by GPU name, so ACT lands on whatever 24 GB card is cheapest at that moment. The job sits in queued for a minute or two, then flips to running.

6. 6
   Watch the run page, not the loss alone

   Live loss, learning rate, gradient norm and GPU telemetry are all on the run page, along with the checkpoints as they are written. Keep two or three checkpoints from the second half of the run; the last one is not automatically the best one.

The same run can be started by an agent over the hosted MCP server, which is Streamable HTTP with JSON-RPC 2.0 at https://www.ay-robots.com/api/mcp, protocol version 2025-06-18. Training tools need an API key as a Bearer token, and start_training refuses to do anything unless confirm_spend is true, because it rents hardware.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "start_training",
    "arguments": {
      "dataset_id": "ds_9c41e7",
      "policy_name": "so100-red-cube-act",
      "model_id": "act",
      "max_steps": 100000,
      "confirm_spend": true
    }
  }
}

The defaults, and when to change them

What the run costs and how long it takes

ACT trains on the 24 GB tier, the same tier as SmolVLA and a fraction of the price of the A100 tier that GR00T and Pi0.5 require. Expect 2 to 5 hours of wall clock for a full 100 000 step run on a 50 episode SO-100 dataset, at 0.30 to 0.60 USD per hour, so about 1 to 3 USD in total. The spread is real: the pool buys on a spot market and the hourly rate moves day to day.

This is why the long schedule is not the drawback it looks like. ACT needs by far the most steps of any supported model, five times as many as GR00T N1.7 by default, and it is still the cheapest run on the platform because those steps are small and the card is cheap. A second run with a different seed or a shorter chunk is a decision you make in a second, not a budget conversation.

Run it on the SO-100

Inference runs on your own machine. The client loads the checkpoint, opens the camera streams and drives the arm directly, so nothing is round-tripping to a server while the gripper is closing. At roughly 20 ms per action step the model fits comfortably inside the 33 ms budget of a 30 fps control loop even on a laptop, and with chunking it is not even asked to run every frame.

1. 1
   Restore the scene exactly

   Cameras in the recorded positions, the same lighting you recorded under, the arm at the same home pose. The policy is matching pixels, so the scene is part of the input.

2. 2
   Load a mid-run checkpoint first

   Not the last one. Take one from around 60 to 80 percent of the run and one from the end, so you have a comparison rather than a single data point.

3. 3
   Run twenty attempts and write each one down

   Twenty, with the object in different positions across the region you demonstrated, and count the failures honestly. Eight out of twenty is a real result you can improve. A remembered "it mostly works" is not a result at all.

4. 4
   Try the other checkpoint under the identical protocol

   Same positions, same order, same number of attempts. Checkpoint choice is worth several percentage points and costs nothing but the time to swap it.

5. 5
   Cluster the failures before you change anything

   Failures group. If nineteen of twenty succeed unless the cube is at the left edge, the fix is twenty new episodes at the left edge, not a different learning rate.

Troubleshooting

When ACT is the right tool, and when it is not

The honest summary is that ACT trades generalization for speed and cost, and on a repetitive tabletop task that is a good trade. The model page at /policies/act carries the same numbers in shorter form, /compare/act-vs-smolvla is the head-to-head against the other 24 GB option, and if your episodes are not yet recorded, start at /learn/record-your-first-dataset instead of here.