The idea in one paragraph
Instead of writing inverse kinematics and a state machine for a task, you teleoperate the arm through the task a few dozen times while recording what the cameras saw and what the joints did. A neural network then learns the mapping from observation to action. At inference time it sees a fresh camera image and outputs the next joint command.
The SO-100 has no force-torque sensing and modest repeatability. Classical approaches lean on precise models of the arm and the world, which is exactly what this hardware cannot give you. Learning from demonstrations sidesteps the model entirely.
What it is good at, and what it is not
| Works well | Struggles |
|---|---|
| Repetitive tabletop manipulation | Tasks needing millimetre precision |
| Tasks that are easy to demonstrate but tedious to code | Long-horizon tasks with many sub-goals |
| Deformable or irregular objects | Situations far outside the demonstrations |
| Grasping where exact geometry is unknown | Anything requiring force feedback the arm cannot sense |
Choosing a policy architecture
You do not need to pick perfectly on the first try. Start simple, get an end-to-end result, then move up if the ceiling is too low.
| Policy | Data appetite | Good first choice when |
|---|---|---|
| ACT | Low, 50+ episodes | You want a working result today on one task |
| Diffusion Policy | Medium, 100+ episodes | The task has several valid ways to succeed |
| SmolVLA | Medium | You want language conditioning without a large GPU |
| GR00T (fine-tuned) | Higher | You want generalisation and have the GPU budget |
A fine-tuned foundation model on 30 inconsistent episodes will perform worse than ACT on the same data, and it will take far longer to tell you so. Prove the data is good with a small policy first.
The full loop
- 1Record
Fifty or more consistent demonstrations of one task, with varied object placement.
- 2Review
Discard failed and inconsistent episodes. This step has more effect on the outcome than any hyperparameter.
- 3Train
Start a training job. Watch the loss curve, but treat it as a smoke signal rather than a measure of success.
- 4Evaluate on hardware
Run the policy on the real arm. Success rate on physical attempts is the only metric that counts.
- 5Diagnose and extend
Where it fails, record more demonstrations of exactly that situation. Targeted data beats more data.
A policy can have a beautiful loss curve and still fail every physical attempt, usually because it learned the average of several incompatible strategies. Always evaluate on the arm.
Frequently asked questions
Do I need my own GPU?▾
Not for training on this platform, that runs in the cloud. Inference on the arm runs on your own machine, and for ACT-sized policies a normal laptop keeps up.
How long does training take?▾
An ACT policy on 50 episodes is typically well under an hour. Fine-tuning a foundation model on a few hundred episodes is a matter of hours, not minutes.
Will a policy trained on my arm work on someone else’s SO-100?▾
Partly, and better if both arms are calibrated. Camera placement matters more than the arm itself, since the policy is largely reading pixels.
Can I combine imitation learning with scripted motion?▾
Yes, and it is often the pragmatic answer. Script the parts that are trivially reliable, such as the approach to a fixed home position, and learn the parts that are not, such as the grasp.