Before you press run

Keep a hand near the power

A policy will do unexpected things, especially the first one. Clear the workspace of anything you care about, keep your hand near the power switch, and never let the arm run unattended until you trust it.

Cameras in exactly the positions used during recording
Same lighting as your recording sessions
The scene reset the way you reset it while recording
Nothing fragile within reach of the arm

Run it

bash

ay-robots run \
  --model act \
  --checkpoint s3://ay-robots-runs/red-cube-pick-v1/step-40000 \
  --task "pick up the red cube and place it in the bowl" \
  --mode cloud

# When the attempt is over, or the moment it does something you do not like
ay-robots stop

The base model is named with --model, your own weights with --checkpoint, and the value for that is the checkpoint URI listed on the run page. A checkpoint in the cloud bucket needs --mode cloud, since local inference only loads a checkpoint that already sits on your machine and refuses an s3:// one outright. Either way the arm is driven from here: camera streams start and joint commands go out. The inference tab in the client is the meter on that loop rather than a view through the policy’s eyes. It counts predictions and motor commands, reports last, p50 and p99 latency, draws the last action it sent as one bar per joint, and prints the last error the model process reported, which is the fastest place to see that a run is up but not moving.

Evaluate honestly

Run twenty attempts with the object in different positions and write down successes. Twenty attempts, not the three that went well. A policy that succeeds eight times out of twenty is a real result you can improve on, and pretending it is fifteen helps nobody.

Success rate	Reading
Under 20%	Something structural is wrong. Check camera positions and the task string first.
20 to 50%	It learned something real. Usually a data quantity or consistency problem.
50 to 80%	Working. Targeted extra data on the failure cases will push it up.
Over 80%	Good result for a first policy on fifty episodes.

Diagnose the common failures

Symptom	Most likely cause
Arm moves confidently to the wrong place	Camera moved since recording. The policy is reading pixels.
Arm approaches correctly but never closes the gripper	Too few episodes captured the exact grasp moment, or the wrist view is occluded.
Arm freezes partway	The observation is outside anything in the training data.
Works from one position, fails everywhere else	You did not vary object placement enough while recording.
Jerky, oscillating motion	Jerky demonstrations. The policy learned exactly what you showed it.

Record for the failure, not in general

If it fails when the cube is on the left, record twenty more episodes with the cube on the left. Twenty targeted episodes beat two hundred more of what you already have.

The loop from here

1
Evaluate
Twenty attempts, written down, no rounding up.
2
Find the pattern
Failures cluster. Work out what the cluster has in common.
3
Record for that pattern
Targeted demonstrations of exactly the situation that fails.
4
Retrain and compare
Same evaluation protocol, so the numbers mean something.

You have a full pipeline

Arm, dataset, trained policy, evaluation on hardware. Everything after this is iteration, and iteration is a much easier problem than getting the first loop closed.

Was this page helpful?

Train your first policy