Before you press run
A policy will do unexpected things, especially the first one. Clear the workspace of anything you care about, keep your hand near the power switch, and never let the arm run unattended until you trust it.
- Cameras in exactly the positions used during recording
- Same lighting as your recording sessions
- The scene reset the way you reset it while recording
- Nothing fragile within reach of the arm
Run it
ay-robots run \
--checkpoint <job-id>:40000 \
--task "pick up the red cube and place it in the bowl"The client loads the checkpoint, starts the camera streams and begins issuing joint commands. The inference tab shows what the policy sees and what it is about to do, one step ahead, which is the most useful debugging view you have.
Evaluate honestly
Run twenty attempts with the object in different positions and write down successes. Twenty attempts, not the three that went well. A policy that succeeds eight times out of twenty is a real result you can improve on, and pretending it is fifteen helps nobody.
| Success rate | Reading |
|---|---|
| Under 20% | Something structural is wrong. Check camera positions and the task string first. |
| 20 to 50% | It learned something real. Usually a data quantity or consistency problem. |
| 50 to 80% | Working. Targeted extra data on the failure cases will push it up. |
| Over 80% | Good result for a first policy on fifty episodes. |
Diagnose the common failures
| Symptom | Most likely cause |
|---|---|
| Arm moves confidently to the wrong place | Camera moved since recording. The policy is reading pixels. |
| Arm approaches correctly but never closes the gripper | Too few episodes captured the exact grasp moment, or the wrist view is occluded. |
| Arm freezes partway | The observation is outside anything in the training data. |
| Works from one position, fails everywhere else | You did not vary object placement enough while recording. |
| Jerky, oscillating motion | Jerky demonstrations. The policy learned exactly what you showed it. |
If it fails when the cube is on the left, record twenty more episodes with the cube on the left. Twenty targeted episodes beat two hundred more of what you already have.
The loop from here
- 1Evaluate
Twenty attempts, written down, no rounding up.
- 2Find the pattern
Failures cluster. Work out what the cluster has in common.
- 3Record for that pattern
Targeted demonstrations of exactly the situation that fails.
- 4Retrain and compare
Same evaluation protocol, so the numbers mean something.
Arm, dataset, trained policy, evaluation on hardware. Everything after this is iteration, and iteration is a much easier problem than getting the first loop closed.