The loss went down and the policy still does nothing

What the number actually measures

Loss is the average distance between the action the model predicts for an observation and the action you performed at that same observation while recording. It is computed entirely against data you produced. It has no term for whether the object ended up in the bowl, no term for contact forces, and no way of knowing that a two-centimeter error at the approach becomes a missed grasp.

There is a second gap that matters more than people expect. During training the model always sees observations that came from your teleoperation. During deployment it sees observations produced by its own previous actions. A small error moves the arm somewhere slightly off the demonstrated path, the next observation is therefore slightly outside anything in the training set, and the next error is larger. Loss is measured on the first kind of observation and the arm operates in the second.

// MCP get_training_job, trimmed
{
  "id": "job_31b7ce",
  "modelType": "act",
  "status": "SUCCEEDED",
  "configJson": {
    "batch_size": 8,
    "gradient_accumulation_steps": 1,
    "learning_rate": 0.00001,
    "steps": 100000,
    "chunk_size": 100,
    "n_action_steps": 100
  },
  "errorMessage": null
}

Cause one: two strategies averaged into one

This is the single most common reason a well-fitted policy does nothing. Suppose half your episodes approach the cube from the left and half from the right. Both are correct demonstrations. Both succeed when you drive them. The model sees the same observation labeled with two contradictory actions, and the fit that minimizes average distance to both is the one in the middle, which drives the gripper straight into the cube.

The signature is confidence. The arm does not hesitate, does not search, does not oscillate. It moves smoothly and decisively to a position that makes no sense, and it does the same thing every attempt. If you find yourself thinking that the policy looks sure of itself, that is the diagnosis, not a figure of speech.

Model choice changes how badly this bites. ACT regresses actions directly, so an averaged mode is exactly what you get, and /policies/act is honest that it learns your demonstrations and nothing beyond them. GR00T N1.7 uses a diffusion action head and Pi0.5 uses flow matching, and both are built to represent a distribution over actions rather than one mean. That helps with genuinely multi-modal tasks. It does not rescue a dataset where you changed your mind about the approach halfway through the recording session.

Cause two: fewer episodes than the model needs to say anything

Every model in the catalog has a floor below which a failed evaluation carries no information. Below the floor, the correct reading of a bad result is that you do not have enough data yet, and no learning rate, step count or architecture change addresses that.

These are floors and not targets. Fifty clean episodes of one narrow task with a fixed camera is a starting point, not a finished dataset. If your fifty episodes also contain three failed grasps you decided to keep, treat the effective count as lower than fifty and the strategy count as higher than one.

Build an evaluation that can be wrong

You cannot debug this from the chart, so the chart has to be replaced by a measurement on hardware. The protocol below is not elaborate, and its only real requirement is that it stays identical between runs, because the whole point is comparing two numbers that were produced the same way.

1. 1
   Write down the reset procedure

   Home pose, object placement, lighting, camera positions, what you do between attempts. Write it on paper next to the arm. An evaluation whose reset drifts across an hour measures your patience.

2. 2
   Choose the twenty start positions before you run anything

   Spread them across the region your demonstrations covered. Fixing them in advance is the part that matters, because a protocol invented attempt by attempt drifts toward positions that have been working.

3. 3
   Run all twenty, scoring by outcome only

   Success or failure, one line each, plus two words on where it broke. Do not restart an attempt because it looked like it was heading for a failure. That is the failure, and it counts.

   bash

   ay-robots run \
     --model act \
     --checkpoint your-account/so100-cube-act-v1 \
     --task "pick up the red cube and place it in the bowl"

4. 4
   Look at where the failures cluster

   Twenty scored attempts almost always fall into a pattern. Failures spread evenly across the workspace mean something different from failures that all happen on the left.

5. 5
   Re-run the same twenty for every candidate

   Two or three checkpoints from the second half of the run, and later the retrained model. The last checkpoint is not automatically the best, and the only way to know is the same protocol on each.

What to change, in order

1. Consistency of the demonstrations you already have. Re-watch ten episodes and check whether you did the task the same way in all ten.
2. Episode count, targeted at the cluster of failures rather than spread evenly.
3. Model, but only after the data is consistent. A pretrained model on inconsistent data is a more expensive version of the same problem.
4. Hyperparameters, last and rarely. The defaults in the catalog are the ones the trainer was tuned around.

The full loop, including how to reset and score, is written out at /learn/run-your-first-policy. If your current run was a foundation model on a first dataset, /train/smolvla-on-so-100 is the cheaper iteration path while the data question is still open.

Frequently asked questions