AY-ROBOTSAI Teleoperation
Sign Up

Checkpoint

  • model checkpoint
  • saved weights
  • ckpt
  • saveSteps
Definition

A checkpoint is the model’s weights written to storage at a particular training step, so one run leaves several of them behind rather than a single final model. The last checkpoint is not automatically the best one: which one wins is settled by attempts on the arm, not by the loss curve.

Last updated 2026-08-09

Why a run produces several

Training here is step-based and the trainer writes weights at intervals along the schedule. That is insurance against a run that ends badly, and it is the only way to compare the model at different points in its own training. A checkpoint you delete cannot be regenerated without paying for the run again.

ModelDefault stepsHow checkpoint spacing is controlled
GR00T N1.720,000saveSteps, the one extra field GR00T exposes
GR00T N1.52,000saveSteps
Pi0.530,000No saveSteps field. You take what the trainer writes
SmolVLA20,000No saveSteps field. The form offers seed and logFreq instead
ACT100,000No saveSteps field. chunkSize, nActionSteps, seed and logFreq instead

Choosing one

  1. Keep two or three from the second half of the run. Earlier ones are usually undertrained and the final one has no special status.
  2. Evaluate each on the arm with the same protocol: same reset, same lighting, same number of attempts, results written down as you go.
  3. Compare success rates, not losses. A lower training loss and a higher success rate on hardware are two different questions, and the second one is the one you care about.
  4. Record the winning step number alongside the job id and the dataset. That triple is what makes a result reproducible three months later.
There is no CLI command for training or for listing checkpoints

Training runs on a rented cloud GPU, started in the dashboard under Training or through the start_training tool on the MCP server, and the run page lists the checkpoints it produced, each as a URI you can paste. What the CLI does is run one. Without a checkpoint the run command executes base weights, which is a useful smoke test before your own policy exists. ACT is the exception: it has no base model and only ever exists as a checkpoint trained on your own task, so a run without one is refused.

bash
# Run one specific checkpoint against the task it was trained on.
# It sits in the cloud bucket, so the GPU pod is what loads it.
ay-robots run \
  --model smolvla \
  --checkpoint s3://ay-robots-runs/cube-to-bowl-smolvla/step-20000 \
  --task "pick up the red cube and place it in the bowl" \
  --mode cloud

# Stop it
ay-robots stop
A checkpoint is a Hugging Face repo id or an s3:// URI. Local mode refuses an s3:// checkpoint outright, so a checkpoint from a cloud run needs cloud mode.

/learn/train-your-first-policy covers loss curves and what they do and do not tell you, and /train has the per-model walkthroughs that end in a checkpoint on real hardware.

Common problems

Where this term does the explaining: symptoms whose cause sits in exactly what it describes.

Hardware
Why does everything move except the gripper?
Every joint responds and the arm moves normally, but the gripper does not open or close. It may also be loose enough to push by hand while the other five joints hold their positions.
Training
The loss went down and the policy still does nothing
Training finished, the loss curve fell smoothly and flattened out, and the policy fails every attempt on the arm. Nothing in the run page looks wrong.
Deployment
It worked yesterday and fails today
The same checkpoint that worked reliably in one session fails in the next, without anything being retrained in between.
Deployment
The arm starts, then stops partway through the task
The policy starts confidently, gets partway into the task, and then the arm stops moving. Sometimes it holds position, sometimes it sags.