The same checkpoint that worked reliably in one session fails in the next, without anything being retrained in between.
A trained checkpoint is a fixed function of its observations, so if the output changed, the input did. The usual four are a moved camera, changed lighting, a changed table or object, and a recalibrated arm. Only the recalibration is repairable without new data, provided you can reproduce the original neutral pose.
The weights are the one thing that did not move
A checkpoint is deterministic in the sense that matters here: the same observation produces the same action today as it did yesterday. So a policy that has got worse overnight is not a policy that changed, it is a policy receiving different input. That narrows the search enormously, and it means the first hour should go into finding the difference rather than into a retraining run.
Work through the four candidates below before you touch anything. Restoring a setup takes minutes. Recording fifty new episodes takes an afternoon, and doing it while the real cause is a camera that got nudged means you record the new episodes through the nudged camera and bake the problem in.
| What changed | What it breaks | Cheapest fix | Needs new data? |
|---|---|---|---|
| A camera moved, or was unplugged and replugged | The pixel-to-world relationship the policy learned, and possibly which stream feeds which input | Restore the exact position and confirm the stream mapping | Only if you cannot restore it |
| Lighting changed | The appearance of every pixel the policy conditions on, including shadows it used as cues | Restore the lighting, or close the blinds and work under fixed light | Not immediately, but eventually yes |
| Table height, arm mount or the object changed | The geometry between the base frame and the workspace | Restore the setup and re-measure the mount position | Yes, if the change is permanent |
| The arm was recalibrated | The joint zero, so every recorded angle now refers to a different physical pose | Recalibrate back to the same neutral pose | No, if you can reproduce that pose |
| You moved to a second arm of the same model | Its own zero point and its own mechanical slop | Calibrate it to the same neutral pose and expect a drop anyway | Often yes |
The camera is the one people underestimate
The policy does not know where the cube is in the room. It knows where the cube is in the image, and it learned an action for that. Move the camera five centimeters and every pixel coordinate in the workspace now corresponds to a different physical location, while the policy continues to act as though it does not. The result is confident, smooth, consistently wrong motion, which is why this failure gets mistaken for a broken checkpoint.
There is a second, sneakier version. Cameras get enumerated in the order the operating system hands them over, so unplugging one, plugging in a webcam for a call, or booting with a different USB device attached can change which physical camera arrives on which named stream. The wrist view then feeds the input the policy expects the scene view on. Check the device list before you conclude anything about geometry.
{
"codebase_version": "v2.1",
"robot_type": "so100",
"fps": 30,
"features": {
"observation.state": { "dtype": "float32", "shape": [6] },
"action": { "dtype": "float32", "shape": [6] },
"observation.images.scene": { "dtype": "video", "shape": [480, 640, 3] },
"observation.images.wrist": { "dtype": "video", "shape": [480, 640, 3] }
}
}ay-robots devices
# SERIAL PORTS
# /dev/cu.usbmodem58FA0821911 SO-100 (6 servos responding)
# CAMERAS
# 0 USB 2.0 Camera 1920x1080 <- was 1 yesterday
# 1 FaceTime HD Camera 1280x720Tape on the desk around the tripod feet, and a note of the height. It is thirty seconds during a session that is going well, and it converts an unrecoverable change into a two-minute restoration later.
Recalibration is the one you can actually undo
Calibration defines what zero means for each joint. Every joint value in your dataset, and therefore every action the policy emits, is expressed relative to that zero. Capture a new zero at a slightly different neutral pose and the whole coordinate frame shifts underneath a policy that has no way of noticing. What you see is a policy that reaches consistently short, or consistently to one side, by roughly the same amount every attempt.
That consistency is the diagnostic. A calibration offset is a constant error: the same miss, in the same direction, at every start position. A data problem varies with the situation, succeeding in the middle of the workspace and failing at the edges. If you can describe the failure as a single vector, look at the calibration.
# Recapture the zero at the same physical neutral pose as the original
ay-robots calibrate
# Then confirm the arm reaches the pose you expect
ay-robots homeThe honest caveat is in the phrase "the same physical neutral pose". Reproducing it by hand is approximate. If your original session is recent and you remember how you set it, you will get close enough that the offset disappears into the noise. If you cannot reproduce it, some offset stays, and at that point a handful of fresh episodes under the current calibration is the cheaper route than fighting for the old zero. The procedure for both is in /learn/record-your-first-dataset.
Why a pretrained model survives more of this
ACT is trained from scratch on your episodes only. Its entire visual world is the frames you recorded, so a change in lighting or background is a change to the only thing it has ever seen. The catalog is direct about this: no pretraining, so it generalizes only as far as your demonstrations reach. That is the same property that makes it fast and data-efficient, and it is not a defect.
SmolVLA and the GR00T family arrive with visual pretraining behind them, and GR00T N1.7 in particular generalizes to object positions your demonstrations never showed. Practically, that buys tolerance for appearance changes: a different shade of light, a slightly different object, a background that is not quite yesterday. /compare/act-vs-smolvla is the version of this trade-off you are most likely to face on a first project.
No amount of pretraining tells a model where you moved the camera. Appearance robustness and spatial calibration are separate things, and every model on the platform fails on a moved camera. The difference between them is how much lighting and object variation they absorb before that fails too.
A ten minute recovery procedure
- 1Change nothing and write down the differences
Time of day, who else used the desk, whether anything was unplugged, whether the arm was moved or recalibrated. Two minutes of recall usually names the cause before any test does.
- 2Check the device list
Confirm the arm is on its port with all actuators answering, and that the cameras appear in the same order and at the same resolutions as before.
bashay-robots devices - 3Verify the calibration zero
Home the arm and look at whether the neutral pose is where it used to be. A visibly different home pose is a calibration change and explains a constant offset on its own.
bashay-robots home - 4Restore the camera and the light before you touch the model
Put the tripod back on its tape marks and reproduce the lighting. This is the step with the best ratio of effort to result, and it is the one people skip.
- 5Re-run the same twenty attempts
Same start positions and same reset as your original evaluation, otherwise you cannot tell recovery from a lucky session. If the number comes back to where it was, you found it.
- 6If the setup is genuinely restored and it is still down, record for the new condition
At that point the environment has changed in a way you cannot undo, and twenty targeted episodes under the new condition beat a retrain on the old ones.
Putting a camera back where it was takes minutes and costs nothing. Recording and retraining takes an afternoon and a GPU rental. Exhaust the first before you commit to the second, and mark the setup while it works so that restoring stays possible.
Frequently asked questions
How precisely does the camera have to go back?▾
Closer than you would guess. There is no threshold to quote, because it depends on how far the arm has to travel and how tight the grasp is, but a policy trained on a fixed camera treats the view as ground truth. Tape marks and a recorded tripod height are worth more than any tolerance number.
Can I make a policy robust to camera position by training on several positions?▾
Yes, and it is the durable answer. Record episodes at each camera position you intend to use, in comparable numbers. The cost is that the same total episode budget now covers several viewpoints instead of one, so plan for more recording rather than the same amount split thinner.
The arm reaches two centimeters short of everything. Recalibrate or re-record?▾
Recalibrate first. A constant offset in one direction is the signature of a shifted zero, and it is the one failure on this page you can fix without new data. Re-record only if you cannot get back to the original neutral pose.
Does daylight really matter that much?▾
It matters more than most people expect, because it moves during the session as well as between sessions. A policy recorded at midday and evaluated at dusk is looking at different shadows on the same table. Working under fixed artificial light with the blinds closed removes an entire class of confusion.
Is a second arm of the same model a drop-in replacement?▾
Not quite. Two SO-100 arms share a design, not a calibration or a mechanical history, so a policy moved between them starts with an offset and whatever backlash differs. Calibrate the new arm to the same neutral pose, evaluate it with the same twenty attempts, and expect to add episodes if the gap is large.
A symptom in one part of the chain is regularly produced by the part before it. These pages cover the neighboring areas.