The AY-Robots training documentation page, describing the run that writes the checkpoint directory this article takes apart
CheckpointsLeRobotGR00TsafetensorsFine-TuningDebugging

What a Robot Policy Checkpoint Contains, and Why It Won't Move

AY-Robots ResearchAugust 23, 202621 min read

A checkpoint is four artifacts pretending to be one. Real file listings from LeRobot, GR00T and Pi0.5 checkpoints, and the eight reasons copying one between machines fails.

A checkpoint is not a model. It is a directory that happens to contain a model, plus three other things that are just as load bearing: the numbers used to scale your robot's joint angles into the range the network was trained on, a config that describes the architecture the weights belong to, and a pile of training state that only matters if you plan to resume. Get any of the four wrong and the arm still moves. It just moves to the wrong place.

This page opens real checkpoints and reads out what is in them. Every file name, byte count and tensor key below was pulled from a public repository and re-checked against it in August 2026, not recalled from memory. The three examples are a SmolVLA base checkpoint, an ACT checkpoint from the LeRobot model zoo, and NVIDIA's GR00T N1.7 base. Then we go through why copying one of these from the machine that trained it to the machine that runs the arm fails more often than it works.

What you need to know

  • A checkpoint has four parts: weights, normalisation statistics, config, and training state. Only the first three affect what the arm does at inference time.
  • The normalisation statistics are tiny. In lerobot/smolvla_base they are a 640 byte file next to a 907 MB weight file, and they decide where in space the joint commands land.
  • LeRobot loads weights with strict=False. A missing or mismatched tensor key produces a log warning, not an exception, so a half-loaded policy runs and looks plausible.
  • Configs carry the device of the machine that wrote them. lerobot/pi05_base ships with "device": "mps" in config.json, so a checkpoint straight off the Hub can name a backend your machine does not have.
  • GR00T normalises with the 1st and 99th percentile (use_percentiles true, use_mean_std false in both config.json and processor_config.json). LeRobot policies default to mean and standard deviation. The two are not interchangeable.
  • Training state (optimizer moments, RNG, scheduler) is often larger than the weights and is useless at inference. Strip it before you copy a checkpoint to a robot.

The four parts of a checkpoint

Whatever framework wrote it, a policy checkpoint decomposes into the same four categories. Knowing which category a file belongs to tells you whether you can delete it, whether it has to travel with the weights, and what breaks if it is stale.

PartTypical filesNeeded at inference?Needed to resume training?What goes wrong if it is wrong
Weightsmodel.safetensors, model-0000N-of-0000M.safetensors, model.safetensors.index.jsonYesYesLoad fails loudly, or silently loads a subset and the arm drifts
Normalisation statisticspolicy_preprocessor_step_*_normalizer_processor.safetensors, statistics.json, experiment_cfg/dataset_statistics.jsonYesYesArm moves smoothly to the wrong coordinates. No error anywhere
Configconfig.json, train_config.json, processor_config.json, policy_preprocessor.jsonYesYesWrong architecture is built, shapes do not match, or the wrong device is selected
Training statetraining_state/optimizer_state.safetensors, rng_state.safetensors, scheduler_state.json, training_step.json, optimizer.ptNoYesResume restarts the optimizer cold and the loss curve jumps
Why the split matters for shipping

The optimizer state for an Adam-family optimizer holds two moment tensors per trainable parameter, so it is roughly twice the size of the trainable weights. For a full fine-tune that means the resume artifacts dominate the directory. When you copy a checkpoint to the machine next to the arm, you want the pretrained_model/ subtree and nothing else.

Case 1: a LeRobot checkpoint on disk

LeRobot writes one directory per saved training step, zero padded to at least six digits, under checkpoints/ in the run output directory, plus a last symlink pointing at the newest one. The layout is documented in the docstring of save_checkpoint. It changed between releases, so the version matters.

text
005000/                              # training step at checkpoint
|-- pretrained_model/
|   |-- config.json                   # policy config
|   |-- model.safetensors             # policy weights
|   |-- train_config.json             # train config
|   |-- processor.json                # processor config (if a preprocessor was passed)
|   `-- step_*.safetensors            # processor state files (if any)
`-- training_state/
    |-- optimizer_param_groups.json
    |-- optimizer_state.safetensors
    |-- rng_state.safetensors
    |-- scheduler_state.json
    `-- training_step.json
lerobot v0.5.1 (7 April 2026), from the save_checkpoint docstring in src/lerobot/utils/train_utils.py. The current release v0.6.1 (3 August 2026) still has this docstring character for character, at src/lerobot/common/train_utils.py.
text
005000/
|-- pretrained_model/
|   |-- config.json
|   |-- model.safetensors             # or pytorch_model_fsdp_0/ for DCP formats
|   |-- train_config.json
|   |-- policy_preprocessor.json
|   |-- policy_preprocessor_step_*.safetensors
|   |-- policy_postprocessor.json
|   `-- policy_postprocessor_step_*.safetensors
`-- training_state/
    |-- optimizer_param_groups.json
    |-- optimizer_state.safetensors   # or optimizer_0/ for sharded runs
    |-- rng_state.safetensors
    |-- scheduler_state.json
    `-- training_step.json            # step + dp_world_size, batch_size, grad_accum, topology
lerobot main branch as of 23 August 2026, from src/lerobot/common/train_utils.py. This is ahead of the released v0.6.1, not what pip gives you.

Three things moved between the release you install and the branch it will become, and one thing that looks like it moved did not. The module changed path in v0.6.0, which is enough to break any script that imported it by path. training_step.json grew from a bare step counter into a snapshot of the run topology, so a resume can complain when the parallelism or the gradient-accumulation cadence changed. And main added artifacts for sharded runs that no release writes yet.

Aspectv0.5.1 (7 Apr 2026)v0.6.1 (3 Aug 2026, current release)main (23 Aug 2026)
Module pathsrc/lerobot/utils/train_utils.pysrc/lerobot/common/train_utils.pysrc/lerobot/common/train_utils.py
save_checkpoint docstringprocessor.json plus step_*.safetensorsidentical to v0.5.1policy_preprocessor.json, policy_postprocessor.json and their step files written out in full
training_step.json{"step": step} and nothing elsestep, num_processes, batch_sizestep, dp_world_size, batch_size, grad_accum_steps
Sharded model and optimizernot writtennot writtenpytorch_model_fsdp_0/ and optimizer_0/ under the DCP checkpoint formats
Processor state file naming{pipeline name}_step_{index}_{registry name}.safetensorssamesame

The last row is the one worth dwelling on, because it looks like a breaking rename and is not one. PolicyProcessorPipeline has built state file names out of the pipeline name, the step index and the registry name since v0.5.1, and v0.5.1's save_checkpoint already accepted both a preprocessor and a postprocessor. Only the docstring changed, from generic placeholders to the concrete names the released code was writing all along. That is why lerobot/smolvla_base on the Hub carries a file called policy_preprocessor_step_5_normalizer_processor.safetensors even though no tagged release names that file anywhere.

Old checkpoints do not have separate normalisation files

Before the processor pipeline existed, LeRobot kept the normalisation buffers inside the weight file itself, under keys like normalize_inputs.buffer_observation_state.mean. Those checkpoints still exist on the Hub; lerobot/act_aloha_sim_transfer_cube_human is one of them. If you load one with current lerobot, the normalisation keys are unexpected and the processor gets no statistics at all. lerobot ships src/lerobot/processor/migrate_policy_normalization.py to lift the buffers out of the state dict into a preprocessor and a postprocessor, and its documented usage example targets that exact repository. Run it once; do not hand-patch the state dict.

What the real repositories contain

These are byte counts from the Hugging Face API, not estimates. They show how lopsided a checkpoint is: the part that is easiest to lose is also the smallest.

RepositoryFileBytes
lerobot/smolvla_basemodel.safetensors906,712,520
lerobot/smolvla_baseconfig.json2,299
lerobot/smolvla_basepolicy_preprocessor.json1,872
lerobot/smolvla_basepolicy_preprocessor_step_5_normalizer_processor.safetensors640
lerobot/smolvla_basepolicy_postprocessor_step_0_unnormalizer_processor.safetensors640
lerobot/pi05_basemodel.safetensors14,467,165,872
lerobot/pi05_baseconfig.json1,896
lerobot/pi05_basepolicy_preprocessor.json1,241
lerobot/pi05_basepolicy_postprocessor.json567
lerobot/act_aloha_sim_transfer_cube_humanmodel.safetensors206,766,560
lerobot/act_aloha_sim_transfer_cube_humantrain_config.json5,290

Note what lerobot/pi05_base does not have: it ships policy_preprocessor.json and policy_postprocessor.json but no normalizer state file. The base Pi0.5 checkpoint carries the pipeline structure without the statistics, because the statistics only exist once you have picked a dataset. lerobot/act_aloha_sim_transfer_cube_human has no processor files at all, because it predates them.

What is actually inside model.safetensors

safetensors is deliberately boring, which is the point. The file is eight bytes holding an unsigned little-endian 64-bit integer N, then N bytes of UTF-8 JSON, then the raw tensor bytes. The JSON header maps each tensor name to its dtype, its shape and a pair of byte offsets into the data region. There is one reserved key, __metadata__, and it may only hold a flat string-to-string map. No pickle, no code execution, no surprises when you open a file someone sent you.

That means you can read the entire structure of a 14 GB weight file by fetching its first few hundred kilobytes. Here is the whole reader, and the output from the ACT checkpoint above.

python
import json, struct, sys

with open(sys.argv[1], "rb") as f:
    n = struct.unpack("<Q", f.read(8))[0]     # header length
    header = json.loads(f.read(n))            # {name: {dtype, shape, data_offsets}}

meta = header.pop("__metadata__", None)
print(f"header bytes: {n}   tensors: {len(header)}   metadata: {meta}")
for name, spec in header.items():
    print(f"{name:60s} {spec['dtype']:5s} {spec['shape']}")
Dump the header of any safetensors file without loading the tensors
text
normalize_inputs.buffer_observation_images_top.mean   F32 [3, 1, 1]
normalize_inputs.buffer_observation_images_top.std    F32 [3, 1, 1]
normalize_inputs.buffer_observation_state.mean        F32 [14]
normalize_inputs.buffer_observation_state.std         F32 [14]
normalize_targets.buffer_action.mean                  F32 [14]
normalize_targets.buffer_action.std                   F32 [14]
unnormalize_outputs.buffer_action.mean                F32 [14]
unnormalize_outputs.buffer_action.std                 F32 [14]
Normalisation buffers inside lerobot/act_aloha_sim_transfer_cube_human/model.safetensors (242 tensors total, 234 of them under the model.* prefix)

The [14] is a bimanual ALOHA setup: seven values per side. An SO-100 records six, five joints plus the gripper, which is exactly the shape lerobot/smolvla_base declares for observation.state and action. Those two shapes will never load into each other, which is the good case. The dangerous case is two checkpoints that agree on shape and disagree on what the numbers mean.

The 640 bytes that decide whether the arm works

Here is the entire normaliser state of lerobot/smolvla_base, decoded. Six tensors, 144 bytes of payload, 488 bytes of JSON header, 640 bytes on disk. It sits next to a 907 MB weight file.

text
so100-blue.buffer.action.mean   [   1.4212, 125.7205, 125.4193,  63.7832, -106.5401,   3.3485]
so100-blue.buffer.action.std    [  14.0820,  39.4314,  18.2381,  23.4898,   38.1112,   5.0483]
so100-red.buffer.action.mean    [   2.4387, 124.9379, 123.5578,  66.9926, -103.8768,   2.8912]
so100-red.buffer.action.std     [  14.2992,  42.7722,  18.8763,  21.6519,   33.5341,   4.4780]
so100.buffer.action.mean        [   1.5959, 119.9442, 109.7698,  56.7055,  -27.4227,  12.0028]
so100.buffer.action.std         [  26.3918,  52.4114,  49.8538,  36.9983,   59.3600,  19.0400]
policy_preprocessor_step_5_normalizer_processor.safetensors from lerobot/smolvla_base, decoded 23 August 2026. The values are raw joint readings in whatever unit the source recordings used, not a normalised range.

Look at the fifth column, the wrist. The so100-blue statistics put its mean at -106.54; the plain so100 statistics put it at -27.42. Mean-and-standard-deviation normalisation means the model emits a normalised value and the postprocessor computes value times std plus mean, so a predicted normalised zero comes out at -106.54 under one set of statistics and at -27.42 under the other. The gap is 79.1 raw units: 2.1 standard deviations under the so100-blue statistics, 1.3 under the plain so100 ones. Whatever unit that arm recorded in, moving a wrist joint by more than a standard deviation is the difference between closing on an object and swiping past it, and it comes out of a 640 byte file with no error message anywhere in the stack.

The AY-Robots glossary entry for the LeRobot dataset format, showing the directory layout and the meta files that hold the statistics
The statistics in a checkpoint are computed from the dataset's meta files. The glossary entry for the LeRobot dataset format shows where they come from before they are baked into a checkpoint.

This is why the LeRobot dataset and the checkpoint are a matched pair, not two independent artifacts. The statistics are computed over the episodes you recorded. Re-record with a different calibration zero point and the same policy weights now expect a coordinate frame your arm no longer produces. If your policy moves confidently to a consistently wrong offset, this is the first thing to check, ahead of anything about the model, and it is the pattern behind a policy that only works in one setup.

LeRobot loads weights with strict=False

PreTrainedPolicy.from_pretrained defaults to strict=False and passes missing and unexpected keys to log_model_loading_keys, which calls logging.warning. It does not raise. A checkpoint whose normalisation buffers did not load, or whose action head did not match, will construct, run, and produce output. If you are reading a scrolling training log over SSH, those two warning lines are the only signal you get. Grep for Missing key(s) when loading model before you trust a load.

Case 2: a GR00T checkpoint is a different animal

NVIDIA's Isaac-GR00T stack builds on the Hugging Face Transformers trainer, so a GR00T checkpoint looks like a language-model checkpoint with robot-specific extras bolted on. Here is the shipped nvidia/GR00T-N1.7-3B repository.

FileBytesPartWhat it holds
model-00001-of-00002.safetensors4,990,519,232WeightsFirst shard
model-00002-of-00002.safetensors1,919,980,184WeightsSecond shard
model.safetensors.index.json104,985WeightsTensor name to shard map. Reports total_parameters 3,144,016,000 and total_size 6,910,361,856
config.json2,071ConfigArchitecture Gr00tN1d7, bfloat16, action_horizon 40, max_state_dim and max_action_dim 132
processor_config.json21,137ConfigModality configs per embodiment tag, use_percentiles true, use_mean_std false
statistics.json4,293,096NormalisationPer-embodiment min, max, mean, std, q01, q99 for every state and action key
experiment_cfg/dataset_statistics.json4,293,095NormalisationThe same statistics under the training-run record
embodiment_id.json2,145ConfigEmbodiment tag to integer index
trainer_state.json3,228,995Training stateThe pre-training run's log history: 19,600 entries, global_step 196000 of max_steps 200000, train_batch_size 32, save_steps 1000
training_args.bin8,259Training statePickled TrainingArguments
scheduler.pt1,263Training stateLR scheduler state
latest17Training stateContains the string global_step196000, matching trainer_state.json

Three structural differences from LeRobot matter. First, GR00T addresses its statistics by embodiment tag, so one checkpoint carries normalisation for many robots at once, and the loader raises a hard ValueError listing the supported tags if you ask for one that is not there. That is a better failure than a warning. Second, it normalises with percentiles: use_percentiles is true and use_mean_std is false in the shipped processor config, so the state and action ranges come from q01 and q99, not from mean and standard deviation. Third, the shipped nvidia/GR00T-N1.7-3B processor config lists only eight embodiment tags, none of them an SO-100 class arm, which is why the base model cannot be driven directly on your arm and has to be fine-tuned first.

experiment_cfg/ is a record, not the loader's input

In nvidia/GR00T-N1.7-3B, config.json and processor_config.json both say max_state_dim: 132, max_action_dim: 132 and use_percentiles: true, while experiment_cfg/final_model_config.json says 128, 128, use_percentiles: false and an action_horizon of 50 against config.json's 40. They disagree because Gr00tPolicy.__init__ loads through AutoModel.from_pretrained(model_dir) and AutoProcessor.from_pretrained(processor_dir), which read the root config files. The experiment_cfg/ tree is a snapshot of how the pre-training run was launched. Treat it as documentation, never as the contract, and do not copy a number out of it into a serving config.

Percentile normalisation versus mean and standard deviation
Why GR00T uses q01 and q99
  • A single stuck servo reading or one corrupt episode does not drag the mean and inflate the std the way it does with moment statistics.
  • The normalised range is bounded and predictable, which suits a diffusion action head trained on a fixed noise schedule.
  • Robots with hard joint limits produce distributions that are closer to uniform than Gaussian, and percentiles describe those better.
What it costs you
  • One percent of your recorded motion sits outside the range on each end and gets squashed, so genuinely fast or extreme motions are compressed.
  • The statistics are not interchangeable with a LeRobot mean-and-std checkpoint. You cannot copy numbers across; you have to recompute from the dataset.
  • Computing percentiles needs the full distribution, so the statistics pass is heavier than a streaming mean and variance.

Why moving a checkpoint between machines usually fails

Every one of these has a specific cause and a specific fix. None of them is the model being bad. The order below is roughly how often we see them, and most of them have a dedicated page under the failure modes index.

Symptom on the new machineCauseFix
Arm moves smoothly to a consistently wrong poseNormalisation statistics missing, stale, or from a different datasetCopy the preprocessor and postprocessor state files with the weights, or recompute stats from the dataset the weights were trained on
Loads fine, output is noiseWeights partially loaded because strict=False swallowed a key mismatchRe-run the load and read the warning lines, or load with strict=True in a throwaway script
RuntimeError about mps or cuda not being availableconfig.json carries the device of the machine that wrote it. lerobot/pi05_base ships with "device": "mps"Override device at load time. Current lerobot auto-switches and logs 'Device X is not available. Switching to Y', so read the warnings
Hangs or fails downloading torchvision weightsACT configs carry pretrained_backbone_weights, for example ResNet18_Weights.IMAGENET1K_V1, which is fetched at constructionPre-warm the torchvision cache on the target machine, or place the checkpoint next to a machine with network access
Shape mismatch on observation.state or actionDifferent degrees of freedom, or a different camera set, than the checkpoint was trained forCheck the input_features and output_features blocks in config.json against your robot before copying anything
Embodiment tag not supported by this checkpointGR00T checkpoint does not carry statistics for the tag you asked forRead the tags the error prints and use one of those, or fine-tune to create the tag you need
Resume restarts from step 0 or jumps in losstraining_state/ was not copied, or save_only_model was true when the checkpoint was writtenCopy the whole step directory including training_state/, and do not set save_only_model if you intend to resume
Import error on lerobot.utils.train_utilsModule moved from lerobot.utils.train_utils to lerobot.common.train_utils in v0.6.0Pin the lerobot version that wrote the checkpoint, or update the import to lerobot.common

There is one more that is not a bug but catches people anyway. Isaac-GR00T's FinetuneConfig, the dataclass its fine-tune CLI is generated from, defaults to save_total_limit 5 and save_steps 1000, so older checkpoints are deleted as new ones are written. If you were planning to compare step 2000 against step 20000 later, you have to raise the limit before the run, not after. The same applies to the GR00T N1.7 on SO-100 guide, where saveSteps is the one extra knob the form exposes.

The AY-Robots failure mode index, a list of concrete robot symptoms such as a policy that only works in one setup, each linking to its diagnosis page
Most of the symptoms in the table above have their own page. The failure index is organised by what you can see happening to the arm, not by which layer of the stack is at fault.
training_args.bin is a pickle

GR00T checkpoints include training_args.bin, which is a torch-serialised (that is, pickled) TrainingArguments object. safetensors exists precisely because pickle can execute arbitrary code on load. You never need this file to run inference. When you pull a checkpoint from somewhere you do not fully control, delete it rather than deserialising it, and keep the safetensors shards, the config files and the statistics.

Checking a checkpoint before you trust it

Five minutes of inspection beats an afternoon of watching an arm miss a cube. This runs against any LeRobot-style checkpoint directory, whether it came off your own GPU, out of a dataset-driven run, or off the Hub.

  1. 1
    List the directory and classify every file

    You are looking for four things: a weight file, a config, something holding statistics, and optionally a training_state directory. If the statistics are missing, stop here. The awk pass keeps this portable; find's -printf is GNU only and fails on a Mac.

    bash
    find outputs/train/my_run/checkpoints/last/ -maxdepth 2 -type f -exec ls -l {} + \
      | awk '{printf "%12s  %s\n", $5, $NF}' | sort -k2
  2. 2
    Read the safetensors header without loading the weights

    Confirms the tensor count, catches an old-style checkpoint whose normalisation buffers are still inside the state dict, and tells you the dtype.

    bash
    python - <<'EOF'
    import json, struct
    f = open("pretrained_model/model.safetensors", "rb")
    n = struct.unpack("<Q", f.read(8))[0]
    h = json.loads(f.read(n))
    h.pop("__metadata__", None)
    print("tensors:", len(h))
    print("norm buffers:", [k for k in h if "normal" in k])
    print("dtypes:", sorted({v["dtype"] for v in h.values()}))
    EOF
  3. 3
    Check the shapes in config.json against your robot

    input_features and output_features declare exactly what the policy expects. An SO-100 class arm records six values, five joints plus the gripper, so observation.state and action both carry shape [6]; that is what lerobot/smolvla_base declares. Camera keys have to match the names your client publishes, not just the count.

    bash
    python -c "import json,sys; c=json.load(open('pretrained_model/config.json')); print(c['type'], c.get('device')); print(json.dumps({'in':c['input_features'],'out':c['output_features']}, indent=1))"
  4. 4
    Print the normalisation statistics and sanity check the ranges

    The means should look like joint angles your arm actually reaches. A mean of 0.0 with a std of 1.0 across the board means the statistics were never filled in.

    bash
    python - <<'EOF'
    import glob, json, struct
    for path in glob.glob("pretrained_model/*normalizer*.safetensors"):
        b = open(path, "rb").read()
        n = struct.unpack("<Q", b[:8])[0]
        h, data = json.loads(b[8:8+n]), b[8+n:]
        print(path)
        for k, v in h.items():
            s, e = v["data_offsets"]
            vals = struct.unpack("<%df" % ((e-s)//4), data[s:e])
            print(" ", k, [round(x, 3) for x in vals])
    EOF
  5. 5
    Load it once with strict=True on a throwaway box

    This is the only cheap way to turn LeRobot's silent warnings into an exception you cannot miss. Do it before the checkpoint goes anywhere near a powered arm.

    python
    from lerobot.policies.factory import get_policy_class
    
    cls = get_policy_class("act")           # or smolvla, pi05
    policy = cls.from_pretrained("pretrained_model", strict=True)
    print(sum(p.numel() for p in policy.parameters()), "parameters loaded")
  6. 6
    Strip the training state before you copy it to the robot

    The pretrained_model subtree is everything inference needs. Leaving optimizer state behind saves bandwidth and removes the pickled files from the machine that talks to the servos.

    bash
    rsync -av --exclude 'training_state/' --exclude '*.bin' --exclude '*.pt' \
      outputs/train/my_run/checkpoints/last/ pi@robot.local:/opt/policy/

Two ways to get a checkpoint you can trust

Train locally or on a GPU you rent yourself, then move the artifacts by hand. Full control, and you own every failure. This is the LeRobot path for SmolVLA or ACT.

bash
git clone https://github.com/huggingface/lerobot.git
cd lerobot && git checkout v0.6.1
pip install -e ".[smolvla]"

lerobot-train \
  --policy.type=smolvla \
  --dataset.repo_id=<your-hf-user>/<your-dataset> \
  --batch_size=2 \
  --steps=20000 \
  --output_dir=outputs/train/smolvla_so100 \
  --job_name=smolvla_so100
Install lerobot from source and train. Pinning the tag fixes the on-disk layout: v0.6.1 writes the processor.json shape from the table above, not the main-branch one.
bash
lerobot-train \
  --config_path=outputs/train/smolvla_so100/checkpoints/last/pretrained_model/train_config.json \
  --resume=true
Resume from the last checkpoint. The config is read back out of the checkpoint, so flags you change now can disagree with what is recorded.
  • You choose the lerobot version, so the on-disk layout does not shift under you.
  • You can inspect every intermediate checkpoint, because nothing prunes them for you.
  • You are responsible for keeping the dataset that produced the statistics. Lose it and you cannot recompute them.
  • Moving the result to the robot is a manual rsync, with all the traps in the table above.
The AY-Robots policies comparison page showing the five trainable policies with parameter counts, GPU tier, inference latency and minimum episodes
The five trainable policies side by side. Parameter count sets the checkpoint size; the dataset format column decides whether your recording can feed that trainer at all.

Where none of this helps

Three honest limits, because a checkpoint you understand perfectly can still fail for reasons that have nothing to do with its contents.

  • A correct checkpoint served from far away is still a bad policy. Per-action-step time on this platform runs from 20 ms for ACT to 485 ms for Pi0.5, and adding public-internet round trips on top turns a working policy into a hesitant one. Remote inference is fine for slow pick and place and wrong for fast reactive motion. See inference latency for what that budget is actually spent on.
  • Nothing in a checkpoint tells you whether the data behind it was any good. Statistics computed over episodes with a frozen camera or a servo at its limit look completely normal. That is a dataset problem and it needs a data collection fix, not a checkpoint fix.
  • Reproducing a GR00T run bit for bit is not possible from the checkpoint. Its fine-tune CLI is generated by tyro from a FinetuneConfig dataclass that has no seed field at all, so two runs on the same data will not match. lerobot has one, defaulting to 1000. If you need reproducibility, that constrains which model you can use.

The flip side is that a checkpoint you can read is a checkpoint you can debug. Most of the reports that start with the policy being broken end with a config field or a statistics file, and both are legible in under a minute once you know where to look. If you are still stuck after the checks above, a falling loss with a policy that does nothing and a policy that freezes mid motion cover the two failure shapes that are not checkpoint problems.

The AY-Robots arena leaderboard, a sortable table of 85 vision-language-action models with 332 benchmark results, each value linked to its source paper or model card
85 models and 332 benchmark results in the arena. Benchmark numbers come from checkpoints someone else trained, on statistics computed over their data, which is exactly why cross-paper comparisons need the source link next to every value.

Frequently asked questions

Can I use a checkpoint trained on someone else's SO-100 on mine?

Sometimes, and it depends entirely on calibration. The weights encode a mapping from camera images and joint angles to joint targets in the coordinate frame their arm produced. If your zero points differ, the same commands land somewhere else. lerobot/smolvla_base carries three separate SO-100 statistic sets (so100, so100-blue, so100-red) precisely because different physical arms produce different distributions; between two of them the wrist mean differs by 79.1 raw units, more than two standard deviations of the tighter set. Start by fine-tuning on a small dataset from your own arm rather than running the checkpoint cold.

How large is a checkpoint going to be?

Weights dominate, and the size is parameter count times bytes per parameter, so dtype matters as much as architecture. Measured: the ACT checkpoint above is 207 MB across 242 float32 tensors, lerobot/smolvla_base is 907 MB, nvidia/GR00T-N1.7-3B reports 3,144,016,000 parameters and 6,910,361,856 bytes across two shards, and lerobot/pi05_base is 14.5 GB with dtype float32 declared in its config. Add roughly twice the trainable weights again if you keep the optimizer state for resuming.

What is the difference between model.safetensors and the sharded model-0000N-of-0000M files?

Only packaging. Above a size threshold the Hugging Face serializer splits the state dict across several files and writes model.safetensors.index.json mapping each tensor name to its shard. LeRobot deliberately pins the shard size to 1TB so a policy always writes exactly one model.safetensors with no index. GR00T lets it shard normally. If you copy a sharded checkpoint, the index file is not optional.

Do I need training_state/ to run the policy?

No. Inference needs the pretrained_model subtree: weights, config, and the processor files that carry normalisation. training_state holds optimizer moments, RNG state, the scheduler and the step counter, and exists only so a run can resume. Isaac-GR00T even has a save_only_model flag that skips writing it, with the documented consequence that you cannot resume from those checkpoints.

Why does my checkpoint try to use a device that does not exist?

Because the device is a field in config.json, written by whichever machine produced the checkpoint. The official lerobot/pi05_base repository ships with "device": "mps", the Apple Silicon backend. Current lerobot repairs this in PreTrainedConfig.__post_init__: if the recorded device is unavailable it auto-selects one and logs "Device 'X' is not available. Switching to 'Y'.". That is a warning, so it scrolls past. Override the device explicitly when you load.

Is safetensors actually safer than a .pt file?

Yes, and the difference is concrete. A .pt or .bin file is a Python pickle, and unpickling can execute arbitrary code. safetensors is a length-prefixed JSON header plus raw bytes, with the one reserved __metadata__ key restricted to a flat string-to-string map. There is no code path that runs attacker-controlled code. This is why GR00T's training_args.bin is worth deleting from any checkpoint you did not produce yourself.

When the arm moves and the pose is wrong

Failure-mode pages for the symptoms in this article: a policy that only works in one setup, a loss that falls while the policy does nothing, a dataset the trainer rejects. Each page starts from what you can see and ends at the file to check.

Open the failure modes

Closing

A checkpoint is four artifacts pretending to be one. The weights are the part everyone talks about and the part least likely to be your problem. The statistics are 640 bytes that decide where the arm goes. The config records the machine that wrote it, down to the GPU vendor. The training state is the largest thing in the directory and does nothing at inference time. Learn to read all four with the header dump above and the class of failures that starts with the model being broken mostly evaporates.

If you want to compare what the different families do before you commit to producing checkpoints in one of their formats, the five trainable policies are laid out with parameter counts and latency, GR00T N1.7 against Pi0.5 goes head to head, and the VLA overview covers why these architectures are shaped the way they are. For the hardware side, the SO-100 guide starts from an unopened parts box, the training docs describe the run itself, and the public dataset directory has recordings you can point a trainer at today.

Ready for high-quality robotics data?

AY-Robots connects your robots to skilled operators worldwide.

Get Started