
A checkpoint is four artifacts pretending to be one. Real file listings from LeRobot, GR00T and Pi0.5 checkpoints, and the eight reasons copying one between machines fails.
A checkpoint is not a model. It is a directory that happens to contain a model, plus three other things that are just as load bearing: the numbers used to scale your robot's joint angles into the range the network was trained on, a config that describes the architecture the weights belong to, and a pile of training state that only matters if you plan to resume. Get any of the four wrong and the arm still moves. It just moves to the wrong place.
This page opens real checkpoints and reads out what is in them. Every file name, byte count and tensor key below was pulled from a public repository and re-checked against it in August 2026, not recalled from memory. The three examples are a SmolVLA base checkpoint, an ACT checkpoint from the LeRobot model zoo, and NVIDIA's GR00T N1.7 base. Then we go through why copying one of these from the machine that trained it to the machine that runs the arm fails more often than it works.
What you need to know
- •A checkpoint has four parts: weights, normalisation statistics, config, and training state. Only the first three affect what the arm does at inference time.
- •The normalisation statistics are tiny. In lerobot/smolvla_base they are a 640 byte file next to a 907 MB weight file, and they decide where in space the joint commands land.
- •LeRobot loads weights with strict=False. A missing or mismatched tensor key produces a log warning, not an exception, so a half-loaded policy runs and looks plausible.
- •Configs carry the device of the machine that wrote them. lerobot/pi05_base ships with "device": "mps" in config.json, so a checkpoint straight off the Hub can name a backend your machine does not have.
- •GR00T normalises with the 1st and 99th percentile (use_percentiles true, use_mean_std false in both config.json and processor_config.json). LeRobot policies default to mean and standard deviation. The two are not interchangeable.
- •Training state (optimizer moments, RNG, scheduler) is often larger than the weights and is useless at inference. Strip it before you copy a checkpoint to a robot.
The four parts of a checkpoint
Whatever framework wrote it, a policy checkpoint decomposes into the same four categories. Knowing which category a file belongs to tells you whether you can delete it, whether it has to travel with the weights, and what breaks if it is stale.
| Part | Typical files | Needed at inference? | Needed to resume training? | What goes wrong if it is wrong |
|---|---|---|---|---|
| Weights | model.safetensors, model-0000N-of-0000M.safetensors, model.safetensors.index.json | Yes | Yes | Load fails loudly, or silently loads a subset and the arm drifts |
| Normalisation statistics | policy_preprocessor_step_*_normalizer_processor.safetensors, statistics.json, experiment_cfg/dataset_statistics.json | Yes | Yes | Arm moves smoothly to the wrong coordinates. No error anywhere |
| Config | config.json, train_config.json, processor_config.json, policy_preprocessor.json | Yes | Yes | Wrong architecture is built, shapes do not match, or the wrong device is selected |
| Training state | training_state/optimizer_state.safetensors, rng_state.safetensors, scheduler_state.json, training_step.json, optimizer.pt | No | Yes | Resume restarts the optimizer cold and the loss curve jumps |
The optimizer state for an Adam-family optimizer holds two moment tensors per trainable parameter, so it is roughly twice the size of the trainable weights. For a full fine-tune that means the resume artifacts dominate the directory. When you copy a checkpoint to the machine next to the arm, you want the pretrained_model/ subtree and nothing else.
Case 1: a LeRobot checkpoint on disk
LeRobot writes one directory per saved training step, zero padded to at least six digits, under checkpoints/ in the run output directory, plus a last symlink pointing at the newest one. The layout is documented in the docstring of save_checkpoint. It changed between releases, so the version matters.
005000/ # training step at checkpoint
|-- pretrained_model/
| |-- config.json # policy config
| |-- model.safetensors # policy weights
| |-- train_config.json # train config
| |-- processor.json # processor config (if a preprocessor was passed)
| `-- step_*.safetensors # processor state files (if any)
`-- training_state/
|-- optimizer_param_groups.json
|-- optimizer_state.safetensors
|-- rng_state.safetensors
|-- scheduler_state.json
`-- training_step.json005000/
|-- pretrained_model/
| |-- config.json
| |-- model.safetensors # or pytorch_model_fsdp_0/ for DCP formats
| |-- train_config.json
| |-- policy_preprocessor.json
| |-- policy_preprocessor_step_*.safetensors
| |-- policy_postprocessor.json
| `-- policy_postprocessor_step_*.safetensors
`-- training_state/
|-- optimizer_param_groups.json
|-- optimizer_state.safetensors # or optimizer_0/ for sharded runs
|-- rng_state.safetensors
|-- scheduler_state.json
`-- training_step.json # step + dp_world_size, batch_size, grad_accum, topologyThree things moved between the release you install and the branch it will become, and one thing that looks like it moved did not. The module changed path in v0.6.0, which is enough to break any script that imported it by path. training_step.json grew from a bare step counter into a snapshot of the run topology, so a resume can complain when the parallelism or the gradient-accumulation cadence changed. And main added artifacts for sharded runs that no release writes yet.
| Aspect | v0.5.1 (7 Apr 2026) | v0.6.1 (3 Aug 2026, current release) | main (23 Aug 2026) |
|---|---|---|---|
| Module path | src/lerobot/utils/train_utils.py | src/lerobot/common/train_utils.py | src/lerobot/common/train_utils.py |
| save_checkpoint docstring | processor.json plus step_*.safetensors | identical to v0.5.1 | policy_preprocessor.json, policy_postprocessor.json and their step files written out in full |
| training_step.json | {"step": step} and nothing else | step, num_processes, batch_size | step, dp_world_size, batch_size, grad_accum_steps |
| Sharded model and optimizer | not written | not written | pytorch_model_fsdp_0/ and optimizer_0/ under the DCP checkpoint formats |
| Processor state file naming | {pipeline name}_step_{index}_{registry name}.safetensors | same | same |
The last row is the one worth dwelling on, because it looks like a breaking rename and is not one. PolicyProcessorPipeline has built state file names out of the pipeline name, the step index and the registry name since v0.5.1, and v0.5.1's save_checkpoint already accepted both a preprocessor and a postprocessor. Only the docstring changed, from generic placeholders to the concrete names the released code was writing all along. That is why lerobot/smolvla_base on the Hub carries a file called policy_preprocessor_step_5_normalizer_processor.safetensors even though no tagged release names that file anywhere.
Before the processor pipeline existed, LeRobot kept the normalisation buffers inside the weight file itself, under keys like normalize_inputs.buffer_observation_state.mean. Those checkpoints still exist on the Hub; lerobot/act_aloha_sim_transfer_cube_human is one of them. If you load one with current lerobot, the normalisation keys are unexpected and the processor gets no statistics at all. lerobot ships src/lerobot/processor/migrate_policy_normalization.py to lift the buffers out of the state dict into a preprocessor and a postprocessor, and its documented usage example targets that exact repository. Run it once; do not hand-patch the state dict.
What the real repositories contain
These are byte counts from the Hugging Face API, not estimates. They show how lopsided a checkpoint is: the part that is easiest to lose is also the smallest.
| Repository | File | Bytes |
|---|---|---|
| lerobot/smolvla_base | model.safetensors | 906,712,520 |
| lerobot/smolvla_base | config.json | 2,299 |
| lerobot/smolvla_base | policy_preprocessor.json | 1,872 |
| lerobot/smolvla_base | policy_preprocessor_step_5_normalizer_processor.safetensors | 640 |
| lerobot/smolvla_base | policy_postprocessor_step_0_unnormalizer_processor.safetensors | 640 |
| lerobot/pi05_base | model.safetensors | 14,467,165,872 |
| lerobot/pi05_base | config.json | 1,896 |
| lerobot/pi05_base | policy_preprocessor.json | 1,241 |
| lerobot/pi05_base | policy_postprocessor.json | 567 |
| lerobot/act_aloha_sim_transfer_cube_human | model.safetensors | 206,766,560 |
| lerobot/act_aloha_sim_transfer_cube_human | train_config.json | 5,290 |
Note what lerobot/pi05_base does not have: it ships policy_preprocessor.json and policy_postprocessor.json but no normalizer state file. The base Pi0.5 checkpoint carries the pipeline structure without the statistics, because the statistics only exist once you have picked a dataset. lerobot/act_aloha_sim_transfer_cube_human has no processor files at all, because it predates them.
What is actually inside model.safetensors
safetensors is deliberately boring, which is the point. The file is eight bytes holding an unsigned little-endian 64-bit integer N, then N bytes of UTF-8 JSON, then the raw tensor bytes. The JSON header maps each tensor name to its dtype, its shape and a pair of byte offsets into the data region. There is one reserved key, __metadata__, and it may only hold a flat string-to-string map. No pickle, no code execution, no surprises when you open a file someone sent you.
That means you can read the entire structure of a 14 GB weight file by fetching its first few hundred kilobytes. Here is the whole reader, and the output from the ACT checkpoint above.
import json, struct, sys
with open(sys.argv[1], "rb") as f:
n = struct.unpack("<Q", f.read(8))[0] # header length
header = json.loads(f.read(n)) # {name: {dtype, shape, data_offsets}}
meta = header.pop("__metadata__", None)
print(f"header bytes: {n} tensors: {len(header)} metadata: {meta}")
for name, spec in header.items():
print(f"{name:60s} {spec['dtype']:5s} {spec['shape']}")normalize_inputs.buffer_observation_images_top.mean F32 [3, 1, 1]
normalize_inputs.buffer_observation_images_top.std F32 [3, 1, 1]
normalize_inputs.buffer_observation_state.mean F32 [14]
normalize_inputs.buffer_observation_state.std F32 [14]
normalize_targets.buffer_action.mean F32 [14]
normalize_targets.buffer_action.std F32 [14]
unnormalize_outputs.buffer_action.mean F32 [14]
unnormalize_outputs.buffer_action.std F32 [14]The [14] is a bimanual ALOHA setup: seven values per side. An SO-100 records six, five joints plus the gripper, which is exactly the shape lerobot/smolvla_base declares for observation.state and action. Those two shapes will never load into each other, which is the good case. The dangerous case is two checkpoints that agree on shape and disagree on what the numbers mean.
The 640 bytes that decide whether the arm works
Here is the entire normaliser state of lerobot/smolvla_base, decoded. Six tensors, 144 bytes of payload, 488 bytes of JSON header, 640 bytes on disk. It sits next to a 907 MB weight file.
so100-blue.buffer.action.mean [ 1.4212, 125.7205, 125.4193, 63.7832, -106.5401, 3.3485]
so100-blue.buffer.action.std [ 14.0820, 39.4314, 18.2381, 23.4898, 38.1112, 5.0483]
so100-red.buffer.action.mean [ 2.4387, 124.9379, 123.5578, 66.9926, -103.8768, 2.8912]
so100-red.buffer.action.std [ 14.2992, 42.7722, 18.8763, 21.6519, 33.5341, 4.4780]
so100.buffer.action.mean [ 1.5959, 119.9442, 109.7698, 56.7055, -27.4227, 12.0028]
so100.buffer.action.std [ 26.3918, 52.4114, 49.8538, 36.9983, 59.3600, 19.0400]Look at the fifth column, the wrist. The so100-blue statistics put its mean at -106.54; the plain so100 statistics put it at -27.42. Mean-and-standard-deviation normalisation means the model emits a normalised value and the postprocessor computes value times std plus mean, so a predicted normalised zero comes out at -106.54 under one set of statistics and at -27.42 under the other. The gap is 79.1 raw units: 2.1 standard deviations under the so100-blue statistics, 1.3 under the plain so100 ones. Whatever unit that arm recorded in, moving a wrist joint by more than a standard deviation is the difference between closing on an object and swiping past it, and it comes out of a 640 byte file with no error message anywhere in the stack.

This is why the LeRobot dataset and the checkpoint are a matched pair, not two independent artifacts. The statistics are computed over the episodes you recorded. Re-record with a different calibration zero point and the same policy weights now expect a coordinate frame your arm no longer produces. If your policy moves confidently to a consistently wrong offset, this is the first thing to check, ahead of anything about the model, and it is the pattern behind a policy that only works in one setup.
PreTrainedPolicy.from_pretrained defaults to strict=False and passes missing and unexpected keys to log_model_loading_keys, which calls logging.warning. It does not raise. A checkpoint whose normalisation buffers did not load, or whose action head did not match, will construct, run, and produce output. If you are reading a scrolling training log over SSH, those two warning lines are the only signal you get. Grep for Missing key(s) when loading model before you trust a load.
Case 2: a GR00T checkpoint is a different animal
NVIDIA's Isaac-GR00T stack builds on the Hugging Face Transformers trainer, so a GR00T checkpoint looks like a language-model checkpoint with robot-specific extras bolted on. Here is the shipped nvidia/GR00T-N1.7-3B repository.
| File | Bytes | Part | What it holds |
|---|---|---|---|
| model-00001-of-00002.safetensors | 4,990,519,232 | Weights | First shard |
| model-00002-of-00002.safetensors | 1,919,980,184 | Weights | Second shard |
| model.safetensors.index.json | 104,985 | Weights | Tensor name to shard map. Reports total_parameters 3,144,016,000 and total_size 6,910,361,856 |
| config.json | 2,071 | Config | Architecture Gr00tN1d7, bfloat16, action_horizon 40, max_state_dim and max_action_dim 132 |
| processor_config.json | 21,137 | Config | Modality configs per embodiment tag, use_percentiles true, use_mean_std false |
| statistics.json | 4,293,096 | Normalisation | Per-embodiment min, max, mean, std, q01, q99 for every state and action key |
| experiment_cfg/dataset_statistics.json | 4,293,095 | Normalisation | The same statistics under the training-run record |
| embodiment_id.json | 2,145 | Config | Embodiment tag to integer index |
| trainer_state.json | 3,228,995 | Training state | The pre-training run's log history: 19,600 entries, global_step 196000 of max_steps 200000, train_batch_size 32, save_steps 1000 |
| training_args.bin | 8,259 | Training state | Pickled TrainingArguments |
| scheduler.pt | 1,263 | Training state | LR scheduler state |
| latest | 17 | Training state | Contains the string global_step196000, matching trainer_state.json |
Three structural differences from LeRobot matter. First, GR00T addresses its statistics by embodiment tag, so one checkpoint carries normalisation for many robots at once, and the loader raises a hard ValueError listing the supported tags if you ask for one that is not there. That is a better failure than a warning. Second, it normalises with percentiles: use_percentiles is true and use_mean_std is false in the shipped processor config, so the state and action ranges come from q01 and q99, not from mean and standard deviation. Third, the shipped nvidia/GR00T-N1.7-3B processor config lists only eight embodiment tags, none of them an SO-100 class arm, which is why the base model cannot be driven directly on your arm and has to be fine-tuned first.
In nvidia/GR00T-N1.7-3B, config.json and processor_config.json both say max_state_dim: 132, max_action_dim: 132 and use_percentiles: true, while experiment_cfg/final_model_config.json says 128, 128, use_percentiles: false and an action_horizon of 50 against config.json's 40. They disagree because Gr00tPolicy.__init__ loads through AutoModel.from_pretrained(model_dir) and AutoProcessor.from_pretrained(processor_dir), which read the root config files. The experiment_cfg/ tree is a snapshot of how the pre-training run was launched. Treat it as documentation, never as the contract, and do not copy a number out of it into a serving config.
- A single stuck servo reading or one corrupt episode does not drag the mean and inflate the std the way it does with moment statistics.
- The normalised range is bounded and predictable, which suits a diffusion action head trained on a fixed noise schedule.
- Robots with hard joint limits produce distributions that are closer to uniform than Gaussian, and percentiles describe those better.
- One percent of your recorded motion sits outside the range on each end and gets squashed, so genuinely fast or extreme motions are compressed.
- The statistics are not interchangeable with a LeRobot mean-and-std checkpoint. You cannot copy numbers across; you have to recompute from the dataset.
- Computing percentiles needs the full distribution, so the statistics pass is heavier than a streaming mean and variance.
Why moving a checkpoint between machines usually fails
Every one of these has a specific cause and a specific fix. None of them is the model being bad. The order below is roughly how often we see them, and most of them have a dedicated page under the failure modes index.
| Symptom on the new machine | Cause | Fix |
|---|---|---|
| Arm moves smoothly to a consistently wrong pose | Normalisation statistics missing, stale, or from a different dataset | Copy the preprocessor and postprocessor state files with the weights, or recompute stats from the dataset the weights were trained on |
| Loads fine, output is noise | Weights partially loaded because strict=False swallowed a key mismatch | Re-run the load and read the warning lines, or load with strict=True in a throwaway script |
| RuntimeError about mps or cuda not being available | config.json carries the device of the machine that wrote it. lerobot/pi05_base ships with "device": "mps" | Override device at load time. Current lerobot auto-switches and logs 'Device X is not available. Switching to Y', so read the warnings |
| Hangs or fails downloading torchvision weights | ACT configs carry pretrained_backbone_weights, for example ResNet18_Weights.IMAGENET1K_V1, which is fetched at construction | Pre-warm the torchvision cache on the target machine, or place the checkpoint next to a machine with network access |
| Shape mismatch on observation.state or action | Different degrees of freedom, or a different camera set, than the checkpoint was trained for | Check the input_features and output_features blocks in config.json against your robot before copying anything |
| Embodiment tag not supported by this checkpoint | GR00T checkpoint does not carry statistics for the tag you asked for | Read the tags the error prints and use one of those, or fine-tune to create the tag you need |
| Resume restarts from step 0 or jumps in loss | training_state/ was not copied, or save_only_model was true when the checkpoint was written | Copy the whole step directory including training_state/, and do not set save_only_model if you intend to resume |
| Import error on lerobot.utils.train_utils | Module moved from lerobot.utils.train_utils to lerobot.common.train_utils in v0.6.0 | Pin the lerobot version that wrote the checkpoint, or update the import to lerobot.common |
There is one more that is not a bug but catches people anyway. Isaac-GR00T's FinetuneConfig, the dataclass its fine-tune CLI is generated from, defaults to save_total_limit 5 and save_steps 1000, so older checkpoints are deleted as new ones are written. If you were planning to compare step 2000 against step 20000 later, you have to raise the limit before the run, not after. The same applies to the GR00T N1.7 on SO-100 guide, where saveSteps is the one extra knob the form exposes.

GR00T checkpoints include training_args.bin, which is a torch-serialised (that is, pickled) TrainingArguments object. safetensors exists precisely because pickle can execute arbitrary code on load. You never need this file to run inference. When you pull a checkpoint from somewhere you do not fully control, delete it rather than deserialising it, and keep the safetensors shards, the config files and the statistics.
Checking a checkpoint before you trust it
Five minutes of inspection beats an afternoon of watching an arm miss a cube. This runs against any LeRobot-style checkpoint directory, whether it came off your own GPU, out of a dataset-driven run, or off the Hub.
- 1List the directory and classify every file
You are looking for four things: a weight file, a config, something holding statistics, and optionally a training_state directory. If the statistics are missing, stop here. The awk pass keeps this portable; find's -printf is GNU only and fails on a Mac.
bashfind outputs/train/my_run/checkpoints/last/ -maxdepth 2 -type f -exec ls -l {} + \ | awk '{printf "%12s %s\n", $5, $NF}' | sort -k2 - 2Read the safetensors header without loading the weights
Confirms the tensor count, catches an old-style checkpoint whose normalisation buffers are still inside the state dict, and tells you the dtype.
bashpython - <<'EOF' import json, struct f = open("pretrained_model/model.safetensors", "rb") n = struct.unpack("<Q", f.read(8))[0] h = json.loads(f.read(n)) h.pop("__metadata__", None) print("tensors:", len(h)) print("norm buffers:", [k for k in h if "normal" in k]) print("dtypes:", sorted({v["dtype"] for v in h.values()})) EOF - 3Check the shapes in config.json against your robot
input_features and output_features declare exactly what the policy expects. An SO-100 class arm records six values, five joints plus the gripper, so observation.state and action both carry shape [6]; that is what lerobot/smolvla_base declares. Camera keys have to match the names your client publishes, not just the count.
bashpython -c "import json,sys; c=json.load(open('pretrained_model/config.json')); print(c['type'], c.get('device')); print(json.dumps({'in':c['input_features'],'out':c['output_features']}, indent=1))" - 4Print the normalisation statistics and sanity check the ranges
The means should look like joint angles your arm actually reaches. A mean of 0.0 with a std of 1.0 across the board means the statistics were never filled in.
bashpython - <<'EOF' import glob, json, struct for path in glob.glob("pretrained_model/*normalizer*.safetensors"): b = open(path, "rb").read() n = struct.unpack("<Q", b[:8])[0] h, data = json.loads(b[8:8+n]), b[8+n:] print(path) for k, v in h.items(): s, e = v["data_offsets"] vals = struct.unpack("<%df" % ((e-s)//4), data[s:e]) print(" ", k, [round(x, 3) for x in vals]) EOF - 5Load it once with strict=True on a throwaway box
This is the only cheap way to turn LeRobot's silent warnings into an exception you cannot miss. Do it before the checkpoint goes anywhere near a powered arm.
pythonfrom lerobot.policies.factory import get_policy_class cls = get_policy_class("act") # or smolvla, pi05 policy = cls.from_pretrained("pretrained_model", strict=True) print(sum(p.numel() for p in policy.parameters()), "parameters loaded") - 6Strip the training state before you copy it to the robot
The pretrained_model subtree is everything inference needs. Leaving optimizer state behind saves bandwidth and removes the pickled files from the machine that talks to the servos.
bashrsync -av --exclude 'training_state/' --exclude '*.bin' --exclude '*.pt' \ outputs/train/my_run/checkpoints/last/ pi@robot.local:/opt/policy/
Two ways to get a checkpoint you can trust
Train locally or on a GPU you rent yourself, then move the artifacts by hand. Full control, and you own every failure. This is the LeRobot path for SmolVLA or ACT.
git clone https://github.com/huggingface/lerobot.git
cd lerobot && git checkout v0.6.1
pip install -e ".[smolvla]"
lerobot-train \
--policy.type=smolvla \
--dataset.repo_id=<your-hf-user>/<your-dataset> \
--batch_size=2 \
--steps=20000 \
--output_dir=outputs/train/smolvla_so100 \
--job_name=smolvla_so100lerobot-train \
--config_path=outputs/train/smolvla_so100/checkpoints/last/pretrained_model/train_config.json \
--resume=true- You choose the lerobot version, so the on-disk layout does not shift under you.
- You can inspect every intermediate checkpoint, because nothing prunes them for you.
- You are responsible for keeping the dataset that produced the statistics. Lose it and you cannot recompute them.
- Moving the result to the robot is a manual rsync, with all the traps in the table above.
The training form picks a model and a dataset, the backend rents a GPU on a spot market sized by the required VRAM, runs the trainer, and writes the checkpoints to object storage. Serving goes through /api/inference/pod, which auto-provisions a pod for the policy so the local robot client talks to an endpoint instead of to a file path.
| Model | GPU tier | Dataset format the trainer needs | Minimum episodes | Typical run cost |
|---|---|---|---|---|
| GR00T N1.7 | A100 80 GB or H100 80 GB | LeRobot v2.0 or v2.1 | 50 | about 4 to 12 USD |
| GR00T N1.5 | A100 80 GB or H100 80 GB | LeRobot v2.0 or v2.1 | 50 | about 4 to 12 USD |
| Pi0.5 | A100 80 GB or H100 80 GB | LeRobot v3.0 | 50 | about 4 to 12 USD |
| SmolVLA | RTX 4090 or any 24 GB card | LeRobot v3.0 | 30 | about 1 to 3 USD |
| ACT | RTX 4090 or any 24 GB card | LeRobot v3.0 | 50 | about 1 to 3 USD |
- The dataset that produced the statistics stays attached to the run, so the pairing is not something you have to remember.
- Inference pods carry an idle watchdog and destroy themselves after an idle period, so a forgotten checkpoint server does not keep billing.
- The same operations are available from a terminal via the CLI and to AI agents via the MCP server.
- GR00T N1.7 and N1.5 are cloud only here. SmolVLA and ACT also run locally, which is the honest place to start if you want the files on your own disk.
GR00T's loader crashes on a LeRobot v3.0 dataset. It has to be converted down to v2.1 first. This is upstream behaviour, not something the platform invents, and it is the single most common reason a GR00T run refuses to start. See the failure page for a rejected v3 dataset.

Where none of this helps
Three honest limits, because a checkpoint you understand perfectly can still fail for reasons that have nothing to do with its contents.
- A correct checkpoint served from far away is still a bad policy. Per-action-step time on this platform runs from 20 ms for ACT to 485 ms for Pi0.5, and adding public-internet round trips on top turns a working policy into a hesitant one. Remote inference is fine for slow pick and place and wrong for fast reactive motion. See inference latency for what that budget is actually spent on.
- Nothing in a checkpoint tells you whether the data behind it was any good. Statistics computed over episodes with a frozen camera or a servo at its limit look completely normal. That is a dataset problem and it needs a data collection fix, not a checkpoint fix.
- Reproducing a GR00T run bit for bit is not possible from the checkpoint. Its fine-tune CLI is generated by tyro from a FinetuneConfig dataclass that has no seed field at all, so two runs on the same data will not match. lerobot has one, defaulting to 1000. If you need reproducibility, that constrains which model you can use.
The flip side is that a checkpoint you can read is a checkpoint you can debug. Most of the reports that start with the policy being broken end with a config field or a statistics file, and both are legible in under a minute once you know where to look. If you are still stuck after the checks above, a falling loss with a policy that does nothing and a policy that freezes mid motion cover the two failure shapes that are not checkpoint problems.

Frequently asked questions
Can I use a checkpoint trained on someone else's SO-100 on mine?▾
Sometimes, and it depends entirely on calibration. The weights encode a mapping from camera images and joint angles to joint targets in the coordinate frame their arm produced. If your zero points differ, the same commands land somewhere else. lerobot/smolvla_base carries three separate SO-100 statistic sets (so100, so100-blue, so100-red) precisely because different physical arms produce different distributions; between two of them the wrist mean differs by 79.1 raw units, more than two standard deviations of the tighter set. Start by fine-tuning on a small dataset from your own arm rather than running the checkpoint cold.
How large is a checkpoint going to be?▾
Weights dominate, and the size is parameter count times bytes per parameter, so dtype matters as much as architecture. Measured: the ACT checkpoint above is 207 MB across 242 float32 tensors, lerobot/smolvla_base is 907 MB, nvidia/GR00T-N1.7-3B reports 3,144,016,000 parameters and 6,910,361,856 bytes across two shards, and lerobot/pi05_base is 14.5 GB with dtype float32 declared in its config. Add roughly twice the trainable weights again if you keep the optimizer state for resuming.
What is the difference between model.safetensors and the sharded model-0000N-of-0000M files?▾
Only packaging. Above a size threshold the Hugging Face serializer splits the state dict across several files and writes model.safetensors.index.json mapping each tensor name to its shard. LeRobot deliberately pins the shard size to 1TB so a policy always writes exactly one model.safetensors with no index. GR00T lets it shard normally. If you copy a sharded checkpoint, the index file is not optional.
Do I need training_state/ to run the policy?▾
No. Inference needs the pretrained_model subtree: weights, config, and the processor files that carry normalisation. training_state holds optimizer moments, RNG state, the scheduler and the step counter, and exists only so a run can resume. Isaac-GR00T even has a save_only_model flag that skips writing it, with the documented consequence that you cannot resume from those checkpoints.
Why does my checkpoint try to use a device that does not exist?▾
Because the device is a field in config.json, written by whichever machine produced the checkpoint. The official lerobot/pi05_base repository ships with "device": "mps", the Apple Silicon backend. Current lerobot repairs this in PreTrainedConfig.__post_init__: if the recorded device is unavailable it auto-selects one and logs "Device 'X' is not available. Switching to 'Y'.". That is a warning, so it scrolls past. Override the device explicitly when you load.
Is safetensors actually safer than a .pt file?▾
Yes, and the difference is concrete. A .pt or .bin file is a Python pickle, and unpickling can execute arbitrary code. safetensors is a length-prefixed JSON header plus raw bytes, with the one reserved __metadata__ key restricted to a flat string-to-string map. There is no code path that runs attacker-controlled code. This is why GR00T's training_args.bin is worth deleting from any checkpoint you did not produce yourself.
When the arm moves and the pose is wrong
Failure-mode pages for the symptoms in this article: a policy that only works in one setup, a loss that falls while the policy does nothing, a dataset the trainer rejects. Each page starts from what you can see and ends at the file to check.
Open the failure modesClosing
A checkpoint is four artifacts pretending to be one. The weights are the part everyone talks about and the part least likely to be your problem. The statistics are 640 bytes that decide where the arm goes. The config records the machine that wrote it, down to the GPU vendor. The training state is the largest thing in the directory and does nothing at inference time. Learn to read all four with the header dump above and the class of failures that starts with the model being broken mostly evaporates.
If you want to compare what the different families do before you commit to producing checkpoints in one of their formats, the five trainable policies are laid out with parameter counts and latency, GR00T N1.7 against Pi0.5 goes head to head, and the VLA overview covers why these architectures are shaped the way they are. For the hardware side, the SO-100 guide starts from an unopened parts box, the training docs describe the run itself, and the public dataset directory has recordings you can point a trainer at today.
Sources
- lerobot main: save_checkpoint docstring, and save_training_step recording dp_world_size, batch_size and grad_accum_steps
- lerobot v0.6.1: the layout the current release actually writes
- lerobot v0.5.1: the pre-v0.6.0 module path, and training_step.json as a bare step counter
- lerobot: PolicyProcessorPipeline, where processor state file names are built
- lerobot: PreTrainedPolicy.from_pretrained with strict=False, and the 1TB single-file shard pin
- lerobot: log_model_loading_keys, which warns on missing keys instead of raising
- lerobot: PreTrainedConfig.__post_init__ and the device auto-switch warning
- lerobot: migrating in-model normalisation buffers to the processor pipeline
- safetensors: format specification, 8-byte header length plus JSON header, __metadata__ restricted to strings
- lerobot/smolvla_base: preprocessor pipeline, the baked-in device and the normalizer state file reference
- lerobot/pi05_base: config.json shipping with device mps and dtype float32
- lerobot/act_aloha_sim_transfer_cube_human: an older checkpoint with normalisation inside the weight file
- nvidia/GR00T-N1.7-3B: processor config, eight embodiment tags, use_percentiles true
- Isaac-GR00T: Gr00tPolicy loading model, processor and embodiment tag, and the ValueError listing supported tags
- Isaac-GR00T: FinetuneConfig defaults, save_steps 1000, save_total_limit 5, save_only_model, and no seed field
Ready for high-quality robotics data?
AY-Robots connects your robots to skilled operators worldwide.
Get Started