From Checkpoint to Running Arm: The Last Mile

Training finishes. The loss curve looks reasonable, a directory full of safetensors appears, and nothing at all happens to the robot. The last mile, taking a finished checkpoint and getting it to drive a real SO-100, is where most of a day disappears, and it is almost never the model's fault. It is a camera key spelled differently, an embodiment tag that does not exist, a backbone the checkpoint still needs to download, or an execution horizon longer than the chunk the model predicts.

This page walks that mile for the three stacks that ship today: lerobot's lerobot-rollout CLI, NVIDIA's Isaac-GR00T policy server, and openpi's websocket server. Real paths, real flags, real error text. Every command, file name and number below was read out of the upstream repositories and model cards on 24 August 2026, and where a stack changed recently the version is named. If you have not trained anything yet, train your first policy first and come back with a checkpoint in hand.

What training actually left you

Three trainers, three different trees on disk. None of them is a single file, and the difference between them is the reason a checkpoint that works on the training pod does nothing on the robot machine. The table below is what each stack writes and what reads it back.

The base GR00T N1.7 repository on Hugging Face shows the shape clearly. This is the complete file list of nvidia/GR00T-N1.7-3B as the Hugging Face model API returned it on 24 August 2026, license and readme files included, because knowing which entries are noise is half the point.

.gitattributes
EXPLAINABILITY.md
LICENSE
PRIVACY.md
README.md
SAFETY_and_SECURITY.md
SUCCESS
config.json
embodiment_id.json
experiment_cfg/conf.yaml
experiment_cfg/config.yaml
experiment_cfg/dataset_statistics.json
experiment_cfg/final_model_config.json
experiment_cfg/final_processor_config.json
experiment_cfg/initial_actions.npz
latest
model-00001-of-00002.safetensors
model-00002-of-00002.safetensors
model-architecture.png
model.safetensors.index.json
processor_config.json
scheduler.pt
statistics.json
trainer_state.json
training_args.bin
wandb_config.json
zero_to_fp32.py

Which of those does a fine-tune of your own actually need on the robot machine? NVIDIA's deployment guide answers it by example: when it pulls a checkpoint down to run inference, it fetches config.json, embodiment_id.json, the model-*.safetensors shards, model.safetensors.index.json, processor_config.json and statistics.json. That is the set to copy. trainer_state.json, scheduler.pt, training_args.bin and wandb_config.json are training bookkeeping. statistics.json is the one nobody misses until it is missing: it carries the normalization statistics that turn raw network output back into joint angles.

The five things that have to match

A policy is a function from a named dictionary of observations to a named dictionary of actions. Deployment is the job of making the robot produce exactly the dictionary the checkpoint was trained on. Five things decide whether it does.

Camera keys deserve their own paragraph because the fix is unintuitive. lerobot ships a rename map, a JSON dictionary passed on the command line that renames observation keys inside the preprocessor pipeline so the policy always sees the names baked into its config. The convention is always source key first, policy key second. It is supported for Pi0, Pi0.5, Pi0Fast, SmolVLA and XVLA, and it renames observations only: action keys are untouched.

Route A: lerobot-rollout, one command from checkpoint to motion

In current lerobot, lerobot-rollout is the single CLI for deploying a trained policy on a real robot, and the imitation-learning guide now points at it rather than at the recording command. It covers ACT, SmolVLA, Pi0.5 and GR00T N1.7 through one interface. Two things you choose: a strategy, which is the shape of the control loop, and an inference backend, which is how chunks are produced.

1. 1
   Install the deployment extras on the robot machine

   The training environment and the robot environment are not the same install. You need the serial driver for the servos and, if you want the live view, the visualization extra. This is the exact line the GR00T policy page uses before its hardware rollout.

   bash

   pip install "lerobot[feetech,viz]"

2. 2
   Find the camera indices, do not guess them

   USB camera indices move between reboots and between machines. Enumerate them, then write the index into the rollout command explicitly.

   bash

   lerobot-find-cameras opencv

3. 3
   Run the base strategy for a fixed duration

   No recording, no dataset, no Hub push. This is the smallest thing that proves the checkpoint drives the arm. Keep a hand near the power switch for the first run.

   bash

   lerobot-rollout \
     --strategy.type=base \
     --policy.path=${HF_USER}/my_policy \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
     --task="Put lego brick into the box" \
     --duration=60 \
     --display_data=true

4. 4
   Read the cadence report instead of guessing

   Every strategy reports what its control loop actually achieved against the --fps target, and so do lerobot-record, lerobot-teleoperate and lerobot-replay, because they pace through the same timer. The breakdown names observe, process_obs, infer, send and record. A high infer share points at the policy or the device, a high observe share points at the cameras, and pacing headroom near zero means the loop is saturated and the next slow tick will cost a frame.

   text

   Cadence summary - whole run, 2 episodes - target 30 Hz x 2 (16.7 ms tick slot, 33.3 ms cycle budget)
     effective cadence: 29.84 Hz policy / 59.67 Hz commands over 40.2 s measured
     cycles over the 33.3 ms work budget: 12/1197 (1.0%) - work mean 18.7 ms, worst 48.1 ms
     loop-body steps (share of measured work):
       observe      mean   3.13 ms -  31.7% of work
       process_obs  mean   0.50 ms -   5.1% of work
       infer        mean   5.18 ms -  52.4% of work
       send         mean   0.40 ms -   4.1% of work
       record       mean   0.67 ms -   6.8% of work
     pacing headroom: 7.4 ms slept per tick on average

The second choice is the backend. The default is synchronous: one policy call per control tick, and the loop blocks until the action comes back. That is fine for ACT at 20 ms per step and painful for a 3 B parameter VLA. Real-Time Chunking runs a background thread that produces the next action chunk while the current one is still being consumed, and guides the new chunk so its opening steps blend with the motion already under way. The technique comes from the Real-Time Execution of Action Chunking Flow Policies paper (Black, Galliker and Levine, June 2025) and lerobot exposes it as a backend flag.

export MODEL_ID=your_trained_model_on_huggingface

lerobot-rollout \
  --strategy.type=base \
  --inference.type=rtc \
  --inference.rtc.execution_horizon=8 \
  --inference.queue_threshold=0 \
  --policy.path=$MODEL_ID \
  --policy.base_model_path=nvidia/GR00T-N1.7-3B \
  --policy.n_action_steps=8 \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=follower_robot \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30, fourcc: \"MJPG\"}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30, fourcc: \"MJPG\"} }" \
  --task="place the vial in the rack" \
  --duration=60 \
  --device=cuda \
  --display_data=true

Route B: the Isaac-GR00T policy server and a thin client

NVIDIA's own stack splits the problem in two. The policy runs inside a server process on the GPU; the robot side is a small client that speaks ZeroMQ with msgpack serialization and carries none of the model dependencies. Upstream gives two reasons for the split, and both are practical: the inference GPU does not have to be the machine holding the serial port, and the robot environment does not have to satisfy the model's dependencies. Isaac-GR00T ships a worked SO-100 example, which is the closest thing upstream has to this article.

1. 1
   Check the checkpoint open-loop before touching hardware

   Predict on an episode the model was trained on and compare against ground truth. The script logs Average MSE and Average MAE across trajectories as unnormalized action error and saves a ground-truth-versus-predicted plot. NVIDIA deliberately publishes no target MSE, because the number does not transfer between datasets: what you read is plot overlap on a training trajectory first, then whether the error falls across successive checkpoints. Pass --save-plot-path explicitly, since the SO-100 guide says plots land in /tmp/open_loop_eval/traj_<id>.jpeg by default while the parameter table lists no default at all.

   bash

   uv run python gr00t/eval/open_loop_eval.py \
     --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich \
     --embodiment-tag NEW_EMBODIMENT \
     --model-path /tmp/so100_finetune/checkpoint-10000 \
     --traj-ids 0 \
     --execution-horizon 16 \
     --steps 400 \
     --modality-keys single_arm gripper \
     --save-plot-path /tmp/open_loop_eval

2. 2
   Start the policy server on the GPU machine

   Bind to 0.0.0.0 only if the robot machine is elsewhere and the link is trusted; 127.0.0.1 plus an SSH tunnel is the safer default. The server prints its embodiment tag, model path, device and listening address before it accepts anything, which is the fastest way to catch a wrong checkpoint path.

   bash

   uv run python gr00t/eval/run_gr00t_server.py \
     --model-path /tmp/so100_finetune/checkpoint-10000 \
     --embodiment-tag NEW_EMBODIMENT \
     --device cuda:0 \
     --host 0.0.0.0 \
     --port 5555

3. 3
   Run the SO-100 client in its own environment

   The client environment is created separately under gr00t/eval/real_robot/SO100 with its own uv sync, then the repo root is installed into it with --no-deps, precisely so the robot side does not pull in flash-attn, onnx and TensorRT. The keys in --robot.cameras have to be the video keys from modality.json (front and wrist in the SO-100 example), and the language instruction has to be the instruction the episodes were labelled with.

   bash

   cd "$GR00T_REPO/gr00t/eval/real_robot/SO100"
   uv sync
   uv pip install --no-deps -e ../../../../
   
   uv run --no-sync python eval_so100.py \
     --robot.type=so101_follower \
     --robot.port=/dev/ttyACM2 \
     --robot.id=orange_follower \
     --robot.cameras="{ front: {type: opencv, index_or_path: 6, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
     --policy_host=localhost \
     --policy_port=5555 \
     --lang_instruction="finish the ham cheese olives sandwich"

Route C: openpi over a websocket

If you fine-tuned Pi0 or Pi0.5 with Physical Intelligence's own openpi rather than through lerobot, the deployment surface is a websocket server on port 8000 and a client package with deliberately minimal dependencies, installed into the robot environment from packages/openpi-client. The observation dictionary is flat and prefixed, the client resizes images itself to keep bandwidth down, and the server does the normalization. Note the shape of the return value: it is a whole action chunk, and upstream expects you to execute several steps of it open-loop before asking again.

from openpi_client import image_tools
from openpi_client import websocket_client_policy

# localhost and 8000 are the defaults
client = websocket_client_policy.WebsocketClientPolicy(host="localhost", port=8000)

observation = {
    "observation/image": image_tools.convert_to_uint8(
        image_tools.resize_with_pad(img, 224, 224)
    ),
    "observation/wrist_image": image_tools.convert_to_uint8(
        image_tools.resize_with_pad(wrist_img, 224, 224)
    ),
    "observation/state": state,
    "prompt": task_instruction,
}

# shape (action_horizon, action_dim)
action_chunk = client.infer(observation)["actions"]

The matching server command is uv run scripts/serve_policy.py policy:checkpoint --policy.config=<config name> --policy.dir=<checkpoint dir>, where the checkpoint directory looks like checkpoints/pi05_libero/my_experiment/20000. The config name is not decorative: it selects the input and output transforms that were used during training, the same classes that map your robot's observations into the model and its actions back out, so a checkpoint served under the wrong config produces well-formed nonsense. This is the openpi equivalent of GR00T's embodiment tag.

The errors in between

Collected from the upstream troubleshooting tables and from what actually goes wrong in the first hour. Read the first column, then check the third before you change anything about the model.

Several of these have a dedicated page: camera not detected, arm not detected, policy freezes mid-motion, loss falls but the policy does nothing and policy only works in one setup. The last one is worth reading before you conclude the deployment is broken, because a policy that works only under the exact lighting it was recorded in is a data problem wearing a deployment costume.

How fast it runs once it runs

Deployment is not finished when the arm moves; it is finished when the arm moves at a usable rate. NVIDIA publishes measured end-to-end timings for GR00T N1.7 at four denoising steps with one camera, comparing PyTorch eager against torch.compile against the full TensorRT pipeline. These are the model's own numbers on the GPU, data processing included, before any camera capture or serial cost. They are not AY-Robots figures and they are not a benchmark reproduced here.

Two things stand out. The end-to-end speedup from TensorRT is real but bounded, from 1.28x on DGX Spark to 3.34x on the L40 in NVIDIA's own table, and it is largest on the cards that were already fastest. And the edge devices sit an order of magnitude behind the datacenter cards, which is the honest reason most SO-100 setups keep a GPU on the desk rather than on the robot.

Doing it yourself vs doing it on AY-Robots

The manual path above is complete and it works. What it costs you is environment management: two Python environments, a gated backbone, a serial port, camera indices that renumber on reboot, and a GPU that has to be alive at the moment you want to test. Here is the same goal, both ways.

Where none of this helps

Worth saying plainly, because the failure is silent and gets blamed on the model. Inference latency on this platform runs from 20 ms per action step for ACT to 485 ms for Pi0.5, with GR00T N1.7 at 152 ms and SmolVLA at 245 ms. Those are the numbers before you add anything. A public-internet round trip added to a 485 ms step does not make the policy worse at the task; it makes the arm arrive late, over and over, which looks identical to a bad policy.

• Remote inference is viable for slow pick-and-place. It is not viable for fast reactive motion, and no amount of chunking hides a link that is genuinely far away.
• Nothing here rescues a dataset problem. A policy that only works under the lighting it was recorded in is a recording issue, and recording is where it gets fixed.
• A LeRobot v3.0 dataset crashes the GR00T loader; it has to be converted down to v2.1 first, which is a training-side problem that surfaces as deployment-side confusion. The dataset format is worth checking before you blame the checkpoint.
• GR00T N1.5, GR00T N1.7 and Pi0.5 are cloud-only on this platform. SmolVLA and ACT also run locally, which makes them the reasonable first target if you want the whole loop on your desk.
• GR00T's fine-tuning entry point exposes no seed, so two GR00T runs on identical data are not bit-for-bit identical. Do not chase a deployment difference that is really run-to-run variance.

Frequently asked questions

The short version

Getting a checkpoint to move an arm is four questions in order. Did the whole checkpoint arrive, config and statistics included? Does the observation dictionary the robot produces use the names the checkpoint expects? Does the checkpoint predict sensible actions on an episode it has already seen? And does the loop close fast enough for the task? Only the last one is about hardware. The first three are file paths and key names, which is why this is a tedious problem rather than a hard one.

If you want the surrounding context, the complete SO-100 guide covers assembly through training, collecting high-quality VLA training data covers the recording side that decides whether deployment can work at all, and the VLA overview covers why these models expect a language instruction in the first place. For the models themselves, the policy comparison has the parameter counts and latencies side by side, and the GR00T N1.7 on SO-100 guide is the training run that produces the checkpoint this page starts from.