Reproducibility in Robot Policy Training: Seeds and Variance

Three different things people call reproducibility

You train a policy on Monday, get 14 successes out of 20 attempts, and write down 70 percent. On Friday you train it again with the same dataset, the same config and the same command, and get 11 out of 20. Nothing changed. Before hunting for a bug, it is worth being precise about which kind of sameness you expected, because three different promises get filed under one word.

Only the third one matters for shipping a robot, and it is the only one you can realistically get for a vision-language-action model. Chasing the first is a common way to lose a week.

What each trainer actually does with a seed

The five policies you can train here come from two codebases that disagree about what a seed is for. ACT, SmolVLA and Pi0.5 go through lerobot. GR00T N1.7 and GR00T N1.5 go through NVIDIA's Isaac-GR00T. That split is the single most important thing to understand here.

lerobot: one seed, wired through everything

In lerobot, seed is a top-level field on TrainPipelineConfig, used for model initialization, dataset shuffling and the evaluation environments. Its default is 1000, not 0 and not 42. The training script calls set_seed once, seeding Python's random, NumPy, torch on CPU, torch on all CUDA devices, and accelerate.

# src/lerobot/configs/train.py (lerobot main, read 2026-08-24)
    # `seed` is used for training (eg: model initialization, dataset shuffling)
    # AND for the evaluation environments.
    seed: int | None = 1000
    # Set to True to use deterministic cuDNN algorithms for reproducibility.
    # This disables cudnn.benchmark and may reduce training speed by ~10-20 percent.
    cudnn_deterministic: bool = False

# src/lerobot/scripts/lerobot_train.py
    if cfg.seed is not None:
        set_seed(cfg.seed, accelerator=accelerator)
    device = accelerator.device
    if cfg.cudnn_deterministic:
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
    else:
        torch.backends.cudnn.benchmark = True
    torch.backends.cuda.matmul.allow_tf32 = True

Batch order is the part lerobot gets genuinely right. EpisodeAwareSampler derives a per-epoch seed with np.random.SeedSequence([seed, epoch]) and feeds it to a fresh torch.Generator. The source comment states the consequence directly: the order is a pure function of (seed, epoch), so every rank reproduces the same permutation without synchronizing an RNG, and resume is sample-exact. Restart from a checkpoint and you get the batches you would have got without the interruption.

# lerobot 0.6.x, ACT on an SO-100 dataset, seed made explicit
lerobot-train \
  --dataset.repo_id=${HF_USER}/so100_pick_place \
  --policy.type=act \
  --policy.device=cuda \
  --output_dir=outputs/train/act_seed1000 \
  --job_name=act_seed1000 \
  --seed=1000 \
  --batch_size=8 \
  --steps=100000

# same thing, second seed, separate output dir
lerobot-train \
  --dataset.repo_id=${HF_USER}/so100_pick_place \
  --policy.type=act \
  --policy.device=cuda \
  --output_dir=outputs/train/act_seed1001 \
  --job_name=act_seed1001 \
  --seed=1001 \
  --batch_size=8 \
  --steps=100000

Isaac-GR00T: the seed exists, but not for you

GR00T's fine-tuning entry point is gr00t/experiment/launch_finetune.py, a tyro CLI over the FinetuneConfig dataclass. That dataclass has fields for the base model path, dataset path, embodiment tag, which module to tune, augmentation parameters, batch size, learning rate and max steps. It has no seed field.

The seed is not absent from the codebase. One layer down, gr00t/experiment/experiment.py calls set_seed(config.data.seed) and passes seed=config.data.seed into the Hugging Face TrainingArguments, and DataConfig.seed defaults to 42. So GR00T runs are seeded, consistently, at 42, and the launcher gives you no way to change it. Varying it means dropping to launch_train.py and writing a full config, which is a different workflow from the documented one.

# Isaac-GR00T, single GPU, from the repo README (main, read 2026-08-24)
CUDA_VISIBLE_DEVICES=0 uv run python \
    gr00t/experiment/launch_finetune.py \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path demo_data/cube_to_bowl_5 \
    --embodiment-tag NEW_EMBODIMENT \
    --modality-config-path examples/SO100/so100_config.py \
    --num-gpus 1 \
    --output-dir /tmp/test_finetune \
    --max-steps 2000 \
    --global-batch-size 32 \
    --dataloader-num-workers 4

# there is no --seed. Confirm the current flag list yourself:
uv run python gr00t/experiment/launch_finetune.py --help

The things a seed cannot fix

A seed only controls pseudo-random number generators. It has no authority over the order in which a GPU accumulates partial sums, and floating point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bits. Any CUDA kernel using atomics to accumulate gradients therefore produces a slightly different result each run, seed or no seed. Over 20000 training steps those last bits become different weights, and different weights become a different end effector trajectory.

The atomics problem is not theoretical for VLA training. Every one of these models has a language backbone with an embedding table, and embedding backward on CUDA accumulates into the same rows from many tokens at once. PyTorch's list of normally-nondeterministic operations that use_deterministic_algorithms(True) makes deterministic includes scatter_add_ on CUDA, index_add on CUDA, index_select backward and gather backward. Those are exactly the ops a transformer's backward pass leans on.

# The hardening block, if you genuinely need bit-identical runs.
# Put this before you build the model, not after.
import random, numpy as np, torch

SEED = 1000
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed_all(SEED)

torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.backends.cuda.matmul.allow_tf32 = False   # lerobot turns this ON unconditionally
torch.use_deterministic_algorithms(True)        # raises on ops with no deterministic kernel

# DataLoader workers: PyTorch reseeds torch per worker, but not numpy or random
def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)

g = torch.Generator()
g.manual_seed(SEED)
loader = torch.utils.data.DataLoader(
    ds, batch_size=8, num_workers=4, worker_init_fn=seed_worker, generator=g
)

How much run-to-run variance to expect

One way to answer this is to reason about the codebase, which we just did. The other is to look at what people who scanned thousands of seeds found, and at what vendors put in their own READMEs.

Two conclusions follow. First, fine-tuning a pretrained foundation model on 50 episodes puts you closer to Picard's ImageNet case than his CIFAR-10 case, so the seed is not the dominant term. Second, the convention of 1 to 3 seeds means most published robot numbers lack the statistical support to distinguish themselves from each other, and neither will yours.

Evaluation noise is bigger than training noise

This is the part that surprises people arriving from a benchmark background. You can spend a week making training deterministic and still not tell two policies apart, because the measurement holds almost all the variance. A rollout either succeeds or it does not. That is a Bernoulli trial, and the standard error of a proportion over n trials is the square root of p(1-p)/n.

Read the second row again. Twenty rollouts is already more than most people run on a real arm, and it still leaves a standard error near 10 points. Score 14 out of 20 and the exact 95 percent confidence interval on the true success rate is 45.7 to 88.1 percent. Score 35 out of 50, the same 70 percent, and it narrows only to 55.4 to 82.1 percent. Your seed-to-seed difference of 5 or 6 points sits comfortably inside the noise of your own measurement.

This is a redirection of effort, not a counsel of despair. If measurement dominates, spend the budget there: standardise the scene, script the reset, use the same object positions in the same order for every checkpoint, and record the rollouts so a disputed success can be re-scored. Projects that take this seriously are explicit about the cost. RoboArena needed more than 600 pairwise real-robot episodes across seven institutions just to rank seven generalist policies against each other.

One cheap trick sidesteps rollout noise almost entirely: open-loop evaluation. Feed the policy observations from held-out episodes and measure the error between predicted and recorded actions. Isaac-GR00T ships gr00t/eval/open_loop_eval.py for exactly this, with per-joint plots and MSE. It costs no hardware time and takes the scene out of the measurement, but it is not itself deterministic: the flow-matching action head starts every prediction from a fresh torch.randn and neither the eval script nor Gr00tPolicy sets a seed, so run it more than once and compare averages. It also will not tell you whether the policy can do the task. Use it to compare seeds, not to decide whether to ship.

A protocol that survives contact with reality

None of this needs a research budget. It needs six things written down and left unchanged between runs.

1. 1
   Pin the dataset by revision, not by name

   A dataset that gained three episodes between Monday and Friday explains your whole score difference on its own. Record the exact commit and the format version, because a LeRobot dataset in v3.0 crashes the GR00T loader and has to be converted down to v2.1 first.

   bash

   # what you should be able to paste into the run log
   curl -s https://huggingface.co/api/datasets/${HF_USER}/so100_pick_place | jq -r '.sha'
   python -c "import json;print(json.load(open('meta/info.json'))['codebase_version'])"
   # -> v2.1  (GR00T)  or  v3.0  (ACT, SmolVLA, Pi0.5)

2. 2
   Pin the code, including the trainer

   Record the lerobot version or the Isaac-GR00T commit hash. Neither trainer stamps its own version into the checkpoint in a way you can rely on, so a run from three months ago is not comparable to one from today unless you wrote it down.

   bash

   pip show lerobot | grep -i version
   git -C Isaac-GR00T rev-parse --short HEAD
   python -c "import torch;print(torch.__version__, torch.version.cuda)"
   nvidia-smi --query-gpu=name,driver_version --format=csv,noheader

3. 3
   Run three seeds, not one

   For ACT and SmolVLA this is genuinely cheap: three runs on the 24 GB tier come to roughly 3 to 9 USD total. For GR00T you cannot vary the seed through the documented CLI, so run the same config three times and let the nondeterministic augmentations do the varying. Measuring that 5 to 6 percent on your own data is worth one afternoon.

   bash

   for S in 1000 1001 1002; do
     lerobot-train \
       --dataset.repo_id=${HF_USER}/so100_pick_place \
       --policy.type=smolvla \
       --policy.device=cuda \
       --seed=$S \
       --output_dir=outputs/train/smolvla_s$S \
       --job_name=smolvla_s$S
   done

4. 4
   Compare the seeds open-loop before touching the arm

   Run every checkpoint against the same held-out episodes and look at the action error. The Isaac-GR00T script below reads GR00T checkpoints; for lerobot policies you need the equivalent on that side. If three runs disagree wildly here, the problem is your dataset or your learning rate, not your luck, and no amount of rollout testing will fix it.

   bash

   # open_loop_eval.py loads a Gr00tPolicy, so it reads GR00T checkpoints.
   # GR00T has no settable seed, so these are plain repeats of one config.
   for R in 1 2 3; do
     uv run python gr00t/eval/open_loop_eval.py \
       --dataset-path /data/so100_holdout \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path outputs/gr00t_run$R/checkpoint-20000 \
       --traj-ids 0 1 2 \
       --execution-horizon 16
   done

5. 5
   Fix the evaluation scene before you fix the training

   Write down starting positions, lighting, camera mounts and reset procedure, then use exactly those for every checkpoint. An unstandardised scene adds more variance than every seed effect in this article combined, and the same discipline pays off upstream when you are collecting the training data in the first place. A policy that scores well in one setup and badly in another is a known failure mode: policy only works in one setup.

   text

   eval_protocol.md
     20 rollouts per checkpoint
     object start positions: A1 A2 A3 A4 A5, cycled, 4 passes
     lighting: overhead only, blinds closed
     reset: operator places object, hands clear, then start
     scored: success = object fully inside the bowl, hand released
     recorded: wrist cam + front cam, kept for 30 days

6. 6
   Report a range, and say how many trials it came from

   "78 percent" is not a result. "14/20, 12/20 and 16/20 across three seeds, so 60 to 80 percent" is a result, and it tells the next person whether your improvement is real. It is also the format that makes a difference between two checkpoints arguable rather than assertable.

   text

   smolvla, so100_pick_place @ sha 4f21ac9, lerobot 0.6.1, RTX 4090
     seed 1000 -> 14/20   (exact 95% CI 45.7 - 88.1%)
     seed 1001 -> 12/20   (exact 95% CI 36.1 - 80.9%)
     seed 1002 -> 16/20   (exact 95% CI 56.3 - 94.3%)
     pooled     42/60 = 70.0%  (exact 95% CI 56.8 - 81.2%)

Doing it yourself against doing it here

Which policy to pick if you care about this

Reproducibility is a real input to model selection, and one that people rarely weigh. If you plan to run many comparisons, being able to set a seed and re-run cheaply is worth more than a couple of points of headline performance.

ACT is the honest recommendation for methodology work. It is around 80 M parameters, trains from scratch rather than fine-tuning a foundation model, has an inference latency near 20 ms per action step, and needs 50 episodes minimum. Run it ten times, learn what your own noise floor looks like, then move to a larger model. The ACT against SmolVLA comparison, the five-policy overview and the background on flow-matching policies behind Pi0.5 cover the rest of the trade, and the model arena lists 85 VLA models with 332 benchmark results, each value linked to its source, which is itself an education in how loosely these figures compare.

What to do on Monday

Pick the cheapest model resembling your task, which for most people on an SO-100 means ACT or SmolVLA. Train it three times with three seeds and write the three numbers down. That is your noise floor, measured on your data and your arm, and it is what lets you interpret every experiment afterwards. Then standardise the evaluation scene and stop changing it. When you move to GR00T or Pi0.5, carry that floor with you as a lower bound: a bigger model on a rented card adds sources of variation rather than removing them. If a change moves your score by less than the floor, it did not move your score. The first-policy walkthrough, the training documentation and the failure-mode index cover the mechanics, and the VLA overview covers what these models are. This article covers deciding whether the mechanics worked.