Multi-GPU Fine-Tuning for VLA Models: When It Helps

Four GPUs, quarter of the wall clock. For a fine-tune on 50 teleop episodes of an SO-100, that is almost never what you get, and the reason is not physics. The flag called batch size means opposite things in the two trainers most people reach for. In LeRobot it is per worker, so adding GPUs multiplies your effective batch. In NVIDIA's Isaac-GR00T N1.7 it is global, so adding GPUs divides the work and leaves the effective batch where it was. Same word, same number, inverted result.

This page describes what the code does today, with the file it came from and the release it shipped in, because both projects moved their multi-GPU story in 2026 and most tutorials online describe the older one. Then the three failures that turn a distributed run into a lost day: a batch that no longer means what you think, a learning rate nobody scaled, and a collective that hangs until NCCL gives up. It ends with the honest part, which is that for a 50-episode policy on a low-cost arm, the second GPU is usually the wrong thing to buy.

What your trainer means by batch size

Before any distributed flag, settle what the number in your config controls. There are three conventions in circulation among VLA trainers, and mixing them up is the single most common way a multi-GPU run produces a worse checkpoint than the single-GPU run it replaced.

The GR00T N1.7 line is the one that surprises people. In gr00t/experiment/experiment.py the per-device batch is literally global_batch_size // num_gpus, and a startup check in warn_configs asserts that the division is exact. Going from one H100 to eight does not train on more data per step. It trains the same 64 samples with 8 on each card, a throughput change and nothing else. To get the larger batch too, raise --global-batch-size yourself, which is what NVIDIA's hardware guide does when it pairs 4 to 8 cards with a global batch of 64 to 640.

LeRobot (main, Aug 2026)
  effective_batch = batch_size x dp_world_size x gradient_accumulation_steps
  # --batch_size=8, 4 workers, --accelerator.gradient_accumulation.steps=2  ->  64

Isaac-GR00T N1.7
  per_device_train_batch_size = global_batch_size // num_gpus     # experiment.py
  accumulated_batch_size      = global_batch_size x gradient_accumulation_steps
  # --global-batch-size 64 --num-gpus 8  ->  8 per card, 64 per update

Hugging Face Trainer (what GR00T sits on)
  effective_batch = per_device_train_batch_size x num_devices x gradient_accumulation_steps

openpi (JAX)
  --fsdp-devices N  shards parameters over N devices; the batch you configured stays the batch

When a second GPU actually helps

There are three separate reasons to add a card, and only one of them is speed. Being clear about which one you are chasing decides whether you want plain data parallelism, sharding, or neither.

Mapped onto the five policies you can train for an SO-100 class arm, the picture is lopsided. Only the ~3 B models sit in a regime where distribution changes the answer, and even there the first question is memory, not speed. The policy comparison page has the params, GPU tier and inference latency for each. The column that matters here is the middle one.

The learning rate and the step count, which nothing scales for you

Both trainers leave this to you, and LeRobot puts it in bold: it does not auto-scale the learning rate or the number of steps when the effective batch grows, and suggests that with 2 GPUs you either double --optimizer.lr or halve --steps. That is the linear scaling rule from Goyal et al.: multiply the learning rate by the same factor as the minibatch, and add a warmup so the first few hundred training steps do not blow up.

• If the effective batch grew by k, the linear rule says multiply the learning rate by k and keep a warmup. GR00T's warmup_ratio default is 0.05 with a cosine scheduler.
• If you would rather not touch the learning rate, divide the step count by k instead. Same samples seen, same schedule shape, fewer updates.
• Do not do both. Doubling the learning rate and halving the steps is a different run than either one, and it is the version people accidentally ship.
• Check the noise scale argument before scaling far. McCandlish et al. show that past a critical batch size, predicted by the gradient noise scale, extra batch buys wall clock and stops buying data efficiency. Its measured domains are MNIST, SVHN, CIFAR-10, ImageNet, Billion Word, Atari, Dota and autoencoders, none of them robot imitation, so on 50 episodes treat the threshold as unmeasured rather than known.

The strongest evidence that bigger is not automatically better comes from NVIDIA's own numbers. Table 6 of the GR00T N1 paper lists pretraining and post-training side by side: the batch drops by more than an order of magnitude while the learning rate does not move at all. The stated reason is one line, smaller batches in post-training to avoid overfitting when fine-tuning in data-limited settings.

The manual path, with the commands that work today

Both sequences were read out of the upstream repositories in August 2026. Pin your version before copying anything: the flag names below are younger than most blog posts about them.

LeRobot: torchrun or accelerate launch, then DDP or FSDP

1. 1
   Install the training extra

   accelerate ships inside LeRobot's training extra. It is used as a plain launcher only; every distributed setting lives in LeRobot's own config system.

   bash

   pip install 'lerobot[training]'

2. 2
   Launch plain DDP across two GPUs

   With no --parallelism.* flags, a multi-process launch is plain DDP: the full model is replicated on every card. --batch_size stays per worker, so this run sees 16 samples per step, not 8.

   bash

   torchrun --nproc-per-node=2 $(which lerobot-train) \
     --dataset.repo_id=${HF_USER}/my_dataset \
     --policy.type=act \
     --policy.repo_id=${HF_USER}/my_trained_policy \
     --batch_size=8 \
     --output_dir=outputs/train/act_multi_gpu \
     --job_name=act_multi_gpu \
     --wandb.enable=true

3. 3
   Compensate for the batch you just doubled

   Pick one of the two adjustments, not both. Doubling the learning rate keeps the update size comparable; halving the steps keeps the sample budget comparable.

   bash

   # option A: linear learning-rate scaling
   torchrun --nproc-per-node=2 $(which lerobot-train) --optimizer.lr=2e-4 ...
   
   # option B: same learning rate, half the steps
   torchrun --nproc-per-node=2 $(which lerobot-train) --batch_size=8 --steps=50000 ...

4. 4
   Shard instead of replicate when the model will not fit

   FSDP2 shards parameters, gradients and optimizer state. --parallelism.dp_shard=-1 shards over however many processes the launcher started. Mixed precision defaults to no, so ask for bf16 explicitly or you will train the shards in fp32.

   bash

   torchrun --nproc-per-node=4 $(which lerobot-train) \
     --dataset.repo_id=${HF_USER}/my_dataset \
     --policy.type=<your_policy> \
     --parallelism.dp_shard=4 \
     --accelerator.mixed_precision=bf16 \
     --output_dir=outputs/train/my_policy_fsdp

5. 5
   Give FSDP a wrap unit if the policy does not declare one

   Policies declare their wrap units on the class: ACT declares ["ACTEncoderLayer", "ACTDecoderLayer"]. If a policy declares nothing and you pass nothing, the run fails at startup rather than wrapping only the root module and giving up every byte of the saving.

   bash

   --accelerator.fsdp.wrap_modules='["MyTransformerBlock"]'   # explicit class names
   --accelerator.fsdp.min_num_params=1000000                  # or wrap anything above 1M params

Isaac-GR00T N1.7: you drive torchrun, and you tell it the count twice

1. 1
   Clone with submodules and set up the environment

   GR00T pins dependencies through uv and needs git-lfs for the demo parquet files. The tested dGPU stack is Python 3.12, CUDA 12.6+, PyTorch 2.7+.

   bash

   sudo apt install git-lfs && git lfs install
   git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T
   cd Isaac-GR00T

2. 2
   Run the single-GPU baseline first

   Get a working run on one card first. It is the reference the multi-GPU run has to match, and the run whose loss curve tells you whether the dataset is the problem.

   bash

   CUDA_VISIBLE_DEVICES=0 uv run python \
       gr00t/experiment/launch_finetune.py \
       --base-model-path nvidia/GR00T-N1.7-3B \
       --dataset-path demo_data/cube_to_bowl_5 \
       --embodiment-tag NEW_EMBODIMENT \
       --modality-config-path examples/SO100/so100_config.py \
       --num-gpus 1 \
       --output-dir /tmp/test_finetune \
       --max-steps 2000 \
       --global-batch-size 32 \
       --dataloader-num-workers 4

3. 3
   Scale out with torchrun, keeping the two counts equal

   --nproc_per_node and --num-gpus must match or the run raises before loading a frame. And --global-batch-size 32 here means 4 samples per card on 8 GPUs, not 32.

   bash

   uv run torchrun --nproc_per_node=8 --master_port=29500 \
       gr00t/experiment/launch_finetune.py \
       --base-model-path nvidia/GR00T-N1.7-3B \
       --dataset-path demo_data/cube_to_bowl_5 \
       --embodiment-tag NEW_EMBODIMENT \
       --modality-config-path examples/SO100/so100_config.py \
       --num-gpus 8 \
       --output-dir /tmp/test_finetune_8gpu \
       --max-steps 2000 \
       --global-batch-size 32 \
       --dataloader-num-workers 4

4. 4
   Use accumulation to buy batch you do not have cards for

   NVIDIA's hardware guide gives the arithmetic directly: 4 GPUs, 8 accumulation steps and a per-GPU batch of 8 gives an effective global batch of 256. In GR00T that is --global-batch-size 32 --gradient-accumulation-steps 8, and the trainer logs the resulting accumulated batch at startup.

   bash

   uv run torchrun --nproc_per_node=4 gr00t/experiment/launch_finetune.py \
       --num-gpus 4 \
       --global-batch-size 32 \
       --gradient-accumulation-steps 8 \
       --max-steps 10000 \
       --save-steps 1000

Two routes to the same checkpoint

The distributed failures that cost a day

Single-GPU training fails loudly. Distributed training fails quietly, or ten minutes later in a stack trace that names a collective and nothing about your code. These are the ones that show up on VLA fine-tunes, with what each actually is.

# Ask NCCL what it is doing. INFO is usually enough; TRACE is very loud.
export NCCL_DEBUG=INFO

# Two GPUs on a host where peer-to-peer is broken or virtualised badly:
# force the fallback path and see whether the hang disappears.
export NCCL_P2P_DISABLE=1

# Multi-node on a box with several NICs: pin the interface NCCL is allowed to use.
export NCCL_SOCKET_IFNAME=eth0

# No InfiniBand, or an IB stack that is present but not usable:
export NCCL_IB_DISABLE=1

# Then reproduce on the smallest possible run.
torchrun --nproc-per-node=2 $(which lerobot-train) --steps=50 --batch_size=2 ...

Two rules make these cheap. First, never debug a distributed run at full scale: reproduce with 50 steps and a batch of 2, because a hang at step 50 costs a minute and a hang at step 5000 costs an hour. Second, if the single-GPU run of the same config was never green, stop; you are debugging your dataset through a distributed launcher, the worst available microscope. The failure-mode index has the single-GPU versions of most of these, including a job that never leaves the queue and a loss that falls while the policy does nothing.

What moved upstream, and when

This is the part that dates fastest, so here is the state as of August 2026 with the release each change landed in. If a tutorial you are following uses a name from the left-hand column, it predates the current code.

The LeRobot YAML change breaks a habit. Older guides tell you to run accelerate config and answer the interactive questions. Current LeRobot refuses to start when accelerate environment variables are set, for a good reason: those variables configure the engine behind LeRobot's back, so the train_config.json saved next to your checkpoint would no longer describe the run that produced it. Put the settings in --parallelism.* and --accelerator.* flags instead.

Where more GPUs stop helping, including here

AY-Robots does not do multi-GPU training. The form picks model, dataset and hyperparameters, and the backend rents one card sized by the VRAM that model needs. If your plan is a 640-sample global batch across eight H100s, this is not the tool; rent a node and use the commands above. What the platform removes is the part that eats the day: environment setup, dataset format conversion, and checkpoint plumbing.

• For ACT and SmolVLA on 30 to 50 episodes, one 24 GB card finishes in 2 to 5 hours for 1 to 3 USD. There is no wall clock problem to solve.
• For GR00T N1.7 and Pi0.5, one 80 GB card finishes in 3 to 6 hours for 4 to 12 USD. Four cards buy back hours you were not going to spend watching it anyway.
• Multi-GPU pays when the model will not fit on the largest card you can get, when you run the same fine-tune dozens of times a week, or when you are pretraining rather than post-training. None of those describe a first SO-100 policy.
• What most often decides whether the policy works is upstream of all this: how many clean episodes you recorded and whether the cameras stayed consistent. That is covered in the data collection write-up.

One more limit, usually discovered after the training question is settled: distribution changes nothing about deployment. A trained vision-language-action model still runs one action step at a time, and on this platform that is 20 ms for ACT up to 485 ms for Pi0.5. Inference has to sit next to the servos for fast tasks; public-internet round trips turn a working policy into a hesitant one. Remote inference is viable for slow pick-and-place, not for fast reactive motion. For deployment numbers before you pick a model, the GR00T N1.7 against Pi0.5 comparison and the model arena with its 85 models and 332 benchmark results are the places to start. the VLA overview covers what these models do in the first place.

Questions people actually ask