Cutting the Cost of Robot Policy Training: 5 Levers

A fine-tuning run is one of the few line items in robot learning with an honest price tag. You can look the rate up before you spend it: on 2026-08-24, RunPod's public pricing page lists an A100 80 GB PCIe at 1.19 USD per hour on the community tier and 1.39 USD on secure cloud, an RTX 4090 at 0.34 USD community, and an H100 PCIe at 1.99 USD community. Hugging Face Jobs bills its a100-large flavour at 2.50 USD per hour. Those rates are set by a market you do not control.

The hours are different. The hours come out of decisions you make in a form: which policy you picked, how many training steps you asked for, whether the dataset was ever capable of training anything, and whether you remembered to destroy the pod. This article is about those decisions, because that is where the money is. The published rates per tier are on the pricing page, and what the platform bills against is described in the billing documentation.

A run is two numbers multiplied, and only one is yours

The arithmetic is not complicated, which is exactly why it is worth writing down. Every optimisation below is an attack on one of these three terms.

run_cost      = wall_clock_hours * gpu_rate_per_hour
wall_clock_hours = setup + (steps * seconds_per_step) / 3600

# Anchor: lerobot's own SmolVLA doc states that 20k steps
# takes "roughly ~4 hrs on a single A100 GPU" at batch_size=64.
seconds_per_step = 14400 / 20000 = 0.72   # includes data loading

# That run on RunPod's community A100 80 GB (1.19 USD/h, listed 2026-08-24):
4.0 h * 1.19 USD/h = 4.76 USD    # plus setup, plus the dataset download

Two things fall out of that. First, setup is not free: pulling a 3 B base checkpoint and a few gigabytes of video onto a fresh pod happens on a clock that is already running. Second, steps is the only term you set directly, and it is the one people copy from a README without thinking.

On AY-Robots the whole run lands in one of two bands, because the backend rents the card by required VRAM. The GR00T N1.7, GR00T N1.5 and Pi0.5 tier runs 3 to 6 hours at 1.20 to 2.00 USD per hour, about 4 to 12 USD per run. The SmolVLA and ACT tier runs 2 to 5 hours at 0.30 to 0.60 USD per hour, about 1 to 3 USD per run. Both are small numbers. Cost only becomes a problem because almost nobody gets a working policy on run one.

Lever 1: stop paying foundation-model prices for a fixed-camera task

The gap between the two tiers is roughly four times the money, and a longer wait for every attempt. It exists because a vision-language-action model with 3 B parameters cannot be fine-tuned on a 24 GB card. NVIDIA's own hardware guide for Isaac-GR00T puts the fine-tuning minimum at 1 GPU with 40 GB+ VRAM and notes that the default configuration, which tunes only the projector and the diffusion action head, keeps peak VRAM under about 35 GB per GPU.

So the first question is not "which model is best" but "does my task need what the big model brings". What it brings is web-scale semantics: language conditioning that survives rephrasing, and some tolerance for objects it has not seen. If your task is one bin, one camera rig, one object type and one instruction, you are paying for generalisation you will never exercise. ACT's original paper reports 80 to 90 percent success on 6 real-world fine manipulation tasks from "only 10 minutes worth of demonstrations", with no pretraining at all. The SmolVLA paper makes the same argument from the other side: the model is "designed to be trained on a single GPU and deployed on consumer-grade GPUs or even CPUs", with performance its authors report as "comparable to VLAs that are 10x larger". Whether that holds for your task is an empirical question, and it is a cheap one to ask.

Lever 2: max_steps is a ceiling somebody picked, not a target

Upstream defaults are chosen so that a demo dataset produces a demo video. They are not sized for your 50 episodes. Read against the source on 2026-08-24: Isaac-GR00T's FinetuneConfig ships max_steps: int = 10000 with global_batch_size: int = 64 and save_steps: int = 1000. lerobot's TrainPipelineConfig ships steps: int = 100_000 with batch_size: int = 8 and save_freq: int = 20_000.

Notice that the platform's GR00T N1.7 default of 20000 steps at batch 32 pushes the same number of samples through the model as NVIDIA's 10000 steps at batch 64. It is the same amount of work, split differently. The ACT default of 100000 steps at batch 8 is lerobot's own default, unchanged. Neither is a claim that your dataset needs that many steps.

The practical move is to shorten the checkpoint interval rather than the run. A run that writes a checkpoint every 1000 steps can be killed the moment the loss curve flattens, and you keep everything up to that point. A run that writes one every 20000 steps has to be watched to the end or thrown away.

# lerobot: save often, log often, and hold out a slice so you can see the plateau
lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=${HF_USER}/mydataset \
  --dataset.eval_split=0.1 \
  --eval_steps=2000 \
  --batch_size=64 \
  --steps=20000 \
  --save_freq=2000 \
  --log_freq=200 \
  --output_dir=outputs/train/my_smolvla \
  --job_name=my_smolvla_training \
  --policy.device=cuda \
  --wandb.enable=true

Lever 3: the run that was already dead at step 0

This is the expensive one, and it is expensive twice: you pay for the GPU hours and you pay for the evening. A run on a dataset that could never have trained anything still bills at full rate, and unless you set up a held-out split, the loss curve will look plausible while it does it.

There are four common ways a dataset kills a run. The format is wrong: a LeRobot dataset in v3.0 crashes the GR00T loader, which wants v2.0 or v2.1. The metadata is wrong: a partial Hugging Face snapshot leaves episode entries in meta/episodes.jsonl that point at files which were never downloaded. The camera keys do not match the modality config the trainer expects. Or the episodes themselves are unusable, which no schema check will catch: the recording-side quality rules decide that long before the trainer ever sees the data.

1. 1
   Convert the dataset before you rent anything

   Isaac-GR00T ships a converter for exactly this. It lives in its own directory with its own pyproject.toml, and the README is explicit that you must install from that directory rather than the repo root.

   bash

   cd scripts/lerobot_conversion
   uv venv && source .venv/bin/activate
   uv pip install -e . --verbose
   python convert_v3_to_v2.py --repo-id <hf_user>/<dataset>

2. 2
   Repair broken episode metadata

   The repair script drops broken episode entries from meta/episodes.jsonl, updates the summary fields in meta/info.json and regenerates stats. The dataset path is positional and --dry-run reports the damage without writing. Running it on a laptop costs nothing. Discovering the problem on a rented A100 costs an hour.

   bash

   python scripts/repair_lerobot_metadata.py /path/to/dataset --dry-run
   python scripts/repair_lerobot_metadata.py /path/to/dataset

3. 3
   Do a 50-step smoke run on the cheap tier

   Before a 20000-step run on an 80 GB card, run 50 steps of ACT or SmolVLA locally or on a 24 GB card. It exercises the loader, the camera keys, the normalisation statistics and the checkpoint writer. Everything that fails at step 0 fails here for free.

   bash

   lerobot-train \
     --policy.type=act \
     --dataset.repo_id=${HF_USER}/mydataset \
     --steps=50 \
     --save_freq=25 \
     --batch_size=2 \
     --output_dir=outputs/smoke \
     --job_name=smoke

4. 4
   Watch the first checkpoint, not the last

   If the first checkpoint at step 1000 or 2000 produces nothing but a mean action, kill the run. The remaining 18000 steps will not repair a dataset. Send the money to a second recording session instead.

Lever 4: the card you stopped and the disk you forgot

Stopping is not destroying, and every marketplace documents this in a place people do not read until the invoice arrives. Vast.ai's billing FAQ states it plainly: "You are charged the storage cost (which depends on the size of your storage allocation) for every second your instance exists and is online, regardless of what state it is in: active, inactive, loading, etc." Their pricing guide adds: "Storage charges continue even when instances are stopped. Delete instances completely to cease storage billing."

RunPod goes one step further and charges more for the idle case. Their pricing documentation lists volume disk at 0.10 USD per GB per month while the pod runs and 0.20 USD per GB per month while it is stopped. Container disk is not charged on a stopped pod; the volume is. And if the balance runs out, pods without a network volume are terminated and their data cannot be recovered, which is how people lose checkpoints they were paying to keep.

If you do rent the card yourself, the discipline is short: write checkpoints to object storage during the run rather than to the pod's volume, verify the objects landed, then destroy the instance rather than stopping it. "I will look at it tomorrow" is how a 200 GB volume ends up costing 40 USD for the month nobody looked at it.

Lever 5: do not buy the same answer twice

Two upstream defaults quietly make repeat runs likely. Isaac-GR00T's resume_from_checkpoint defaults to False, documented in the source as "Default False so a rerun against an existing output_dir starts fresh instead of silently merging with a previous experiment". That is the right default for correctness and the wrong one for your wallet if you assumed a rerun would continue. And save_only_model, which looks like a tidy way to save disk, carries the note "Cannot resume training from these checkpoints".

# GR00T: resume rather than restart, and keep the optimizer state so you can
uv run python gr00t/experiment/launch_finetune.py \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path /data/my_so100_dataset \
  --embodiment-tag NEW_EMBODIMENT \
  --modality-config-path examples/SO100/so100_config.py \
  --output-dir ./outputs/run_a \
  --max-steps 20000 \
  --save-steps 1000 \
  --save-total-limit 20 \
  --resume-from-checkpoint    # tyro turns the bool field into this flag

The same logic applies to hyperparameter sweeps. On a 4 to 12 USD run, a five-point learning-rate sweep is 20 to 60 USD, and on a 50-episode dataset it will mostly tell you about noise. Change one thing at a time, and change the thing with the largest effect first: more episodes beats a better learning rate almost every time.

Doing it yourself against doing it here

Where a cheaper run does not help at all

Three bills are unaffected by anything in this article, and it is worth naming them so the optimisation stops at the right place.

That last row is the honest limit of the whole cloud-training story. Training happily lives in a data centre. Inference latency does not: for fast tasks the policy has to sit next to the servos. This is the same trade-off examined in the general VLA overview, and the practical consequences are in the training documentation. The recording hours in the row above are the same problem seen from the other end: teleoperation time is the one input to a policy that no amount of GPU discount reduces.

A worked budget for a first working policy

Assume an SO-100, one task, and the ordinary case where the first attempt does not work. This is the sequence that gets there for the least money, with the platform's own cost bands.

The reason this ordering is cheap is not that any single run is cheap. It is that steps 2 and 3 remove most of the reasons a 12 USD run fails, before you buy it. If you want the guided version of the same sequence, the first-policy walkthrough runs through it, and the per-model guides such as SmolVLA on the SO-100 and ACT on the SO-100 carry the exact defaults for each combination.