Renting GPUs for Robot Policy Training: The Spot Market

A policy fine-tune is an odd workload to hand a cloud GPU. It runs for hours, not weeks. It is bursty: one run, a look at the result on the arm, another run three days later. And it survives being killed halfway through, as long as somebody wrote a checkpoint to disk. Those three properties are what the spot GPU market was built around, which is why a GR00T N1.7 fine-tune costs about as much as lunch.

This is about the market, not the model: where the hourly number comes from, what "interruptible" means on the three providers people use (three different things), which parts of the bill are not the GPU, and the trainer flags that decide whether an interruption costs ninety seconds or the whole run. Upstream figures were read on 2026-08-24.

Three providers, three different meanings of "spot"

People say "spot GPU" as if it were one product. It covers at least three market designs. Vast.ai still runs a real auction: clients set a bid that determines priority, and the docs state that "the current highest bid determines the instance that runs; any others are paused". On-demand rentals outrank every bid on the same machine. The important word is paused: the container stops, the disk stays, and the disk keeps billing.

Runpod took the opposite route. Its create-pod REST body has no bid field at all, only a boolean, interruptible, producing a pod that "can be rented at a lower cost but can be stopped at any time". No price signal, no notice. The runpodctl CLI is candid about the outcome: its status table lists stopped_outbid and terminated_outbid as "the only involuntary stop with a real recorded cause". Everything else surfaces as stopped_by_runpod with no machine-readable reason.

AWS abandoned bidding years ago. Spot prices are now "set by Amazon EC2" and move with long-term supply and demand rather than with your bid, at a stated "up to 90% off compared to On-Demand". In exchange you get the best warning in the business, a notice issued "two minutes before Amazon EC2 stops or terminates your Spot Instance".

# EC2 only: poll the interruption notice from inside the instance
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/spot/instance-action
# -> {"action": "stop", "time": "2017-09-18T08:22:00Z"}   (404 when nothing is pending)

What an A100 hour actually costs

Published on-demand rates are the anchor, because interruptible pricing floats below them. These are list prices read from the providers' own pages on 2026-08-24: Runpod's Secure Cloud, and Lambda's public GPU cloud.

Halve the marketplace numbers for interruptible and you land close to what AY-Robots quotes: 1.20 to 2.00 USD per hour on the A100 80 GB and H100 tier, 0.30 to 0.60 USD per hour on the 24 GB tier. Over a run that is 4 to 12 USD for GR00T N1.7, GR00T N1.5 or Pi0.5 across 3 to 6 hours, and 1 to 3 USD for SmolVLA or ACT across 2 to 5 hours.

One line in that table decides most of your bill. The 3B models need an 80 GB card and will not fit on the 0.27 USD/h A5000 however patient you are. That hard VRAM floor is why the policy comparison lists a GPU tier per model. Force it anyway and you get the symptom on the out-of-memory page.

Three lines on the bill that are not the GPU

Storage keeps billing when the card does not

Runpod publishes this openly: container disk 0.10 USD per GB per month, volume disk 0.10 running and 0.20 idle, network storage 0.07 under 1 TB. Read that again. The idle rate is double the running rate, and the docs say in words that you are still charged for volume disk storage while stopped.

Vast.ai works the same way with less signposting: storage is "billed continuously while your instance exists, regardless of running state", rates vary by host, and only deleting the instance stops the meter. The per-host rate sits in the search output as storage_cost, in USD per GB per month, if you sort on it.

Bandwidth, and the download tax before step 0

Runpod says pods are "billed by the second for compute and storage, with no fees for data ingress or egress". Vast.ai does charge: inet_up_cost and inet_down_cost are per-host fields in USD per GB. Either way there is a larger cost nobody bills you directly, which is the GPU time spent downloading weights. Repository sizes from the Hugging Face API, 2026-08-24:

Ten minutes of an H100 at 3.29 USD per hour is 55 cents spent before a single gradient is computed, and you pay it again on every fresh instance an interruption pushes you onto. Your LeRobot dataset rides on top. That is the argument for a persistent volume even when you are being frugal: cache the weights once.

The clock that starts before training does

Billing starts at instance creation, not at step 0. Image pull, dependency install, dataset download, format conversion, then training. On the A100 tier that preamble is routinely fifteen to twenty minutes. Bake it into the image rather than typing it while the meter runs. If a job never gets that far, the stuck-in-queued page covers what queueing means here: nobody took your bid.

The idle card: four ways to pay for nothing

1. The run finished and you never destroyed the instance. On Vast.ai storage bills for as long as the instance exists. On Runpod the stopped pod bills its volume at the doubled idle rate.
2. You stopped instead of terminating, to keep the environment. Runpod's docs warn that restarting gives no guarantee of GPU allocation and you may be assigned zero GPUs if capacity has changed. You paid to hold a disk and lost the card anyway.
3. You debugged on the expensive card. A wrong dataset path, a v3.0 dataset the GR00T loader refuses, a CUDA mismatch: two minute fixes billed at the training rate. Reproduce them on a 0.27 USD/h A5000 first, and see the v3.0 rejection page for the conversion case.
4. You ran out of balance mid-run. Runpod stops your pods at zero balance and terminates the ones without a network volume. Indistinguishable from an outbid, except there is nothing left to resume from.

The honest version of "spot GPUs are cheap" is that the compute is cheap and the discipline is the expensive part. The inference side of this platform solves it with an idle watchdog: pods provisioned by the pod API destroy themselves after an idle period. A hand-rented card has no such thing unless you write it.

Making a run survive an interruption

Interruptible training reduces to two questions: how many training steps can you afford to lose, and can you get back on the horse. The first is checkpoint cadence, the second is the resume path. Both are trainer defaults, and none of them were written with spot cards in mind.

1. 1
   Measure your step rate first

   Let the run reach steady state and read steps per second from the log. Pick a cadence that lands a checkpoint every 10 to 15 minutes. That is the work you are agreeing to lose.

   bash

   # a 3B model on one A100 at roughly 1.5 steps/s
   # 10 minutes of work = 1.5 * 600 = ~900 steps -> round down and use it

2. 2
   Set the cadence explicitly

   None of the three defaults suits an interruptible card. Override on the command line, not in a config file you will forget.

   bash

   # lerobot: ACT or SmolVLA on a 24 GB card
   lerobot-train \
     --dataset.repo_id=$HF_USER/so100_pick_place \
     --policy.type=act \
     --output_dir=/workspace/outputs/act_run1 \
     --job_name=act_so100 \
     --steps=100000 \
     --save_freq=2000 \
     --policy.device=cuda
   
   # Isaac-GR00T: SAVE_STEPS is an env var read by examples/finetune.sh
   SAVE_STEPS=250 MAX_STEPS=20000 bash examples/finetune.sh \
     --base-model-path nvidia/GR00T-N1.7-3B \
     --dataset-path /workspace/data/so100_pick_place \
     --embodiment-tag new_embodiment \
     --output-dir /workspace/outputs/groot_run1

3. 3
   Get the checkpoint off the instance

   A checkpoint on a paused Vast.ai instance is only as safe as the instance, and on a terminated Runpod pod without a network volume it is gone.

   bash

   # background sync loop, started from the on-start command
   while true; do
     aws s3 sync /workspace/outputs s3://my-bucket/runs/ \
       --exclude '*/wandb/*' --only-show-errors
     sleep 300
   done &

4. 4
   Resume onto a fresh instance

   lerobot rebuilds the config from the checkpoint's own train_config.json, so you pass a path and a flag. Isaac-GR00T takes a bare switch mapping onto the Hugging Face Trainer.

   bash

   # lerobot: point at the checkpoint's own train_config.json
   lerobot-train \
     --config_path=/workspace/outputs/act_run1/checkpoints/last/pretrained_model/train_config.json \
     --resume=true
   
   # Isaac-GR00T: the same command as before, plus one flag
   SAVE_STEPS=250 MAX_STEPS=20000 bash examples/finetune.sh \
     --base-model-path nvidia/GR00T-N1.7-3B \
     --dataset-path /workspace/data/so100_pick_place \
     --embodiment-tag new_embodiment \
     --output-dir /workspace/outputs/groot_run1 \
     --resume-from-checkpoint

5. 5
   Destroy the instance yourself

   Do not rely on the run ending cleanly, and do not rely on stopping. Destroy is the only state in which storage billing stops on both marketplaces.

   bash

   vastai destroy instance 9876543
   runpodctl pod delete <pod_id>

The save_only_model trap

Isaac-GR00T's finetune.sh exposes --save-only-model, which maps to the Hugging Face Trainer argument of the same name: it saves "only model weights, not optimizer/scheduler/RNG state", which "prevents resuming training from the checkpoint". Use it for the final artifact you ship to inference, never for the intermediate checkpoints you are relying on.

Picking an offer on Vast.ai without reading four hundred rows

The marketplace is heterogeneous in ways that matter more than the headline price. Two offers at the same rate can differ threefold in download speed, and an unreliable host costs more in re-downloads than it saves. The query language exposes all of it.

pip install vastai
vastai set api-key $VAST_API_KEY

# interruptible offers only, one 80 GB card, fast link, reliable host
vastai search offers --type bid \
  'gpu_name=A100_SXM4 gpu_ram>=79 num_gpus=1 rentable=true verified=true \
   inet_down>=200 reliability>0.98 duration>=1' \
  --storage 60 \
  -o 'min_bid'

Five fields deserve attention before the price. min_bid is the current minimum bid in USD per hour for interruptible, so it is the real floor. reliability is the host's machine reliability score. duration is the maximum rental in days, and a host offering less than your run length cuts you off on schedule regardless of bids. storage_cost and inet_down_cost are the two invisible lines from the previous section, and dlperf_usd is performance per dollar.

# create an interruptible instance with a per-machine bid
vastai create instance 1234567 \
  --image pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime \
  --disk 60 \
  --bid_price 0.90 \
  --ssh --direct \
  --onstart-cmd "bash /workspace/run_finetune.sh"

# raise the bid if you keep getting paused; with no --price,
# vast picks a currently winning bid for you
vastai change bid 9876543 --price 1.20

# the only command that stops storage billing
vastai destroy instance 9876543

Runpod's equivalent is shorter because there is less to decide. The create-pod body defaults nearly everything the spec documents: cloudType SECURE, gpuCount 1, containerDiskInGb 50, volumeInGb 20, and interruptible false.

curl -X POST https://rest.runpod.io/v1/pods \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "imageName": "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04",
    "gpuTypeIds": ["NVIDIA H100 80GB HBM3"],
    "gpuCount": 1,
    "interruptible": true,
    "containerDiskInGb": 50,
    "volumeInGb": 100
  }'

# or from the CLI, which reports outbid stops honestly
runpodctl config --apiKey=$RUNPOD_API_KEY
runpodctl pod get <pod_id>     # runtimeStatusReason: stopped_outbid
runpodctl pod delete <pod_id>

Doing it yourself versus letting the platform rent the card

The manual path is viable and worth doing once, if only to see what the abstraction hides. Same goal on both sides below: a fine-tuned policy you can run on an SO-100, trained on interruptible capacity.

# minimum viable spot wrapper: resume if a checkpoint exists
if [ -L /workspace/outputs/run1/checkpoints/last ]; then
  lerobot-train \
    --config_path=/workspace/outputs/run1/checkpoints/last/pretrained_model/train_config.json \
    --resume=true
else
  lerobot-train --dataset.repo_id=$HF_USER/my_dataset \
    --policy.type=smolvla --output_dir=/workspace/outputs/run1 \
    --job_name=smolvla_so100 --save_freq=2000 --policy.device=cuda
fi

Where a cheap GPU does not help at all

Worth saying plainly, because spot economics make it tempting to treat compute as the bottleneck. It usually is not.

• Bad data stays bad. Thirty episodes with an inconsistent camera position produce a broken policy on an H100 as reliably as on a 4090. The minimum episode counts here are 50 for GR00T N1.7, GR00T N1.5, Pi0.5 and ACT, and 30 for SmolVLA, and those are floors rather than targets. Collecting data worth training on is the harder half of the job.
• Renting a GPU for inference is a different problem. Per action step these models cost 20 ms for ACT, 152 ms for GR00T N1.7, 165 ms for GR00T N1.5, 245 ms for SmolVLA and 485 ms for Pi0.5. Public-internet round trips on top turn a working policy into a hesitant one, so remote inference suits slow pick-and-place, not fast reactive motion.
• Spot pricing cannot buy VRAM. No bid gets a 3B model onto a 24 GB card. If your budget caps out there, pick a smaller model rather than a cheaper A100. GR00T N1.7 against SmolVLA is the honest version of that trade.

Which vision-language-action architecture is worth the 80 GB card at all is covered in the model overview, with the benchmark numbers in the arena. If you are still deciding whether to build an arm, the SO-100 setup guide is the place to start.

A worked budget for one GR00T N1.7 run

One fine-tuning run of GR00T N1.7 on an SO-100 dataset, on interruptible capacity. Platform figures from the pricing page; upstream figures read on 2026-08-24.

The pattern worth noticing: the GPU line is the one you control least and the storage line is the one you control entirely. People spend an hour shaving 20 cents off the rate, then leave a volume running for a month. Both the billing docs and the try page assume the same thing, that for a first working policy the compute is not what will stop you.