The job reaches RUNNING and then fails with a CUDA out of memory message, either in the first seconds or after thousands of steps.
GR00T N1.7, GR00T N1.5 and Pi0.5 request the A100 80 GB tier; SmolVLA and ACT fit on a 24 GB card. If you lower the batch size to fit, raise gradient accumulation by the same factor so the effective batch stays put, and if you cannot, raise the step count, because a smaller batch means each step sees less data.
Where the memory goes
Three things share the card: the weights, the optimizer state that shadows them, and the activations held for the batch currently in flight. The first two are properties of the model you picked and you cannot negotiate with them. The third scales with how many samples you process at once, and that is the only knob in your hands. This is why the GPU tier is an attribute of the model in the catalog rather than something derived from your dataset.
| Model | GPU tier requested | Default batch | Accumulation | Effective batch |
|---|---|---|---|---|
| GR00T N1.7 | A100 80 GB | 32 | 1 | 32 |
| GR00T N1.5 | A100 80 GB | 1 | 16 | 16 |
| Pi0.5 | A100 80 GB | 1 | 16 | 16 |
| SmolVLA | RTX 4090, 24 GB | 2 | 8 | 16 |
| ACT | RTX 4090, 24 GB | 8 | 1 | 8 |
Read the middle rows carefully. GR00T N1.5 and Pi0.5 are already at batch size 1. There is no smaller batch. Those two recipes exist in that shape because a 3 billion parameter model with video observations leaves very little room on an 80 GB card, and the authors of the defaults spent the remaining headroom on accumulation instead. If you are hitting out of memory with one of them at defaults, lowering the batch is not available as a move.
The gap between the tiers is more than a factor of three, and the three billion parameter models are on the far side of it. There is no combination of batch size and accumulation that fine-tunes GR00T on a consumer card. If you need the 24 GB tier, you need a different model, not a different configuration.
Read when it died before you change anything
An out of memory in the first seconds means the configuration never fit. Nothing grew, nothing leaked, the first batch simply did not have room. An out of memory after thousands of successful steps means something about one particular batch was larger than the ones before it, and on video data that usually means episodes of uneven length or resolution in the same set.
// MCP get_training_job on a run that died at step 0
{
"id": "job_9e4471",
"modelType": "smolvla",
"status": "FAILED",
"configJson": {
"batch_size": 8,
"gradient_accumulation_steps": 8,
"learning_rate": 0.0001,
"steps": 20000,
"seed": 42,
"log_freq": 100
},
"errorMessage": "CUDA out of memory"
}That example is the common self-inflicted version. Somebody raised the batch to speed the run up, did not lower the accumulation, and quadrupled both the memory footprint and the effective batch at the same time. The defaults for SmolVLA are batch 2 with accumulation 8, and they were chosen to sit inside 24 GB.
One thing that is easy to forget: the number of camera streams multiplies the pixels in every sample. A dataset with a scene camera and a wrist camera carries twice the image data per frame of a single-camera set, at the same batch size. Adding a second camera to your recording setup and keeping the same batch is a real memory increase, not a rounding difference.
Gradient accumulation is the trade you actually have
Accumulation runs several smaller forward and backward passes and sums their gradients before it takes one optimizer step. The optimizer sees the same total number of samples as it would from one large batch, but the card only ever holds one small batch of activations at a time. Batch 2 with accumulation 8 and batch 16 with accumulation 1 both give the optimizer 16 samples per step. The first fits in far less memory and takes eight passes to get there.
What you pay is wall clock. Eight passes per step take about as long as eight passes, so the run gets longer in proportion to how far you split the batch. Since the platform rents GPUs by the hour, that is also the cost axis: a run that fits on a cheaper card at higher accumulation can still be the cheaper run overall, and often is, because the hourly gap between the tiers is larger than the throughput gap.
Effective batch is batch size times accumulation steps. If you halve one, double the other and the optimizer sees exactly what the defaults intended. This is the one adjustment that changes memory without changing the training recipe.
A smaller batch shortens the run, even at the same step count
Every trainer here is step-based, not epoch-based. The step count is a count of optimizer updates, so the amount of data a run sees is batch size times accumulation times steps. Drop the batch without touching accumulation or steps and you have quietly shortened the run.
samples seen = batch_size x gradient_accumulation_steps x steps
GR00T N1.7 defaults: 32 x 1 x 20000 = 640 000
Batch halved to 16: 16 x 1 x 20000 = 320 000 half the exposure
Halved, steps doubled: 16 x 1 x 40000 = 640 000 same exposure, longer run
Halved, accumulation 2: 16 x 2 x 20000 = 640 000 same exposure, same step countThis is where a lot of confused model comparisons come from. Somebody lowers the batch to make a run fit, leaves the step count at the documented default, gets a weaker policy, and concludes the model is worse than advertised. The model saw half the data. Raise the accumulation, or raise the steps, and then judge it.
What to do, in order
- 1Confirm which tier the model asks for
Check the model on /policies before touching anything. If it is GR00T N1.7, GR00T N1.5 or Pi0.5, it is on the 80 GB tier and this is not a batch size conversation.
- 2Compare your configuration against the documented defaults
Pull configJson from the job record and read batch_size and gradient_accumulation_steps against the table above. In most cases someone raised one without lowering the other.
- 3Restore the defaults and start again
The defaults are the fastest way back to a run that fits. Change one thing at a time afterwards, if you still want to.
bashcurl -X POST https://www.ay-robots.com/api/mcp \ -H "Authorization: Bearer $AY_ROBOTS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "start_training", "arguments": { "dataset_id": "ds_44c108", "policy_name": "so100-cube-smolvla-v2", "model_id": "smolvla", "max_steps": 20000, "confirm_spend": true } } }' # start_training takes max_steps but not batch size. Batch and accumulation # live in the Training form in the dashboard. - 4If you must shrink the batch, raise accumulation by the same factor
Halve the batch, double the accumulation. The effective batch is unchanged and the schedule keeps its meaning.
- 5If you cannot keep the product, raise the step count instead
Work out the samples you gave up and add steps to compensate. Then expect the run to take longer and budget for it, because the hourly rental does not care why the run is long.
- 6Ask whether the smaller model answers your question
If you are still validating a dataset, the 24 GB tier is the right place to be. /train/smolvla-on-so-100 gets you a result in an afternoon, and /compare/groot-n1-7-vs-smolvla sets out what the bigger model buys once the data is worth it.
Frequently asked questions
Can I fine-tune GR00T N1.7 on my own 4090?▾
No. It requests an 80 GB card and 24 GB is not close. SmolVLA and ACT are the two models on this platform that train on a 4090, and both are reasonable ways to check whether a dataset is worth A100 time.
Does raising accumulation slow training down?▾
Per optimizer step, yes, roughly in proportion to how many passes you split the batch into. Per sample it is close to a wash. What you are buying is the ability to run at all on a card that would otherwise refuse the batch.
Why do GR00T N1.5 and Pi0.5 default to batch size 1?▾
Because they are three billion parameter models with video inputs, and at that size a single sample already occupies a large share of an 80 GB card. Their recipes spend the remaining room on 16 accumulation steps instead, which gives an effective batch of 16 without ever holding 16 samples at once.
The run died at step 8000, not at the start. Same fix?▾
Not usually. A configuration that ran 8000 steps fits. Look for what was different about the batch that failed: unusually long episodes, a higher resolution recording mixed into the set, or a second camera present in only part of the data.
Am I billed for a run that fails on out of memory?▾
The GPU is rented by the hour from the moment the container starts, so a run that fails after eight hours has rented eight hours. This is the practical argument for watching the first few hundred steps rather than starting a long run and walking away.
A symptom in one part of the chain is regularly produced by the part before it. These pages cover the neighboring areas.