The job reaches RUNNING and then fails with a CUDA out of memory message, either in the first seconds or after thousands of steps.
GR00T N1.7, GR00T N1.5 and Pi0.5 request the A100 80 GB tier; SmolVLA and ACT fit on a 24 GB card. On the two GR00T trainers you can halve the batch and double gradient accumulation, which keeps the effective batch where it was. On ACT, SmolVLA and Pi0.5 that field does nothing, so a smaller batch really is a smaller batch and you compensate with more steps.
Where the memory goes
Three things share the card: the weights, the optimizer state that shadows them, and the activations held for the batch currently in flight. The first two are properties of the model you picked and you cannot negotiate with them. The third scales with how many samples you process at once, and that is the only knob in your hands. This is why the GPU tier is an attribute of the model in the catalog rather than something derived from your dataset.
| Model | GPU tier requested | Default batch | Accumulation the form proposes | Effective batch |
|---|---|---|---|---|
| GR00T N1.7 | A100 80 GB | 32 | 1, applied | 32 |
| GR00T N1.5 | A100 80 GB | 1 | 16, applied | 16 |
| Pi0.5 | A100 80 GB | 1 | 16, not applied | 1 |
| SmolVLA | RTX 4090, 24 GB | 2 | 8, not applied | 2 |
| ACT | RTX 4090, 24 GB | 8 | 1, not applied | 8 |
The last two columns are the ones people get wrong. Only the two GR00T trainers take a gradient accumulation setting. ACT, SmolVLA and Pi0.5 run on the lerobot trainer, which has no such option, so the number the form proposes for them is stored on the job record and then dropped before the trainer starts. Their effective batch is their batch size, full stop, which is why the multiplication that gives Pi0.5 an effective batch of 16 is wrong: it trains at 1.
That also changes what is left to try when one of them runs out of memory. GR00T N1.5 and Pi0.5 both sit at batch size 1, and there is no smaller batch than that. Accumulation rescues neither of them here: on N1.5 it moves the effective batch, not the memory a single pass needs, and on Pi0.5 it moves nothing at all. What is left in both cases is what one sample contains, which means the number of camera streams and the resolution behind them.
The gap between the tiers is more than a factor of three, and the three billion parameter models are on the far side of it. There is no combination of batch size and accumulation that fine-tunes GR00T on a consumer card. If you need the 24 GB tier, you need a different model, not a different configuration.
Read when it died before you change anything
An out of memory in the first seconds means the configuration never fit. Nothing grew, nothing leaked, the first batch simply did not have room. An out of memory after thousands of successful steps means something about one particular batch was larger than the ones before it, and on video data that usually means episodes of uneven length or resolution in the same set.
// MCP get_training_job on a run that died at step 0
{
"id": "job_9e4471",
"modelType": "smolvla",
"status": "FAILED",
"configJson": {
"batchSize": 8,
"gradAccum": 8,
"lr": 0.0001,
"maxSteps": 20000,
"seed": 42,
"logFreq": 100
},
"errorMessage": "CUDA out of memory"
}That example is the common self-inflicted version, though not for the reason most people give. Somebody raised the batch to speed the run up and quadrupled the activations the card has to hold. Lowering the gradAccum back to 1 afterwards would have changed nothing, because it was doing nothing in either direction. The documented batch for SmolVLA is 2, and that single number is what was chosen to sit inside 24 GB.
One thing that is easy to forget: the number of camera streams multiplies the pixels in every sample. A dataset with a scene camera and a wrist camera carries twice the image data per frame of a single-camera set, at the same batch size. Adding a second camera to your recording setup and keeping the same batch is a real memory increase, not a rounding difference.
Gradient accumulation is a GR00T trade, not a platform-wide one
Accumulation runs several smaller forward and backward passes and sums their gradients before it takes one optimizer step. The optimizer sees the same total number of samples as it would from one large batch, but the card only ever holds one small batch of activations at a time. Batch 1 with accumulation 16 and batch 16 with accumulation 1 both give the optimizer 16 samples per step. The first fits in far less memory and takes sixteen passes to get there, which is exactly the recipe GR00T N1.5 ships.
What you pay is wall clock. Sixteen passes per step take about as long as sixteen passes, so the run gets longer in proportion to how far you split the batch. Since the platform rents GPUs by the hour, that is also the cost axis, and it is the argument for splitting the batch on the tier you are already paying for rather than assuming a cheaper tier will take the same recipe. It will not: the 24 GB tier hosts SmolVLA and ACT, and neither of them accumulates anything.
Effective batch is batch size times accumulation steps. On GR00T N1.7 and N1.5 you can hold that product still: halve one, double the other, and the optimizer sees what the defaults intended. It is the one adjustment that changes memory without changing the recipe. On ACT, SmolVLA and Pi0.5 the second factor is always 1, whatever the form recorded.
What to pull instead on ACT, SmolVLA and Pi0.5
The training form already knows this and grays the Gradient Accumulation field out when one of those three models is selected. The field is disabled, not broken: it would accept a number the trainer never reads, so the form refuses to pretend. Everything you have on these models runs through the batch size, which decides how much memory the run needs and how many samples each optimizer step sees at the same time. There is nothing that separates the two.
In practice that means the ladder is short. If SmolVLA does not fit at batch 2, batch 1 is the next rung and it is the last one, and the run then learns from one sample per step. If Pi0.5 does not fit at batch 1 there is no rung left at all. Below that, the only remaining reductions are in the data itself: fewer camera streams per episode, or a lower recording resolution, both of which shrink a single sample rather than the number of samples held at once.
And if a large effective batch is genuinely what your run needs, that is a model choice rather than a settings choice. GR00T N1.7 trains at 32 samples per step on the 80 GB tier, and GR00T N1.5 reaches 16 by accumulating. Everything else on the platform trains at the batch size printed in the table above, so the realistic options are to move to GR00T or to accept the small batch and give the run more steps.
A smaller batch shortens the run, even at the same step count
Every trainer here is step-based, not epoch-based. The step count is a count of optimizer updates, so the amount of data a run sees is batch size times accumulation times steps, where the middle factor is 1 on every model except the two GR00T ones. Drop the batch and add nothing back and you have quietly shortened the run.
samples seen = batchSize x gradAccum x maxSteps
gradAccum counts on GR00T N1.7 and N1.5, and is 1 elsewhere
GR00T N1.7 defaults: 32 x 1 x 20000 = 640 000
Batch halved to 16: 16 x 1 x 20000 = 320 000 half the exposure
Halved, steps doubled: 16 x 1 x 40000 = 640 000 same exposure, longer run
Halved, accumulation 2: 16 x 2 x 20000 = 640 000 same exposure, same step count
SmolVLA defaults: 2 x 1 x 20000 = 40 000 the 8 in the form is not applied
Batch dropped to 1: 1 x 1 x 20000 = 20 000 half the exposure
Dropped, steps doubled: 1 x 1 x 40000 = 40 000 the only way back on this trainerThis is where a lot of confused model comparisons come from. Somebody lowers the batch to make a run fit, leaves the step count at the documented default, gets a weaker policy, and concludes the model is worse than advertised. The model saw half the data. On GR00T, raise the accumulation or the steps; on ACT, SmolVLA and Pi0.5, raise the steps. Then judge it.
What to do, in order
- 1Confirm which tier the model asks for
Check the model on /policies before touching anything. If it is GR00T N1.7, GR00T N1.5 or Pi0.5, it is on the 80 GB tier and this is not a batch size conversation.
- 2Compare your configuration against the documented defaults
Pull the job record with get_training_job and read configJson.batchSize and configJson.gradAccum against the table above. Those are the names on the record, in camelCase; the same two numbers appear as Batch Size and Grad Accum on the run in the Training section of the dashboard. On a GR00T job both reached the trainer. On an ACT, SmolVLA or Pi0.5 job the gradAccum is a record of what the form proposed, not of what the run did, so read the batch size on its own. In most cases someone raised it.
bashcurl -X POST https://www.ay-robots.com/api/mcp \ -H "Authorization: Bearer $AY_ROBOTS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "get_training_job", "arguments": { "job_id": "job_9e4471" } } }' # Read configJson.batchSize and configJson.gradAccum out of the response. # A run started through start_training records only maxSteps, because that # tool takes no batch arguments, and the trainer then keeps its own defaults. - 3Restore the defaults and start again
The defaults are the fastest way back to a run that fits. Change one thing at a time afterwards, if you still want to.
bashcurl -X POST https://www.ay-robots.com/api/mcp \ -H "Authorization: Bearer $AY_ROBOTS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "start_training", "arguments": { "dataset_id": "ds_44c108", "policy_name": "so100-cube-smolvla-v2", "model_id": "smolvla", "max_steps": 20000, "confirm_spend": true } } }' # start_training takes max_steps but not batch size. The batch lives in the # Training form in the dashboard, next to the accumulation field that stays # grayed out for smolvla. - 4On GR00T, shrink the batch and raise accumulation together
Halve the batch, double the accumulation. The effective batch is unchanged and the schedule keeps its meaning. This move exists on GR00T N1.7 and N1.5 and nowhere else on the platform.
- 5On ACT, SmolVLA and Pi0.5, shrink the batch and add steps
The accumulation field is grayed out for these three because their trainer has no such setting, so a smaller batch is genuinely a smaller batch. Work out the samples you gave up and add steps to compensate. Then expect the run to take longer and budget for it, because the hourly rental does not care why the run is long.
- 6Ask whether the smaller model answers your question
If you are still validating a dataset, the 24 GB tier is the right place to be. /train/smolvla-on-so-100 gets you a result in an afternoon, and /compare/groot-n1-7-vs-smolvla sets out what the bigger model buys once the data is worth it.
Frequently asked questions
Can I fine-tune GR00T N1.7 on my own 4090?▾
No. It requests an 80 GB card and 24 GB is not close. SmolVLA and ACT are the two models on this platform that train on a 4090, and both are reasonable ways to check whether a dataset is worth A100 time.
The Gradient Accumulation field is grayed out. Is the form broken?▾
No, you have ACT, SmolVLA or Pi0.5 selected. Those three run on the lerobot trainer, which has no accumulation setting, so the form disables the field rather than accepting a number that would be dropped on the way to the GPU. Select GR00T N1.7 or N1.5 and the field becomes editable.
Does raising accumulation slow training down?▾
On GR00T, per optimizer step yes, roughly in proportion to how many passes you split the batch into, while per sample it is close to a wash. What you buy is the ability to run at all on a card that would otherwise refuse the batch. On the other three models the question does not come up, because the setting is not there to raise.
Why do GR00T N1.5 and Pi0.5 default to batch size 1?▾
Because they are three billion parameter models with video inputs, and at that size a single sample already occupies a large share of an 80 GB card. N1.5 spends the remaining room on 16 accumulation steps and reaches an effective batch of 16 without ever holding 16 samples at once. Pi0.5 is offered the same 16 in the form, but its trainer ignores it, so it trains at an effective batch of 1.
The run died at step 8000, not at the start. Same fix?▾
Not usually. A configuration that ran 8000 steps fits. Look for what was different about the batch that failed: unusually long episodes, a higher resolution recording mixed into the set, or a second camera present in only part of the data.
Am I billed for a run that fails on out of memory?▾
The GPU is rented by the hour from the moment the container starts, so a run that fails after eight hours has rented eight hours. This is the practical argument for watching the first few hundred steps rather than starting a long run and walking away.
Was this page helpful?
A symptom in one part of the chain is regularly produced by the part before it. These pages cover the neighboring areas.