Running a VLA Policy on a 4090: What Fits in 24 GB

What 24 GB actually buys you

The RTX 4090 is the card most people mean when they say consumer GPU. For anyone running a vision-language-action model next to a robot arm on their own desk, the only spec that decides anything is the 24. Compute headroom is generous. Memory is the wall.

Those figures are NVIDIA's own, from the RTX 4090 product page. Three things compete for the 24 GB during inference: model weights, activations for the forward pass, and whatever else your machine already has on that GPU. The last is not a rounding error. A CUDA context costs a few hundred megabytes before you load anything, and a desktop with a browser open can hold a gigabyte or two.

Model by model: what fits at inference

Every figure below comes from a model card, a repository README or a paper read for this article in August 2026, and the source column says which. Nothing is extrapolated from parameter counts.

The pattern runs against the intuition people bring from LLM work. The current generation of open robot policies is small. A 3B VLA in bfloat16 is roughly 6 GB of weights. The 7B autoregressive generation, of which OpenVLA is the reference, is the largest thing you can realistically serve, and even that fits. If a policy does not fit on a 4090 today, it is usually because it was never released.

Measured latency on a real 4090

Most VRAM tables stop at whether a model loads. What decides whether your arm moves smoothly is how long a forward pass takes, and there is one published measurement on exactly this card. Appendix D of the Pi0 paper gives the per-component breakdown in Table I, timed with three camera images on a GeForce RTX 4090.

73 ms is roughly 13.7 replans per second, which sounds far too slow for a 50 Hz arm. It would be, except that a flow matching policy emits a chunk, not one action. The Pi0 authors run inference every 0.5 seconds on their 50 Hz robots, after executing 25 actions, and every 0.8 seconds on the 20 Hz UR5e and Franka setups. They tried temporal ensembling of overlapping chunks, found it hurt, and execute chunks open-loop instead.

OpenVLA is the counter-example. The paper states it runs at approximately 6 Hz on one NVIDIA RTX 4090 without compilation or speculative decoding, and the repository's data-collection advice follows directly: collect at 5 to 10 Hz, and if your robot uses a 50 Hz controller, downsample to 5 Hz first and check the task is still solvable. That is a constraint on which tasks you can attempt.

For GR00T N1.7, NVIDIA publishes an end-to-end inference frequency table measured with four denoising steps and one camera. There is no RTX 4090 row in it, which is worth stating plainly rather than interpolating. The nearest workstation-class datapoints:

The interesting part is not the top row. An H100 in plain PyTorch eager mode manages only 11.7 Hz, because the action head dominates: NVIDIA's H100 breakdown splits 85.8 ms into 6.2 ms of data processing, 31.3 ms of backbone and 48.2 ms of action head. torch.compile takes the action head to 12.0 ms and the total to 48.6 ms; a full TensorRT pipeline reaches 27.9 ms. NVIDIA puts torch.compile at 1.1 to 1.9x and a full TensorRT pipeline at 1.5 to 3.3x over eager across every platform it tested, so serving GR00T without compiling leaves a large part of the speed on the floor whatever the card.

Quantisation: what it buys, what it costs

A 3B VLA already fits in 24 GB, so quantisation on a 4090 is a narrow tool: serving the 7B autoregressive class with room left over, or squeezing a policy onto a smaller card. Here is what OpenVLA measured.

The mechanics are ordinary Hugging Face. The flags live in BitsAndBytesConfig, and for a robot policy you want NF4 with a bfloat16 compute dtype so the matmuls stay fast.

import torch
from transformers import AutoModelForVision2Seq, AutoProcessor, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

processor = AutoProcessor.from_pretrained("openvla/openvla-7b", trust_remote_code=True)
vla = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b",
    quantization_config=quantization_config,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
)

print(vla.get_memory_footprint())

What is genuinely out of reach on 24 GB

Almost nothing in the current open ecosystem is too large to serve on a 4090. Plenty is too large to train on one. That is where the disappointment comes from: people read a model card that says 16 GB and assume it covers fine-tuning too.

LeRobot's own compute guide is blunt about where a consumer card sits: RTX 3090 and 4090 are comfortable for light behaviour cloning, diffusion policies and SmolVLA, and tight for VLAs at batch 1. A Pi0.5 LoRA run at over 22.5 GB on a 24 GB card leaves under 1.5 GB for the dataloader, the CUDA context and fragmentation. It runs until one image batch is slightly larger than usual. The out-of-memory during training page covers the escape routes.

The manual path: SmolVLA on a 4090 next to an SO-100

This is the configuration that makes sense on a 24 GB card in 2026: a small SmolVLA policy served from the GPU box, driving an SO-100 over USB. SmolVLA is 450 M parameters on SmolVLM-2, limited to 64 visual tokens per frame with no image tiling, and skipping half the VLM layers (the action expert reads features up to layer N = L/2). That is why it lands at ~2 GB instead of 14.

1. 1
   Install LeRobot with the async extras

   The async inference stack is an optional dependency group. Install it now, not after your first server refuses to start.

   bash

   pip install lerobot
   lerobot-info
   
   # from a clone, for the async inference server and client:
   pip install -e ".[async]"
   pip install -e ".[smolvla]"

2. 2
   Fine-tune smolvla_base on your own episodes

   SmolVLA is a base model, so it needs your task. The LeRobot docs recommend ~50 episodes and note that 25 was not enough in their own experiments. Twenty thousand steps takes roughly 4 hours on an A100, so budget longer on a 4090 and lower the batch size until it fits.

   bash

   lerobot-train \
     --policy.path=lerobot/smolvla_base \
     --dataset.repo_id=${HF_USER}/mydataset \
     --batch_size=64 \
     --steps=20000 \
     --output_dir=outputs/train/my_smolvla \
     --job_name=my_smolvla_training \
     --policy.device=cuda \
     --wandb.enable=true

3. 3
   Start the policy server on the 4090 box

   The server starts empty and learns which policy to load during the first handshake, so there is no model path here.

   bash

   python -m lerobot.async_inference.policy_server \
     --host=127.0.0.1 \
     --port=8080

4. 4
   Start the robot client next to the arm

   The client owns the serial port, the cameras and the action queue. It streams observations over gRPC and executes chunks as they arrive.

   bash

   python -m lerobot.async_inference.robot_client \
     --server_address=127.0.0.1:8080 \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=follower_so100 \
     --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \
     --task="Grasp a lego block and put it in the bin." \
     --policy_type=smolvla \
     --pretrained_name_or_path=${HF_USER}/my_smolvla \
     --policy_device=cuda \
     --actions_per_chunk=50 \
     --chunk_size_threshold=0.5 \
     --aggregate_fn_name=weighted_average \
     --debug_visualize_queue_size=True

5. 5
   Tune the queue until it never empties

   chunk_size_threshold decides when the client sends a fresh observation. Near 0.0 it collapses to synchronous inference and the arm stalls at every chunk boundary; near 1.0 it saturates the GPU. The LeRobot blog reports g ~ 0.7, the tutorial suggests 0.5 to 0.6. Watch the plotted queue and keep it above zero.

   bash

   # rerun the client with the queue plot on, and vary one number:
   --chunk_size_threshold=0.5   # more sequential, lower GPU load
   --chunk_size_threshold=0.7   # LeRobot's reported sweet spot
   --chunk_size_threshold=0.9   # near-continuous replanning, high load

6. 6
   Run the policy without the async stack, if you prefer

   For a first smoke test, lerobot-rollout drives the arm directly. The commented RTC flags switch on real-time chunking, for when the inference machine is weaker than a 4090.

   bash

   lerobot-rollout \
     --strategy.type=base \
     --robot.type=so101_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_blue_follower_arm \
     --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \
     --task="Grasp a lego block and put it in the bin." \
     --policy.path=${HF_USER}/my_smolvla
     # --inference.type=rtc
     # --inference.rtc.execution_horizon=10

Two ways to get a policy running

You can own the whole stack or rent the annoying parts. The difference is whether you want to spend your evenings on CUDA versions or on episodes.

Where a 4090 stops helping

A fast card does not fix a slow link. On this platform the control loop runs at 20 to 485 ms per action step depending on the model, and that is the budget the whole system fits inside. Adding a public-internet round trip to a 245 ms SmolVLA step or a 485 ms Pi0.5 step turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place, not for fast reactive motion.

The LeRobot team measured sub-100 ms round-trip latency hosting SmolVLA on an RTX 4090 over their own local network, with roughly a 2x speedup in task completion from async inference at comparable success. Both halves matter: the 2x comes from never idling, the sub-100 ms from the network being a switch away.

If you script this, the CLI and the MCP server expose the same operations, so an agent can start a run, poll it and fetch a checkpoint without a browser. Useful for launching five variants of the same LeRobot dataset rather than clicking through five forms.

A decision rule that survives contact

Pick the smallest policy that solves your task. On a 24 GB card the cost of being wrong about size is paid at training time, not serving time.

1. One task, fixed cameras, no language: ACT. 80 M parameters, 20 ms per step here, trains locally in under an hour. No base model, so it only exists after you train it.
2. Language instructions and some generalisation: SmolVLA. 450 M parameters, ~2 GB at inference, 30 episodes minimum, still comfortably local.
3. Pretrained world knowledge of a 3B foundation model: GR00T N1.7 or Pi0.5. Both serve on a 4090; neither fine-tunes on one.
4. Compare first: ACT against SmolVLA, GR00T N1.7 against Pi0.5, or the Arena with 85 VLA models and 332 benchmark results.

If you do not own an arm yet, the ordering flips: record data first, worry about the GPU second. The data collection guide is a better use of a weekend than a driver upgrade, the desktop client records LeRobot-format datasets from a teleop session, and the dataset directory has material to test the pipeline against first.