Quantisation for On-Robot Inference: INT8, FP8 and What It Costs

Quantisation is the first thing people reach for when a vision-language-action model is too slow on the robot, and the thing most often misread. Cutting weights from 16 bits to 8 does halve the memory. It does not reliably halve the latency, and on the best-measured VLA in public it cost thirteen points of task success while the arithmetic stayed accurate. The failure was not numerical. The policy got slower and stopped matching the control rate it was trained at.

What follows is what INT8 and FP8 cost and save next to the servos: which formats need which silicon, which runtimes will build a quantised engine for a VLA today, and where quantisation is simply the wrong lever. Search note: the American spelling is quantization, and every flag name below uses it.

The formats, and what silicon each one needs

A quantised tensor is a low-precision value plus a scale. TensorRT's scheme is symmetric: values are stored as signed INT8, FP8E4M3, signed INT4 or FP4E2M1, converting back is one multiply, and rounding is round-to-nearest ties-to-even with a clamp. The two FP8 encodings come from the FP8 Formats for Deep Learning paper: E4M3 spends four exponent bits and three mantissa bits, E5M2 the reverse. Inference uses E4M3; E5M2 is mainly for gradients.

The measurement that should change your plan

The cleanest published numbers on quantised VLAs are in the OpenVLA paper. OpenVLA is a 7B model on a Prismatic VLM with a Llama 2 backbone, evaluated on real-robot BridgeData V2 tasks at three precisions on the same hardware.

Read the middle row twice. 8-bit cost 13.2 points of success rate; 4-bit cost nothing measurable, though it discards twice as much information. The paper is explicit: "8-bit quantization slows down inference across most GPUs, due to the overhead of the added quantization operations", and at 1.2 Hz against a controller recorded at 5 Hz this "significantly changes the system dynamics compared to the training dataset". The model was not wrong. It was late.

Why 8-bit can be slower than 16-bit

• bitsandbytes LLM.int8() is not a plain INT8 matmul. Per the LLM.int8() paper it quantises vector-wise, then splits outlier feature dimensions into a separate 16-bit matmul. Over 99.9 percent of values go through INT8, but you run two matmuls plus a scatter instead of one.
• Weight-only 4-bit skips that: dequantise to bf16, one matmul, fewer kernels, less memory traffic.
• Quantise and dequantise nodes are not free. Unfused, each is an extra full pass over the tensor.
• Batch size 1 is the robot's only batch size, and a 7B model at batch 1 is memory-bandwidth bound. Halving arithmetic width buys far less than the FLOP count suggests.
• None of this shows up at batch 32, which is how most quantisation results are reported.

What the runtimes actually support, August 2026

Support is not the same as a working VLA path. A runtime can advertise FP8 and still offer no route from a diffusion-head robot policy to a quantised engine.

Two rows contradict the marketing. TensorRT-LLM's FP8 guide states FP8 is supported only above compute capability 8.9: Ada, Hopper, Blackwell and later. An AGX Orin 64GB is Ampere, quoted at 275 sparse INT8 TOPS, with no FP8 path. Jetson Thor (25 August 2025) is Blackwell: 2070 sparse FP4 TFLOPS, 517 dense FP8 TFLOPS, 128 GB LPDDR5X at 273 GB/s, 130 W. If your plan says FP8 and your board says Orin, the plan is wrong.

NVIDIA's own GR00T path is bf16, and still 2.35x faster

The Isaac-GR00T deployment guide documents a full TensorRT pipeline for GR00T N1.7: export to ONNX, build engines, verify against PyTorch, benchmark. The precision flag exists and accepts exactly one value.

# Isaac-GR00T, scripts/deployment/build_trt_pipeline.py
# --precision   default: bf16   "Precision for ONNX export and TRT engine build (bf16 only)"

uv run python scripts/deployment/build_trt_pipeline.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA \
  --export-mode full_pipeline \
  --batch-size 1 \
  --workspace 8192 \
  --steps export,build,verify,benchmark

The speedups are real anyway. NVIDIA's table, measured at 4 denoising steps with 1 camera, shows what compilation alone does.

Note the ordering. Orin gains 2.35x with no precision reduction at all, beating the 1.49x that QVLA reports for mixed-precision quantisation of OpenVLA-OFT and the 1.52x in Mix-QVLA. Kernel fusion, static shapes and removing Python from the loop are cheaper than bit width and do not touch the weights. Do them first.

Do it yourself: the two paths that work

bitsandbytes if you are VRAM-bound and want an answer in ten minutes. TensorRT Model Optimizer if you want throughput and will build a calibration loop.

Path A: bitsandbytes, data-free, memory-first

1. 1
   Check compute capability first

   LLM.int8() needs capability 7.5 or newer (Turing); NF4 and FP4 need 6.0 or newer (Pascal). The library itself needs Python 3.10+ and PyTorch 2.4+.

   bash

   python -c "import torch; print(torch.cuda.get_device_capability())"
   # (8, 9) = Ada, FP8-capable
   # (8, 7) = Jetson Orin, INT8 only
   # (7, 5) = Turing, the LLM.int8() floor
   
   pip install --upgrade transformers accelerate bitsandbytes

2. 2
   Load the backbone in 4-bit NF4

   This is the config the OpenVLA repository uses for quantised LoRA fine-tuning in vla-scripts/finetune.py, gated behind --use_quantization, which asserts LoRA is also on.

   python

   import torch
   from transformers import AutoModelForVision2Seq, BitsAndBytesConfig
   
   quantization_config = BitsAndBytesConfig(
       load_in_4bit=True,
       bnb_4bit_quant_type="nf4",
       bnb_4bit_compute_dtype=torch.bfloat16,
       bnb_4bit_use_double_quant=True,   # a further ~0.4 bits per parameter
   )
   
   model = AutoModelForVision2Seq.from_pretrained(
       "openvla/openvla-7b",
       quantization_config=quantization_config,
       low_cpu_mem_usage=True,
       trust_remote_code=True,
       device_map="auto",
   )
   print(model.get_memory_footprint())

3. 3
   If you insist on 8-bit, learn the threshold knob

   llm_int8_threshold decides which hidden-state values count as outliers and stay in fp16. The documented default is 6. Setting it to 0.0 speeds inference up at some accuracy cost, which is exactly the trade the OpenVLA numbers warn about.

   python

   quantization_config = BitsAndBytesConfig(
       load_in_8bit=True,
       llm_int8_threshold=6.0,
       llm_int8_skip_modules=["lm_head"],
   )

4. 4
   Measure at batch 1, on the board, with the cameras attached

   Not throughput. Wall-clock milliseconds from observation to action at batch 1, including preprocessing, on the machine that holds the arm. Then compare against the rate your episodes were recorded at.

   python

   import time, torch
   
   for _ in range(5):            # warmup: engines, caches, autotune
       _ = policy.select_action(obs)
   
   torch.cuda.synchronize()
   t0 = time.perf_counter()
   for _ in range(50):
       _ = policy.select_action(obs)
   torch.cuda.synchronize()
   print(f"{(time.perf_counter() - t0) / 50 * 1000:.1f} ms per action step")

Path B: TensorRT Model Optimizer, calibration-first

ModelOpt produces a quantised checkpoint TensorRT can compile, and simulates quantisation in PyTorch first, so you can measure the accuracy hit before committing to an engine build.

# pip install -U nvidia-modelopt[all]
import modelopt.torch.quantization as mtq

# Named configs shipped by the library:
#   mtq.INT8_DEFAULT_CFG        per-tensor activations, per-channel weights
#   mtq.INT8_SMOOTHQUANT_CFG    shifts activation outliers into the weights first
#   mtq.FP8_DEFAULT_CFG         compute capability 8.9 or newer
#   mtq.NVFP4_DEFAULT_CFG       Blackwell only

def forward_loop(model):
    # 128 to 512 real observations is the documented range.
    # Use frames from YOUR episodes, in the lighting you will deploy in.
    for batch in calib_loader:
        model(batch)

model = mtq.quantize(model, mtq.INT8_DEFAULT_CFG, forward_loop)

The calibration set is what people get wrong. It sets the scale factors, and therefore the clamp points. Calibrate on bright tidy frames, deploy under a different lamp, and real activations clip. Draw calibration frames from your own LeRobot dataset: same scene, same cameras, same exposure. If the dataset is thin, fix that first; the guide on collecting high-quality VLA training data is the prerequisite.

If PTQ costs too much, quantisation-aware training is the escalation. torchao ships a QAT recipe (prepare, fine-tune, convert) and reports recovering 96 percent of the accuracy degradation on hellaswag and 68 percent of the perplexity degradation on wikitext for Llama 3 against plain PTQ. For a robot policy that means folding QAT into the fine-tuning run rather than bolting it on afterwards.

Two ways to get a fast policy onto an SO-100

Is it worth it

Where quantisation is the wrong lever

That last row deserves a positive statement. When the model genuinely does not fit, quantisation is the enabling step. BitVLA trains a VLA natively in 1-bit, every parameter ternary, on top of BitNet b1.58 2B4T, and reports 11.0x lower memory and 4.4x lower end-to-end latency than OpenVLA-OFT at comparable performance. That is co-designing the policy for the precision rather than squeezing a bf16 checkpoint afterwards. The overview of VLA models and the Pi0 flow-matching write-up sit one layer below this page.

The smaller policies here make the same point without exotic arithmetic. SmolVLA is around 450 M parameters at 245 ms per step, ACT around 80 M at 20 ms, and both run locally on a 24 GB card. Choosing a smaller architecture is a cleaner route to a latency target than quantising a 3 B model into a shape its authors never validated. See ACT against SmolVLA, or the arena with 85 VLA models and 332 benchmark results.

An order of operations that does not waste a week

1. Measure per-step latency at batch 1 on the target board with the cameras attached.
2. Write down the control rate your episodes were recorded at. If the two disagree, that is the bug, and precision will not fix it.
3. Remove Python from the hot loop. On Orin, NVIDIA measured 1.56x from torch.compile alone before any TensorRT work.
4. Compile to TensorRT at bf16 and verify cosine similarity against PyTorch: 2.35x on Orin with zero precision loss.
5. Only now consider precision: weight-only 4-bit if VRAM-bound, INT8 PTQ with real calibration data if compute-bound.
6. Re-run the closed-loop success rate. A throughput number without a success rate is not evidence.
7. If PTQ costs too much, move to QAT inside the fine-tuning run rather than accepting the loss.

Frequently asked questions

The short version

Quantisation is a memory tool that sometimes pays out as a speed tool. Treated as the default answer to "the policy is too slow", it costs a week and produces a model that is smaller, no faster and worse at the task. The order that works: measure at batch 1, match the control rate, compile, verify, then reach for bit width. To test the latency question before buying a board, drive a real SO-100 over the internet. The SO-100 setup guide covers the hardware, the training docs cover what the platform sends to the trainer, and pricing covers what a run costs.