LoRA vs Full Fine-Tuning for VLA Models: What Actually Trains

The question, stated precisely

When people ask whether to use LoRA or full fine-tuning on a vision-language-action model, they are asking three things at once. How much GPU memory do I need. How much accuracy do I give up. And can I do it at all with the trainer I have. Only the third has a hard answer, and for one of the two big VLA trainers it changed in the last two releases.

All of it was checked on 23 August 2026 against the original LoRA paper, OpenVLA's fine-tuning table, and the Isaac-GR00T and lerobot source trees at the tags named. Where upstream changed, the tag is part of the claim.

# W0 is frozen. Only A and B carry gradients.
#
#   h = W0 @ x  +  (lora_alpha / r) * (B @ A @ x)
#
#   W0 : (d_out, d_in)   frozen, bfloat16
#   A  : (r,     d_in)   trainable, random Gaussian init
#   B  : (d_out, r)      trainable, initialised to zero
#
# Trainable params per adapted layer: r * (d_in + d_out)
#
# For d_in = d_out = 2048 and r = 32:
#   LoRA :  32 * (2048 + 2048) =   131,072
#   full :       2048 * 2048   = 4,194,304   -> 32x more

Full fine-tuning of a 3 B VLA is not what most people run

The phrase gets used loosely, and the vendor defaults sit in the middle. At n1.7-release the GR00T fine-tune entry point is gr00t/experiment/launch_finetune.py, a thin tyro wrapper around the FinetuneConfig dataclass in gr00t/configs/finetune_config.py. This is what it sets when you pass no tuning flags at all.

The shipped recipe is already a partial fine-tune. NVIDIA's hardware recommendation guide puts a number on it: the default "tunes the projector + diffusion action head (not the full LLM backbone), keeping peak VRAM under ~35 GB per GPU", while tune_llm or tune_visual means "80 GB+ per GPU recommended". The same page sets the floor at 40 GB. That is the memory story for GR00T N1.7. You are choosing between roughly 35 and 80 GB, and LoRA is not the only lever in between.

The one clean measurement: OpenVLA's fine-tuning table

OpenVLA is still the best apples-to-apples comparison published for a VLA: five strategies on one 7 B model, the same Franka-Tabletop tasks, the same batch size, LoRA on all linear layers. Table 1 in section 5.3 of OpenVLA: An Open-Source Vision-Language-Action Model reports the following, success averaged over 33 rollouts per approach, VRAM measured at batch 16.

Read the error bars before the means. Full fine-tuning scored 1.5 points higher than rank 32, and both intervals are around 7 points wide. The paper concludes that LoRA "achieves the best trade-off between performance and training memory consumption" while training 1.4 percent of the parameters, a new task taking 10 to 15 hours on one A100, an 8x compute reduction.

Two other rows matter more than the headline. Last-layer-only collapsed to 30.3 percent, so a VLA has to change deep representations to adapt. Frozen-vision cost more VRAM than LoRA and scored twenty points lower: freezing the wrong half is worse than freezing most of everything. Rank 64 bought nothing over rank 32, and the paper recommends r = 32.

Where the VRAM actually goes

LoRA's saving does not come from the weights, it comes from the optimizer. Under AdamW every trainable parameter carries two fp32 moment estimates plus an fp32 gradient on top of itself. Freeze it and all three disappear. The frozen weight still sits there in bfloat16, but that is the small part.

• Frozen parameter: weight only, bfloat16.
• Trainable parameter: weight, gradient, two AdamW moments.
• Activations scale with batch size and sequence length, and LoRA does not remove them. That is why rank 32 on OpenVLA still needed 59.7 GB at batch 16.

LoRA on the stacks you would actually use, August 2026

lerobot: supported, and the defaults are per policy

lerobot documents its PEFT integration in docs/source/peft_training.mdx. The config object is PeftConfig in src/lerobot/configs/default.py, exposed on lerobot-train under the peft prefix. On main (0.6.2 in pyproject, latest release v0.6.1 of 3 August 2026) its fields are target_modules, full_training_modules, method_type, init_type, r and lora_alpha.

1. 1
   Install the PEFT extra

   The feature sits behind an optional dependency; without it the run fails at config time.

   bash

   pip install "lerobot[peft]"
   
   # lerobot pins peft>=0.18.0,<1.0.0 in pyproject.toml
   python -c "import peft; print(peft.__version__)"

2. 2
   Point at a pretrained base, never at scratch

   LoRA adapts existing weights, and the base policy class enforces it.

   bash

   # correct
   --policy.path=lerobot/smolvla_base
   
   # wrong: raises ValueError before training starts
   --policy.type=smolvla

3. 3
   Turn the adapter on and pick a rank

   lerobot's documented command, retargeted at an SO-100 dataset. Rank 64 with alpha 64 gives a scaling of 1.0.

   bash

   lerobot-train \
     --policy.path=lerobot/smolvla_base \
     --dataset.repo_id=<your_hf_user>/<your_so100_dataset> \
     --policy.device=cuda \
     --policy.optimizer_lr=1e-3 \
     --policy.scheduler_decay_lr=1e-4 \
     --steps=100000 \
     --batch_size=32 \
     --peft.method_type=LORA \
     --peft.r=64 \
     --peft.lora_alpha=64

4. 4
   Raise the learning rate by about 10x

   The docs say it "can usually be scaled by a factor of 10 compared to the learning rate used for full fine-tuning (e.g., 1e-4 normal, so 1e-3 using LoRA)". Leaving it at the full value is the commonest reason a LoRA run looks like it is doing nothing.

   bash

   # full fine-tune
   --policy.optimizer_lr=1e-4
   
   # LoRA equivalent
   --policy.optimizer_lr=1e-3 --policy.scheduler_decay_lr=1e-4

5. 5
   Optionally retarget the adapted modules

   SmolVLA adapts the LM expert's q and v projections plus the state and action projections. This moves the adapter to the MLPs instead.

   bash

   --peft.target_modules='(model\.vlm_with_expert\.lm_expert\..*\.(down|gate|up)_proj|.*\.(state_proj|action_in_proj|action_out_proj|action_time_mlp_in|action_time_mlp_out))'

Read the per-policy defaults in the source. Four policies define adapter targets on main: SmolVLA, Pi0, Pi0.5 and MolmoAct2. SmolVLA adapts the LM expert's q and v projections plus the state and action projections; Pi0 and Pi0.5 use the same set on the gemma_expert self-attention. SmolVLA and Pi0 leave modules_to_save empty. Pi0.5 does not: with proprioceptive memory on it adds model.proprio_history_proj, a layer absent from lerobot/pi05_base that an adapter cannot start from.

That is the general pattern rather than a Pi0.5 quirk. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success builds OpenVLA-OFT the same way: a rank-32 adapter of 111 M parameters on the pretrained backbone, plus a fully trained 151 M action head and a 17 M proprioception projector, 279 M trainable in all. New heads train in full, only pretrained weights get an adapter. lerobot exposes that split as --peft.full_training_modules.

Isaac-GR00T: LoRA was there, and then it was removed

This is the part that costs an afternoon. GR00T N1.5 shipped a LoRA path: scripts/gr00t_finetune.py at the n1.5-release tag has lora_rank (default 0, off), lora_alpha (16), lora_dropout (0.1) and lora_full_model (False, so only the action head is adapted). The helper in gr00t/utils/peft.py picked targets by walking the model for Linear layers named q_proj, v_proj, k_proj, to_q, to_v or to_k. The README answered it as an FAQ, recommending "the full model finetuning for better performance".

That file is gone. The listings for the later tags contain zero paths matching lora or peft (363 files at n1.6-release, 423 at n1.7-release), and gr00t/utils/peft.py returns 404 at n1.6-release, n1.7-release and main. The config dataclass behind the new entry point has no LoRA field, so an N1.5-era command makes tyro reject the argument.

# Isaac-GR00T at tag n1.5-release: LoRA available
python scripts/gr00t_finetune.py \
  --dataset-path ./demo_data/robot_sim.PickNPlace \
  --num-gpus 1 \
  --lora_rank 64 --lora_alpha 128

# Isaac-GR00T at tag n1.7-release: no such flags exist
CUDA_VISIBLE_DEVICES=0 python gr00t/experiment/launch_finetune.py \
  --base_model_path nvidia/GR00T-N1.7-3B \
  --dataset_path demo_data/cube_to_bowl_5 \
  --embodiment_tag NEW_EMBODIMENT \
  --modality_config_path examples/SO100/so100_config.py \
  --num_gpus 1 --output_dir /tmp/test_finetune \
  --max_steps 2000 --global_batch_size 32 \
  --learning_rate 1e-4 --save_steps 1000 --save_total_limit 5

# The memory lever on N1.7 is which components you tune:
#   default                  -> projector + diffusion head, ~35 GB peak
#   tune_llm or tune_visual  -> 80 GB+ per GPU recommended

The lever that replaced it

Removing the adapter did not remove the choice, it moved into the unfreeze flags. lerobot's own GR00T policy, which loads nvidia/GR00T-N1.7-3B on its nvidia/Cosmos-Reason2-2B backbone, adds two finer flags to NVIDIA's four: tune_vlln and tune_top_llm_layers. The first, on by default, trains the vision-language LayerNorm and VL self-attention projector in the action head. The second defaults to 0 and unfreezes only the top n language-backbone layers. OpenVLA measured that shape as sandwich fine-tuning: 62.1 percent against 68.2 for rank 32, on a different model, so take it as a hint.

ACT: the question does not apply

ACT trains from scratch. There is no base model to adapt, and lerobot enforces that: the base policy class raises before the first training step if no pretrained path is given, and ACT defines no PEFT targets at all. If you are running ACT on an SO-100, the question is closed.

if not self.config.pretrained_path:
    raise ValueError(
        "Training from scratch using PEFT is unlikely to yield good results. "
        "Supply a `policy.pretrained_path` to fine-tune an existing model."
    )

# 1. what card is this, and does peft install
nvidia-smi --query-gpu=name,memory.total --format=csv
pip install "lerobot[peft]"
pip show lerobot peft | grep -i -e name -e version

# 2. dataset version: lerobot wants v3.0, GR00T wants v2.0/v2.1
head -20 <dataset_root>/meta/info.json

# 3. run it: rank and alpha explicit, 10x lr, seed pinned
lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=<your_hf_user>/<your_so100_dataset> \
  --policy.device=cuda \
  --policy.optimizer_lr=1e-3 \
  --steps=20000 --batch_size=32 \
  --peft.method_type=LORA \
  --peft.r=64 --peft.lora_alpha=64 \
  --seed=1000

# 4. the adapter references the base by path: keep them together
ls outputs/train/*/checkpoints/last/pretrained_model/

Does the accuracy gap matter on a single-task arm?

Mostly, no. Most SO-100 owners work in one regime: one task, one camera setup, one arm, 50 to 200 recorded episodes. That is narrow, in-distribution adaptation, where LoRA and full fine-tuning converge. The OpenVLA-OFT authors read the earlier result the same way: LoRA "enables effective adaptation to single-arm robots operating at low control frequencies", with bimanual arms at 25 to 50 Hz as the case it did not cover.

The gap opens up in the opposite regime: a lot of data, or a target far from what the base model saw. LoRA Learns Less and Forgets Less tested that on language models with roughly 100 K prompt-response pairs and 20 B continued-pretraining tokens, and found LoRA substantially behind in standard low-rank settings, with full fine-tuning learning perturbations of 10 to 100x higher rank. The OpenVLA repository says the same operationally: fully fine-tune only "if the fine-tuning distribution varies drastically from the pretraining distribution".

What I would actually do

If the goal is a working policy on one arm, spend the effort elsewhere. What kills SO-100 policies is not adapter rank. It is a wrist camera that moved between recording and inference, an action chunking horizon that does not match the control loop, duplicated frames, and inference latency added by a network hop. The failure-mode pages and the guide on collecting high-quality VLA training data pay back better than a rank sweep.

1. Start with the vendor default recipe. On GR00T that already is a partial fine-tune.
2. Reach for LoRA when a number forces you to: a card that cannot hold the run, or a disk that cannot hold the checkpoints. Then set lora_alpha explicitly and raise the learning rate by about 10x, the two things that explain most reports of a LoRA run that trains but does not learn.
3. Do not treat a two-point difference on GR00T as signal: no seed, 5 to 6 percent documented variance. And check the trainer version against the flags you are about to use, every time.