Vision Encoders in Robot Policies: SigLIP, DINOv2, Frozen Backbones

Every vision-language-action model on this site begins the same way. A camera frame goes into a pretrained image encoder, comes out as a small number of tokens, and everything downstream - the language model, the action head, the servo commands - only ever sees those tokens. If the encoder discards the detail your task depends on, no amount of training on the action head gets it back.

So two questions are worth answering before you rent a GPU: which encoder is actually inside the policy you picked, and is that encoder being trained or held fixed while you fine-tune. The answers differ per model, they are usually one level deeper than the model card, and for one of the five policies you can train here the default flipped between the paper and the current code.

The encoder is the narrowest part of the pipe

Look at how few numbers actually cross from the camera into the transformer. GR00T N1 encodes images at 224 by 224 and applies pixel shuffle, leaving 64 image token embeddings per frame. SmolVLA lands on the same number from the other end: SmolVLM2-500M runs a patch-16 tower at 512 by 512, a 32 by 32 grid of 1024 patches, and a pixel shuffle with scale factor 4 collapses that by 16 to exactly 64. Pi0.5 spends more, with SigLIP-So400m at patch 14 on a 224 by 224 image giving a 16 by 16 grid, so 256 tokens per camera.

That last row is the odd one out on purpose. ACT has no vision-language model at all. The ALOHA paper describes ResNet-18 backbones turning a 480 by 640 by 3 image into a 15 by 20 by 512 feature map, flattened to a 300 by 512 sequence per camera, with no language conditioning and no internet pretraining beyond ImageNet. That is why it trains from scratch on your task and why it is the cheapest row on the training matrix.

Which encoder each policy actually ships

The model cards mostly say 'a pretrained vision transformer'. The configs say which one. Everything below was read out of the repositories on 2026-08-24, and the version matters, because at least one of these changed twice in a year.

GR00T N1.7 and N1.5: from SigLIP-2 to Qwen3-VL

The GR00T N1 white paper describes an Eagle-2 vision-language backbone that is itself fine-tuned from a SmolLM2 language model and a SigLIP-2 image encoder. The published GR00T-N1-2B had 2.2B parameters total with 1.34B in the VLM. GR00T N1.5 keeps that lineage; its model card states that RGB camera frames are processed through a pre-trained SigLIP2 vision transformer. GR00T N1.7 does not. The Isaac-GR00T README is explicit that the backbone changed from the vendored Eagle model nvidia/Eagle-Block2A-2B-v2 to nvidia/Cosmos-Reason2-2B, which is post-trained from Qwen3-VL-2B-Instruct and encodes images in their native aspect ratio without padding.

# gr00t/model/modules/qwen3_backbone.py (Isaac-GR00T, main, read 2026-08-24)
class Qwen3Backbone(torch.nn.Module):
    def __init__(
        self,
        model_name: str = "nvidia/Cosmos-Reason2-2B",
        tune_llm: bool = False,
        tune_visual: bool = False,
        ...
        tune_top_llm_layers: int = 0,
    ):

    def set_trainable_parameters(self, tune_llm, tune_visual, tune_top_llm_layers):
        for p in self.parameters():
            p.requires_grad = True
        if not tune_llm:
            self.model.language_model.requires_grad_(False)
        if not tune_visual:
            self.model.visual.requires_grad_(False)

Pi0.5: SigLIP-So400m, and nothing frozen

Pi0.5 inherits PaliGemma, which pairs SigLIP-So400m with Gemma-2B. In openpi the instantiation is unambiguous, and the model's freeze filter returns nothing at all unless you explicitly ask for a LoRA variant. Even then the filter matches paths under the language model, never the image tower. In other words, on the reference implementation the SigLIP encoder trains along with everything else.

# src/openpi/models/pi0.py  (openpi, main, read 2026-08-24)
img = nnx_bridge.ToNNX(
    _siglip.Module(
        num_classes=paligemma_config.width,
        variant="So400m/14",
        pool_type="none",
        scan=True,
        dtype_mm=config.dtype,
    )
)

# src/openpi/models/pi0_config.py
def get_freeze_filter(self) -> nnx.filterlib.Filter:
    ...
    if not filters:
        return nnx.Nothing        # default: nothing is frozen

SmolVLA: a 93M SigLIP, frozen, and only half the language model

SmolVLA is the one that optimises hardest for small GPUs, and it does it in three places at once. The SmolVLM paper puts the 500M variant together from a 93M SigLIP-B/16 and SmolLM2-360M, against a 428M SigLIP-SO400M in the 2.2B variant. The SmolVLA paper then drops image tiling, keeps only the global image, and skips half the language decoder: the action expert reads features up to layer N, with N = L/2 in practice. SmolVLM2-500M has 32 decoder layers, and lerobot's default is 16.

# src/lerobot/policies/smolvla/configuration_smolvla.py  (lerobot v0.5.1)
resize_imgs_with_padding: tuple[int, int] = (512, 512)
freeze_vision_encoder: bool = True
train_expert_only: bool = True
train_state_proj: bool = True
optimizer_lr: float = 1e-4
vlm_model_name: str = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
num_vlm_layers: int = 16              # first 16 of the VLM's 32 layers
self_attn_every_n_layers: int = 2

ACT: the only backbone you can actually swap

ACT is the exception where the encoder is a config field rather than a consequence of the base model. lerobot validates that the value starts with 'resnet' and gives the backbone its own learning rate, which is the classic detection-transformer pattern of training the CNN more gently than the head. The paper's own hyperparameters - learning rate 1e-5, batch size 8, chunk size 100 - line up with what this platform sends for ACT, and the code carries a comment that the original implementation used 7 decoder layers where lerobot uses 1.

# src/lerobot/policies/act/configuration_act.py  (lerobot v0.5.1)
vision_backbone: str = "resnet18"
pretrained_backbone_weights: str | None = "ResNet18_Weights.IMAGENET1K_V1"
replace_final_stride_with_dilation: int = False
optimizer_lr: float = 1e-5
optimizer_lr_backbone: float = 1e-5
chunk_size: int = 100
n_action_steps: int = 100

What freezing actually does

Freezing a vision encoder means its weights stop receiving gradients. Two things follow, and only one of them is the one people expect. The obvious one: the encoder can no longer adapt to your cameras, your lighting or your table. The less obvious one: you save far less memory than you think, because the activations still have to be computed and kept for the backward pass through everything above the encoder.

The experiment that settled it, and the one that did not

OpenVLA is the only widely cited VLA that ran the ablation cleanly and published the table. It is built on the Prismatic-7B VLM: a 600M-parameter visual encoder that passes image patches separately through SigLIP and DINOv2 and concatenates the feature vectors channel-wise, a two-layer MLP projector, and a Llama 2 7B backbone. The paper is blunt that this contradicted the VLM literature: prior work found freezing vision encoders during VLM training leads to higher performance, and they found the opposite for control. OpenVLA itself is on the arena page for OpenVLA with its published benchmark numbers.

Read the table as a ranking of what your gradient budget should buy. Spending it on the vision encoder plus a low-rank update everywhere else beats spending it on 6.76 billion parameters that exclude the encoder. In a separate appendix experiment the gap is starker still: across evaluations where both were run, fine-tuning the vision encoder averaged 80.0% success against 46.7% frozen, and some frozen runs were abandoned for near-zero performance and unstable robot behaviour.

The DINOv2 question got a much smaller answer. Dropping the DINOv2 half of the fused encoder moved mean success from 45.6% to 40.6% in the Bridge-only ablation, which the authors call about a 5 percent reduction against a 30 percent drop from removing Open X-Embodiment pretraining. Their summary is that DINOv2's low-level spatial features aid generalisation in only some cases. If you hoped a spatial encoder was the missing ingredient, the published evidence says data comes first, which is where the Open X-Embodiment work ends up too. If the architecture terms above are new, this introduction to VLA models is the place to start.

So why does GR00T freeze it, then?

This is the part where dating the claim matters. The GR00T N1 white paper's hyperparameter table lists the backbone's vision encoder as unfrozen for both pre-training and post-training, with only the text tokenizer frozen. The current Isaac-GR00T fine-tuning config, read from main on 2026-08-24, ships the opposite default.

# gr00t/configs/finetune_config.py  (Isaac-GR00T, main, read 2026-08-24)
tune_llm: bool = False
tune_visual: bool = False
tune_projector: bool = True
tune_diffusion_model: bool = True

global_batch_size: int = 64
learning_rate: float = 1e-4
max_steps: int = 10000
weight_decay: float = 1e-5
warmup_ratio: float = 0.05
save_steps: int = 1000
save_total_limit: int = 5

Two things reconcile it. The audience changed: the paper describes pre-training at batch size 16,384 over 200,000 gradient steps, while the shipped fine-tuning path targets one GPU and a few thousand steps. And the cost changed: the hardware guide says default fine-tuning keeps peak VRAM under about 35 GB per GPU, while --tune-visual or --tune-llm pushes the recommendation to 80 GB or more. Both GR00T versions sit on the 80 GB tier here for that reason, which is also why a run costs more than on the 24 GB pair.

When the encoder is your bottleneck, and when it is not

Most failures blamed on the vision encoder are not the vision encoder. Here is the split that has held up in practice, mapped onto the failure pages on this site.

A cheap way to find out which one you have: ablate the images and see whether the policy notices. If you feed the trained policy black frames and the predicted joint targets barely change, the policy is not reading your cameras at all - it is extrapolating from proprioception, and unfreezing the encoder will not help. If the predictions collapse when you swap two camera streams, the encoder is doing real work and the question becomes whether it does it well enough.

1. 1
   Record the baseline predictions

   Run the trained checkpoint open-loop over a held-out episode and store the predicted action chunk for every step. Do this before touching any flag.

   bash

   uv run python gr00t/eval/open_loop_eval.py \
     --dataset-path /data/my_so100_task \
     --embodiment-tag NEW_EMBODIMENT \
     --model-path /checkpoints/run-a/checkpoint-2000

2. 2
   Blind the policy and repeat

   Feed the same episode with every camera replaced by a constant frame. Compare the mean absolute deviation of the predicted joint targets against the baseline. A small deviation means the images are not driving the actions.

   python

   obs = dict(baseline_obs)
   for key in obs:
       if key.startswith("video."):
           obs[key] = np.zeros_like(obs[key])
   pred_blind = policy.get_action(obs)

3. 3
   A/B the freeze flag with repeats, not once

   For GR00T, run the same config twice per arm of the test, because the README documents 5-6% run-to-run variance from non-deterministic augmentation. Everything else in the command stays identical.

   bash

   # arm A: shipped default, vision tower frozen
   CUDA_VISIBLE_DEVICES=0 uv run python gr00t/experiment/launch_finetune.py \
     --base-model-path nvidia/GR00T-N1.7-3B \
     --dataset-path demo_data/my_task \
     --embodiment-tag NEW_EMBODIMENT \
     --modality-config-path examples/SO100/so100_config.py \
     --num-gpus 1 --max-steps 2000 --global-batch-size 32 \
     --output-dir /tmp/ft_frozen
   
   # arm B: same run with the vision tower training
   CUDA_VISIBLE_DEVICES=0 uv run python gr00t/experiment/launch_finetune.py \
     --base-model-path nvidia/GR00T-N1.7-3B \
     --dataset-path demo_data/my_task \
     --embodiment-tag NEW_EMBODIMENT \
     --modality-config-path examples/SO100/so100_config.py \
     --num-gpus 1 --max-steps 2000 --global-batch-size 32 \
     --tune-visual \
     --output-dir /tmp/ft_tuned

4. 4
   Do the same on the lerobot side

   lerobot exposes the policy dataclass fields directly on the command line, so the flag is the config field. SmolVLA ships frozen, Pi0.5 ships unfrozen, so the interesting direction is opposite for each.

   bash

   lerobot-train \
     --policy.path=lerobot/smolvla_base \
     --dataset.repo_id=${HF_USER}/mydataset \
     --policy.freeze_vision_encoder=false \
     --policy.train_expert_only=false \
     --batch_size=64 --steps=20000 \
     --output_dir=outputs/train/smolvla_unfrozen \
     --job_name=smolvla_unfrozen \
     --policy.device=cuda

5. 5
   Judge on the arm, not on the loss curve

   Validation loss on teleoperated data rewards copying the operator's smoothing, not completing the task. Score both checkpoints over the same set of physical trials and count successes.

Testing the freeze question: by hand, or here

Where this platform does not help you

Three honest limits, because pretending otherwise wastes your money rather than ours. None of them is fixed by picking a different checkpoint.

• You cannot toggle the freeze from the form. The training form sends batch size, learning rate, max steps, gradient accumulation and the per-policy extras listed above. If you want the unfrozen arm of the GR00T experiment, that is a local Isaac-GR00T run today.
• You cannot change the encoder. Picking a policy picks its vision tower. The only encoder that is genuinely a config field is ACT's ResNet variant, and that is a lerobot feature rather than a platform one.
• Latency is physics, not a setting. The control loop runs at 20 ms per action step for ACT and 485 ms for Pi0.5, with GR00T N1.7 at 152 ms, GR00T N1.5 at 165 ms and SmolVLA at 245 ms. Adding public-internet round trips on top turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place and not for fast reactive motion - keep the policy next to the servos when the task is fast.

One more worth saying out loud: no encoder setting fixes a dataset. The minimum episode counts here are 50 for GR00T N1.7, GR00T N1.5, Pi0.5 and ACT, and 30 for SmolVLA, and those are minimums for the job to be worth running rather than targets for a policy that generalises. If your episodes all come from one camera pose under one lamp, an unfrozen SigLIP will happily learn that lamp. How to collect high-quality VLA training data covers the variation that pays, and the dataset directory shows what other people's coverage looks like.

Frequently asked questions