How a Policy Sees Your Workspace: Resolution, Crop, Colour, FPS

A camera on a bench sees a workspace. A policy does not. Between the sensor and the first layer of the network sit a resize, a crop, a colour transform, a lossy video codec and a sampling clock, and each throws away information the model then cannot learn from. Most people never look at that chain. They look at the preview window, decide the scene is clear, record fifty episodes and are surprised when the arm reaches confidently for a spot two centimetres to the left of the cube.

If the arm itself is new to you, start with the complete SO-100 setup guide and come back. Every default below was read out of the source or the paper it comes from. Library numbers are from lerobot 0.6.1, published to PyPI on 3 August 2026, and from the Isaac-GR00T repository as of 25 August 2026. Upstream moves, so re-check the sources at the bottom before you trust a number.

The chain between the sensor and the first layer

Recording a LeRobot dataset looks like one step from the outside. It is at least six, and each row below is a place where the picture the model receives stops matching the one in your preview.

The last row surprises people. By the time a 1920x1080 stream reaches Pi0.5 it is a 224x224 square with black bars, and no care upstream puts those pixels back. The recording tutorial walks the capture side, and the guide to collecting high-quality VLA training data covers what to record. This page is about what happens after.

Resolution: four models, four different answers

There is no shared convention. Four of the five policies on the policies page are compared below, and each handles the incoming frame differently. That difference is not documented in one place anywhere upstream. This table is assembled from the config files and papers in the sources.

Two things follow. ACT is the only one whose input resolution is fully your decision, which cuts both ways: nobody stops you feeding it 1920x1080 and watching the ResNet18 feature map and the GPU bill grow together. And GR00T N1.7 is the only one that does not letterbox, because its vision-language-action backbone changed in that release to one that encodes images in their native aspect ratio without padding.

What that looks like in millimetres

Resolution arguments stay abstract until you convert them into working units. Take a common setup: one front camera at 640x480 covering roughly 40 cm of bench. The arithmetic is just division, but it decides whether a policy can see the gap between the fingers and the object.

Recorded frame:      640 x 480  = 307,200 px
Workspace width:     400 mm across 640 px  = 0.63 mm per pixel

Pi0.5, resize_with_pad to 224 x 224:
  scale = min(224/480, 224/640) = 0.35
  content = 224 x 168 px, centred, plus a 28-row black bar top and bottom
  content is 37,632 px, about 12% of what you recorded
  400 mm across 224 px = 1.79 mm per pixel

SmolVLA, resize_imgs_with_padding to 512 x 512:
  scale = min(512/480, 512/640) = 0.80
  content = 512 x 384 px, plus a single 128-row bar along the top
  content is 196,608 px, about 64% of what you recorded
  400 mm across 512 px = 0.78 mm per pixel

ACT, no resize:
  content = 640 x 480 px, 100% of what you recorded
  400 mm across 640 px = 0.63 mm per pixel

A 1.79 mm pixel is not a disqualification; Pi0.5 does useful manipulation at 224x224. But if your task hinges on a 2 mm alignment you are asking the model to resolve it inside roughly one pixel, and no amount of extra training steps will conjure that detail back. That is also why a quarter of a padded SmolVLA input is bar rather than scene for a 4:3 camera: the model spends part of a fixed token budget looking at filler. SmolVLA reduces each frame to 64 visual tokens using a pixel shuffle on the global image, with no tiling. Spread those 64 tokens evenly across the square and roughly 16 of them land entirely in the bar for a 4:3 source.

Crop: the model trains on a picture it never sees again

This is the easiest part to miss and the hardest to debug. The GR00T N1.7 fine-tuning recipe builds two separate transform pipelines, one for training and one for evaluation, and they are not the same picture. Read the lists below as the actual order of operations, from the albumentations builder in the repository.

# gr00t/model/gr00t_n1d7/image_augmentations.py
# build_image_transformations_albumentations(...)

# training pipeline, wrapped in A.ReplayCompose
A.SmallestMaxSize(max_size=max_size, interpolation=cv2.INTER_AREA)
FractionalRandomCrop(crop_fraction=fraction_to_use)
A.SmallestMaxSize(max_size=max_size, interpolation=cv2.INTER_AREA)
A.Rotate(limit=random_rotation_angle, p=1.0)      # only if the angle is non-zero
A.ColorJitter(brightness=..., contrast=..., saturation=..., hue=..., p=1.0)

# evaluation pipeline, a plain deterministic A.Compose
A.SmallestMaxSize(max_size=max_size, interpolation=cv2.INTER_AREA)
FractionalCenterCrop(crop_fraction=fraction_to_use)
A.SmallestMaxSize(max_size=max_size, interpolation=cv2.INTER_AREA)

The random crop is the point, not a bug: it stops the model memorising exact pixel coordinates. The ReplayCompose wrapper replays the same random draw across every camera view in one sample, so your front and wrist images stay geometrically consistent. But it does mean the framing at deployment is the centre crop, which the model saw only as one draw among many. If you set crop_fraction aggressively, you are also trimming the edges of your deployed field of view, and an object that sat near the frame border in your demonstrations may simply not be in the picture any more.

Pi0.5 does the same with different tooling. The paper lists its augmentation stack explicitly, applied to all input images in this order, and the shipped brightness, contrast and saturation values are identical to GR00T's defaults. That shared recipe is not visible in the head-to-head comparison of the two models, which reports what they do, not how they were fed.

# pi-0.5 paper, appendix: image augmentation, applied in this order
transforms = [
    augmax.RandomCrop(int(width * 0.95), int(height * 0.95)),
    augmax.Resize(width, height),
    augmax.Rotate((-5, 5)),
    augmax.ColorJitter(brightness=0.3, contrast=0.4, saturation=0.5),
]

Colour: three separate ways to lose it

1. Channel order

2. Deliberate colour damage during training

Both cloud-tier models jitter colour hard on purpose, and the shipped defaults are more aggressive than people expect.

A saturation factor of 0.5 means the model regularly trains on frames whose colour intensity is halved or half again as strong. That is the mechanism that lets a fine-tuned policy survive a change of light bulb. It is also why GR00T's README warns of 5 to 6 percent variance between runs from non-deterministic image augmentations, worth remembering before you conclude that one checkpoint beat another by four points.

3. Normalisation, which is not the same across policies

ACT normalises visual input with MEAN_STD, computed from your dataset's own statistics. SmolVLA and Pi0.5 both use IDENTITY for VISUAL and leave the pixels alone, normalising only state and action. So an ACT checkpoint carries your dataset's image statistics inside it, and swapping the dataset under a resumed ACT run changes what the network sees at its input. Pi0.5 also normalises state and action with QUANTILES, which needs q01 and q99 in meta/stats.json; an older dataset without them fails on the first batch.

Frame rate: what 30 fps buys and what it costs

LeRobot's DatasetRecordConfig defaults to fps=30, and almost every published SO-100 command uses it. That number is not just a recording setting: it is the unit every action count in every policy config is denominated in, and nothing in those configs reminds you of it.

Read the last two columns together. A frame arrives every 33 ms at 30 fps. Pi0.5 costs 485 ms per action step here, so about fifteen frames arrive in the time a single action step takes; GR00T N1.7 at 152 ms spans about four and a half; ACT at 20 ms is the only one that fits inside a frame interval. This is why action chunking exists at all: the policy commits to a burst of future actions precisely because it cannot think fast enough to act on every frame. It is also why inference latency is a data-collection concern and not only a deployment one, a point the primer on vision-language-action models develops further. Per-model numbers sit on the Pi0.5 and GR00T N1.7 pages.

Your frames are lossy before the model ever sees them

When video storage is on, which is the recording default, LeRobot does not keep your frames. It keeps an MP4 of them. The encoder defaults balance compression, quality and seek speed, and the documentation is explicit that changing them affects both recording CPU load and training image quality.

# lerobot 0.6.1 RGBEncoderConfig defaults, set with --dataset.rgb_encoder.<field>
vcodec        = "libsvtav1"   # "auto" picks a hardware encoder, falling back to this
pix_fmt       = "yuv420p"     # 4:2:0, so chroma is stored at half resolution in both axes
g             = 2             # a keyframe every 2 frames
crf           = 30            # abstract quality value, mapped per codec
preset        = 12            # when unset and vcodec is libsvtav1
fast_decode   = 0
video_backend = "pyav"

# these land in meta/info.json per camera, populated from the FIRST episode only
#   stream-derived, read back from the encoded file:
#     video.height, video.width, video.codec, video.pix_fmt, video.fps, video.channels
#   encoder-derived, copied from RGBEncoderConfig:
#     video.g, video.crf, video.preset, video.fast_decode, video.video_backend

Two consequences. yuv420p stores colour at half resolution in each axis, so a thin coloured edge, a red wire against a grey gripper for example, is degraded before any resize happens. And the info block is written once, from the first episode, assuming every episode used the same encoder; changing settings mid-recording is unsupported and the metadata will quietly describe only the first episode. The dataset documentation describes the rest of that metadata.

There is also a timing check you will meet eventually: a LeRobot dataset asserts that consecutive timestamps are separated by 1/fps plus or minus tolerance_s, which defaults to 1e-4 seconds. That window is deliberately tight: it is the guard that catches a camera silently dropping to 15 fps mid-episode. If it fires, do not raise the tolerance. Find out which camera stalled.

Which slot does each camera land in?

Resolution and colour are half the question. The other half is identity: a policy has named image inputs baked into its pretrained config, and your dataset's key names decide which camera occupies which slot. Get it wrong and the model reads your wrist view as the scene view, producing a policy that behaves plausibly and grasps nothing.

# your dataset keys on the left, the keys the policy expects on the right
lerobot-train \
  --dataset.repo_id=$HF_USER/so100_pick_place \
  --policy.path=lerobot/pi05_base \
  --rename_map='{"observation.images.front": "observation.images.base_0_rgb", \
                 "observation.images.wrist": "observation.images.left_wrist_0_rgb"}'

# fewer cameras than the policy expects: add masked placeholder slots
#   --policy.empty_cameras=1
# adds observation.images.empty_camera_0, filled with a masked dummy tensor
# (padded with -1 for SigLIP-based encoders, with a zero attention mask)
# supported by PI0, PI05, PI0Fast, SmolVLA and XVLA

# GR00T instead reads its video keys from meta/modality.json.
# The SO100 example declares exactly two:  "front" and "wrist"

The empty_cameras mechanism is well designed: the unused slot is masked rather than filled, so the model is told the view is absent instead of being taught that a black rectangle is a valid observation. Filling a missing camera with black frames by hand, which people do, teaches exactly the wrong thing.

Audit your own dataset in ten minutes

Before you spend a GPU hour, check what you recorded. All four steps run against a dataset you already have, locally or from the public dataset directory.

1. 1
   Read the metadata the recorder wrote

   meta/info.json holds the resolution, codec, pixel format and fps of every camera stream. If two cameras disagree on width, height or fps, stop and fix that first.

   bash

   python - <<'EOF'
   import json, pathlib, os
   root = pathlib.Path(os.environ.get("HF_LEROBOT_HOME", os.path.expanduser("~/.cache/huggingface/lerobot")))
   info = json.loads((root / "YOUR_USER/YOUR_DATASET/meta/info.json").read_text())
   for key, feat in info["features"].items():
       if feat.get("dtype") == "video":
           v = feat["info"]
           print(f'{key:45s} {v["video.width"]}x{v["video.height"]} '
                 f'{v["video.codec"]} {v["video.pix_fmt"]} @ {v["video.fps"]} fps '
                 f'crf={v.get("video.crf")}')
   EOF

2. 2
   Confirm what the cameras were actually asked for

   Auto-discovery reports each camera's default stream profile, which is often not what you assumed. A camera advertising 1920x1080 at 15 fps will not deliver a 30 fps dataset whatever you pass to the recorder.

   bash

   lerobot-find-cameras opencv
   # or: lerobot-find-cameras realsense

3. 3
   Pin resolution and fps explicitly when you record

   Never rely on the default profile. Name width, height and fps for every camera, and keep them identical unless you have a specific reason not to.

   bash

   lerobot-record \
       --robot.type=so100_follower \
       --robot.port=/dev/tty.usbmodem58760431541 \
       --robot.id=my_follower_arm \
       --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, \
                         wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}}" \
       --teleop.type=so100_leader \
       --teleop.port=/dev/tty.usbmodem58760431551 \
       --teleop.id=my_leader_arm \
       --dataset.repo_id=$HF_USER/so100_pick_place \
       --dataset.num_episodes=50 \
       --dataset.single_task="Pick up the cube and place it in the bin" \
       --display_data=true

4. 4
   Look at what the model will look at

   Downscale a representative frame to your policy's target size and inspect it at 100 percent. If you cannot tell whether the gripper is open, neither can the model. Cheapest test on the list, and the one people skip.

   python

   import cv2
   
   frame = cv2.imread("sample_frame.png")           # BGR, as OpenCV decodes it
   h, w = frame.shape[:2]
   
   for name, size, centred in (("pi05", 224, True), ("smolvla", 512, False)):
       s = min(size / h, size / w)                   # both keep the aspect ratio
       small = cv2.resize(frame, (int(w * s), int(h * s)), interpolation=cv2.INTER_AREA)
       dh, dw = size - small.shape[0], size - small.shape[1]
       if centred:                                   # openpi: centred, black
           top, left = dh // 2, dw // 2
       else:                                         # lerobot smolvla: left and top only
           top, left = dh, dw
       canvas = cv2.copyMakeBorder(small, top, dh - top, left, dw - left,
                                   cv2.BORDER_CONSTANT, value=(0, 0, 0))
       cv2.imwrite(f"asseen_{name}.png", canvas)
       print(name, canvas.shape, "content:", small.shape, "pad top/left:", top, left)

Two ways to get this right

pip install 'lerobot[smolvla]'

# check what the cameras really offer before recording anything
lerobot-find-cameras opencv

# record with resolution and fps pinned explicitly
lerobot-record \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --teleop.type=so100_leader \
    --teleop.port=/dev/ttyACM1 \
    --dataset.repo_id=$HF_USER/so100_pick_place \
    --dataset.num_episodes=50 \
    --dataset.single_task="Pick up the cube and place it in the bin"

# fine-tune from the published base checkpoint, not from scratch
lerobot-train \
    --policy.path=lerobot/smolvla_base \
    --dataset.repo_id=$HF_USER/so100_pick_place \
    --steps=20000 \
    --policy.device=cuda \
    --output_dir=outputs/train/smolvla_so100

Is fighting the preprocessing chain worth it?

Where this platform does not help

One segment of this chain is harder on a hosted service, not easier. Inference has to sit next to the servos for fast tasks. The control loop on this platform runs at 20 to 485 ms per action step depending on the model, and adding public-internet round trips turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place. It is not viable for fast reactive motion, and no amount of image tuning changes that.

• It cannot recover detail your camera never resolved. A 224x224 input is a hard ceiling on what Pi0.5 can see, wherever the GPU sits.
• It cannot fix a scene a human could not solve from the camera images alone. If you cannot do the task from the recorded views, the policy has no path either.
• It cannot make GR00T runs reproducible: that entry point has no seed and the augmentations are non-deterministic.
• For a fast reactive task the honest answer is local inference next to the arm, not a rented pod.

It cuts the other way too: the fastest model here, ACT at 20 ms per action step, has no base checkpoint at all, so it only exists once you have trained it on your task. There is no shortcut around collecting the data. To see what other people's data produced first, the model arena lists 85 VLA models with 332 benchmark results, each value linked to its paper or model card.