Writing Task Instructions for a VLA Model

A vision-language-action model takes three things on every control step: camera frames, joint positions, and a string of text. Two of those come from hardware you can measure and calibrate. The third is typed by a person once, at recording time, and then typed again later, possibly by a different person, when the policy is actually driving the arm. It is the only input to the model that nobody checks.

That string is not a caption and it is not documentation. It is tokenized, embedded, and concatenated with the image tokens before the action head runs. Change the wording and you change the input tensor, which changes the actions. This article traces where the instruction lives in a LeRobot dataset, what each of the five policies you can train on this platform does with it, how phrasing shifts behaviour, and the specific ways training-time and inference-time instructions drift apart. Everything about upstream repos below was read from the source on 2026-08-24 and is dated where it matters.

The instruction is a model input, not a label

Three ecosystems, three names for the same tensor. In LeRobot the field is called task. In NVIDIA's Isaac-GR00T it is an annotation channel, canonically annotation.human.task_description. In Physical Intelligence's openpi it is a field called prompt. All three end up in the same place: a token sequence that the vision-language backbone attends over while producing an action chunk.

This matters for fine-tuning because the instruction is part of what the model conditions on, not part of what it is scored against. If every episode in your dataset carries the same string, the model learns that this string means the whole behaviour, and it will happily produce that behaviour for anything close to it. If your episodes carry five different strings for what is really one behaviour, you have spent your episodes teaching the model that the words do not matter.

Where the string lives between your keyboard and the GPU

The instruction is written once and then referenced by number. The dataset layout keeps every unique string in one small metadata table with an integer task_index, and each frame in the parquet file carries only that integer. Which file that table is depends on the dataset version: LeRobot v3.0 writes meta/tasks.parquet, while the v2.1 layout that GR00T reads writes meta/tasks.jsonl. Both hold the same two columns. This indirection is why renaming a task after the fact is a metadata edit rather than a rewrite of millions of rows, and also why a mismatched index maps silently onto the wrong sentence.

// meta/tasks.jsonl - the v2.1 layout, one JSON object per line.
// LeRobot v3.0 stores the same two columns in meta/tasks.parquet.
{"task_index": 0, "task": "cube into yellow bowl"}
{"task_index": 1, "task": "cube into green bowl"}

// one row of data/chunk-000/*.parquet, GR00T flavour
{
  "observation.state": [-0.01, "...", 0],
  "action": [-0.010, "...", 0],
  "timestamp": 0.049,
  "annotation.human.task_description": 0,   // int index into the task table
  "task_index": 0,
  "episode_index": 0,
  "index": 0
}

Two consequences follow from this indirection. First, an episode recorded with a typo keeps that typo as a separate vocabulary entry forever, so "pick up the red cube" and "pick up the red cube " with a trailing space are two different tasks as far as the loader is concerned. Second, if you merge two datasets that were recorded on different days with slightly different wording, you have quietly built a two-task dataset out of one behaviour.

What each trainable policy does with your words

The five policies on the policy comparison page handle language in three different ways. GR00T reads a named annotation channel. Pi0.5 and SmolVLA read a task string through a tokenizer processor with an explicit token budget. ACT reads nothing. The token budgets below come from the policy configuration files in the LeRobot repository, read on 2026-08-24.

The original Pi0 uses 48 tokens as well; only Pi0.5 raised it to 200, and it needs the headroom because the discretized robot state is packed into the same prompt string as text. That design detail is easy to miss and it changes how much room your actual sentence has. If you are choosing between models on other grounds, the GR00T N1.7 against Pi0.5 comparison covers the rest of the trade-off.

The exact prompt string each model builds

Knowing that the string is tokenized is not enough, because none of these stacks tokenizes the raw string you typed. Each one wraps, cleans or reformats it first, and the wrapper is part of what the model learned.

GR00T: the annotation key has to match in three places

GR00T is the strictest of the three about naming. The same annotation is referenced from the parquet column, from meta/modality.json, and from the Python modality config, and the three spellings differ from each other by design. Datasets published by different groups use different keys, which is a common reason a GR00T fine-tune refuses to start.

# examples/SO100/so100_config.py - the language block of the SO-100 modality config
so100_config = {
    "video":  ModalityConfig(delta_indices=[0], modality_keys=["front", "wrist"]),
    "state":  ModalityConfig(delta_indices=[0], modality_keys=["single_arm", "gripper"]),
    "action": ModalityConfig(delta_indices=list(range(0, 16)), modality_keys=[...]),
    # Language: task instruction from annotation field in the dataset
    "language": ModalityConfig(
        delta_indices=[0],
        modality_keys=["annotation.human.task_description"],
    ),
}

register_modality_config(so100_config, embodiment_tag=EmbodimentTag.NEW_EMBODIMENT)

The episode loader asserts that the key you name exists under annotation in modality.json. If it does not, you get an assertion failure before the first training step, which is the good outcome. There is a second path: the loader also accepts the bare keys task and sub_task, which it reads from episodes.jsonl instead of the parquet column. Worth knowing before you write a conversion script for nothing.

Pi0.5: your sentence plus 256 bins of robot state

openpi builds one long prefix out of the instruction and the proprioceptive state. The state is digitized into 256 bins and written out as integers, in text, inside the same prompt. Underscores become spaces and newlines are flattened before anything is tokenized, which is why an underscore-joined LIBERO environment name works as an instruction without any manual cleanup.

# openpi/src/openpi/models/tokenizer.py - PaligemmaTokenizer.tokenize
cleaned_text = prompt.strip().replace("_", " ").replace("\n", " ")

if state is not None:
    # Pi0.5 format: the state is part of the discrete language input
    discretized_state = np.digitize(state, bins=np.linspace(-1, 1, 256 + 1)[:-1]) - 1
    state_str = " ".join(map(str, discretized_state))
    full_prompt = f"Task: {cleaned_text}, State: {state_str};\nAction: "
    tokens = self._tokenizer.encode(full_prompt, add_bos=True)
else:
    # Pi0 format: the state goes to the continuous action expert instead
    tokens = self._tokenizer.encode(cleaned_text, add_bos=True) + self._tokenizer.encode("\n")

SmolVLA: a trailing newline you did not type

LeRobot inserts a processor step before the tokenizer whose only job is to append a newline to your task string if it does not already end with one. The registry name is still smolvla_new_line_processor for backward compatibility with serialized processor configs. LeRobot's own docstring says the step exists because certain tokenizers, PaliGemma among them, expect a newline at the end of the prompt; openpi calls the same character the start-of-answer token. It applies to both a single string and a list of strings.

# lerobot/src/lerobot/policies/smolvla/processor_smolvla.py
input_steps = [
    steps.rename_observations,
    steps.add_batch_dim,
    NewLineTaskProcessorStep(),          # appends "\n" if the task lacks one
    TokenizerProcessorStep(
        tokenizer_name=config.vlm_model_name,   # HuggingFaceTB/SmolVLM2-500M-Video-Instruct
        padding=config.pad_language_to,          # "longest"
        padding_side="right",
        max_length=config.tokenizer_max_length,  # 48
    ),
    steps.to_device,
    steps.normalize,
]

How phrasing changes what the arm does

The clearest public illustration of this is the LIBERO-Spatial suite. LIBERO ships four task suites, and Spatial is ten tasks that share a verb, an object class and a target. The only thing that varies is a prepositional phrase saying which black bowl.

• pick up the black bowl between the plate and the ramekin and place it on the plate
• pick up the black bowl next to the ramekin and place it on the plate
• pick up the black bowl from table center and place it on the plate
• pick up the black bowl on the cookie box and place it on the plate
• pick up the black bowl in the top drawer of the wooden cabinet and place it on the plate
• pick up the black bowl on the stove and place it on the plate
• pick up the black bowl next to the plate and place it on the plate

Seven of the ten, verbatim from the task list at the end of the Isaac-GR00T LIBERO example (the underscores are environment names; the instruction is the same string with spaces). A policy that ignores the words scores at chance on this suite because the images look nearly identical. A policy that reads them has to ground "between", "next to" and "on" against the scene. NVIDIA reports 195 of 200 successful rollouts on LIBERO-Spatial for GR00T N1.7, fine-tuned at 20,000 max steps with a global batch size of 640 across 8 GPUs. That is a multi-GPU run, not anything you would launch for a single SO-100.

RT-2 made the format explicit. Its robot data was converted into a standard visual-question-answering string, and every demonstration in the underlying RT-1 dataset was annotated with a verb plus one or more object nouns from a small fixed skill set. That structure, verb first and object second, is still the shape that every VLA in this family was trained on. Our RT-2 walkthrough covers the web-knowledge transfer side of the paper in more depth.

# RT-2 (arXiv 2307.15818): robot data reformatted as visual question answering
Q: what action should the robot take to [task instruction]?  A:
# a possible answer, one integer per action dimension
1 128 91 241 5 101 127

# the chain-of-thought variant adds a plan step before the action tokens
Instruction: I'm hungry. Plan: pick rxbar chocolate. Action: 1 128 124 136 121 158 111 255.

Physical Intelligence put a limit on this in the Pi0.5 paper. Their model does hierarchical inference: it predicts a high-level subtask such as "pick up the plate" and then predicts actions conditioned on that subtask. Removing the verbal instruction data in an ablation, roughly 11 percent of the high-level mobile manipulation examples, degraded performance significantly. The limitation they state themselves is the one to remember: the prompts the model copes with are simple ones, and how complex a prompt it can take is set by the training data. You do not get instruction-following you did not record.

Rules that survive contact with a real dataset

The SmolVLA authors ran into the practical version of this at scale. Building their pretraining mixture from 481 community datasets covering 22.9 thousand episodes, they found task annotations that were placeholders like "task desc", one-word commands like "Hold" or "Up", or simply absent. Their fix was to re-annotate with Qwen2.5-VL-3B-Instruct, prompted to produce a short action-oriented sentence starting with a verb and capped at 30 characters. That cap is a useful anchor for how short a good instruction is.

1. 1
   Write the instruction before you record, not after

   Decide the exact string, put it in a shell variable, and never retype it. Every retype is a chance to introduce a trailing space or a capital letter that forks your task vocabulary. Start with a verb and name the object and the target explicitly.

   bash

   TASK="put the red cube in the white bowl"
   
   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader \
     --dataset.repo_id=$HF_USER/red-cube-white-bowl \
     --dataset.num_episodes=50 \
     --dataset.single_task="$TASK"

2. 2
   Keep it under the shortest budget you might ever use

   48 tokens is the SmolVLA and Pi0 limit. If there is any chance you will train the same dataset on SmolVLA later, write for 48 tokens now. A sentence of roughly ten to fifteen plain words is comfortably inside every budget on this platform.

   text

   good:  put the red cube in the white bowl
   good:  open the top drawer and put the bowl inside
   bad:   Task 3b - operator should grasp the red cube from the mat, lift it clear
          of the fixture, translate to the bowl and release (see SOP rev 2)

3. 3
   Use words the frozen backbone already knows

   GR00T freezes the language model during fine-tuning by default (tune_llm is False), and SmolVLA's default recipe sets train_expert_only to True and freeze_vision_encoder to True, so it trains the action expert rather than the full VLM. Neither is going to learn your internal vocabulary from 50 episodes. Say "red cube", not "part A"; say "bowl", not "target receptacle".

4. 4
   Verify what actually landed in the dataset

   Read the task table back before you spend money on a GPU. It takes three seconds and it catches every typo, every stray space and every accidental second task. LeRobot v3.0 writes it as parquet, so cat will not help you here.

   bash

   DS=~/.cache/huggingface/lerobot/$HF_USER/red-cube-white-bowl
   
   # LeRobot v3.0: meta/tasks.parquet, indexed by the task string
   python -c "import pandas as pd; print(pd.read_parquet('$DS/meta/tasks.parquet'))"
   
   # after converting down to v2.1 for GR00T it is JSON lines again
   cat $DS/meta/tasks.jsonl
   # {"task_index": 0, "task": "put the red cube in the white bowl"}
   
   # more than one row here means your dataset is multi-task,
   # whether or not you meant it to be

5. 5
   Pin the same string on the inference side

   Put the instruction in one file that both the recording script and the rollout script read. This is the single change that prevents most instruction drift, and it costs nothing.

   bash

   echo "put the red cube in the white bowl" > task.txt
   
   lerobot-rollout \
     --strategy.type=base \
     --policy.path=outputs/train/checkpoints/last/pretrained_model \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --task="$(cat task.txt)" \
     --duration=60

How training and inference drift apart

Drift is not exotic. It is the default outcome of a stack where the training string and the inference string are typed in two different terminals on two different days. Here are the four mechanisms worth knowing, all verified against upstream source on 2026-08-24.

Hugging Face says the quiet part out loud in its own SmolVLA guide. The evaluation command on that page carries an inline comment next to the --task flag telling you to use the same task description you used in your dataset recording. That comment exists because people do not, and the resulting failure looks like a bad checkpoint rather than a typo. The same page also notes that the SmolVLA reference dataset was 50 episodes across 5 cube positions, and that a 25-episode attempt performed badly.

LeRobot's interactive rollout mode makes the coupling visible in a way that is worth trying once. Start a rollout with --interactive=true and the robot stays idle until you type /start. From there /subtask sets the instruction the policy follows, live, without restarting anything. Changing the wording mid-run and watching the arm change its mind is the fastest way to find out whether your policy is actually language-conditioned or has just memorised one trajectory.

Two ways to get an aligned instruction into a trained policy

# 1. record with one fixed instruction
TASK="put the red cube in the white bowl"
lerobot-record --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
  --teleop.type=so100_leader --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=$HF_USER/red-cube --dataset.num_episodes=50 \
  --dataset.single_task="$TASK"

# 2. confirm the vocabulary is exactly one row
python -c "import pandas as pd; print(pd.read_parquet('$HOME/.cache/huggingface/lerobot/$HF_USER/red-cube/meta/tasks.parquet'))"

# 3a. SmolVLA reads the task column directly, nothing to convert
lerobot-train --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=$HF_USER/red-cube --batch_size=64 --steps=20000 \
  --output_dir=outputs/train/my_smolvla --policy.device=cuda

# 3b. GR00T needs LeRobot v2.1 plus a meta/modality.json with an annotation block
python scripts/lerobot_conversion/convert_v3_to_v2.py \
  --repo-id $HF_USER/red-cube \
  --root ~/.cache/huggingface/lerobot/$HF_USER/red-cube

# 4. run it back with the identical string
lerobot-rollout --strategy.type=base --policy.path=<checkpoint> \
  --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
  --task="$TASK" --duration=60

Multi-task datasets, and when they are worth it

Everything above pushes towards one instruction per dataset, which is the right default for a first training run. It is also the choice that gives up the entire point of a language-conditioned model. The decision is really about how many episodes you are prepared to record.

If you do go multi-task, balance matters. Forty episodes of "put the cube in the red bowl" and ten of "put the cube in the blue bowl" produce a policy that goes to the red bowl and occasionally hesitates. LeRobot also ships lerobot-annotate, which fills language_persistent and language_events columns on an existing dataset with a vision-language model, and can push the work to a Hugging Face Jobs GPU instead of your laptop. It writes new language columns rather than rewriting the task table, so it is a relative of the re-annotation the SmolVLA authors ran over 481 community datasets, not the same operation. Our guide on collecting high-quality VLA training data covers the recording side of a balanced multi-task set.

# lerobot-annotate: fill language columns on an existing dataset with a VLM
uv run lerobot-annotate \
    --root=/path/to/dataset \
    --vlm.model_id=Qwen/Qwen2.5-VL-7B-Instruct

# or push the work to a Hugging Face Jobs GPU instead of your machine
uv run lerobot-annotate \
    --repo_id=user/dataset \
    --new_repo_id=user/dataset_annotated \
    --push_to_hub=true \
    --job.target=h200

When the words are not the problem

Instruction drift is a real failure mode and it is cheap to rule out, which is why it belongs near the top of a debugging list rather than in the middle. But it is not the biggest lever on most setups, and pretending otherwise wastes time.

There is a second limit worth stating plainly. Language conditioning does not survive a slow control loop. Inference on this platform runs from 20 ms per action step for ACT to 485 ms for Pi0.5, and adding public-internet round trips on top of that turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place, not for fast reactive motion, and no amount of prompt engineering compensates. If you want the fastest loop, the ACT against SmolVLA comparison is the honest starting point, and the arena has 85 VLA models with 332 benchmark results if you want to see how the wider field measures instruction following.

One last framing that has held up across every stack examined here. The instruction is not an interface to the robot. It is an interface to the training data. What you can say to a policy is bounded by what somebody said while recording it, and the wider the range of phrasings in the dataset, the wider the range the policy will accept. That is a data collection decision made months before anyone types a prompt, and it is worth reading the VLA overview with that in mind.