Language Annotations for VLA Training: The Task String

What a language annotation actually is

Every LeRobot dataset already carries language whether you thought about it or not. When you record an episode, the recorder attaches a task string to every frame. LeRobot then deduplicates: the string is written once into the task metadata with an index, and each parquet row keeps only that integer. An eighty-episode dataset with two instructions holds two strings on disk and 70,000 integers pointing at them.

That indirection makes the annotation cheap to store and easy to get wrong. Nothing in the pipeline reads what you typed. Type test and the dataset is valid, the trainer accepts it, the loss falls, and you have taught a vision-language-action model that test means whatever motion you demonstrated.

{"task_index": 0, "task": "cube into yellow bowl"}
{"task_index": 1, "task": "cube into green bowl"}

Where the string lives depends on the dataset version

Where the string enters the dataset

On the manual path there is exactly one place the instruction is set: a flag on the recorder. This is the LeRobot recording command as it stands on lerobot main on 23 August 2026, and it is the flow the desktop client wraps when recording from a teleoperation session.

1. 1
   Record with an explicit task string

   The value is copied onto every frame of every episode in this invocation. There is no per-episode override: one run of the recorder produces one task.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_follower \
     --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader \
     --dataset.repo_id=$HF_USER/pick_red_cube \
     --dataset.num_episodes=50 \
     --dataset.single_task="Pick up the red cube and put it in the box"

2. 2
   Read back what actually landed on disk

   Do this before you rent a GPU. total_tasks says how many distinct strings you ended up with, the task table says what they are. A v3.0 dataset has meta/tasks.parquet and no tasks.jsonl at all, so pick the right reader for your codebase_version.

   python

   import json, os, pandas as pd
   root = os.path.expanduser("~/.cache/huggingface/lerobot/USER/pick_red_cube")
   
   info = json.load(open(f"{root}/meta/info.json"))
   print(info["codebase_version"], info["total_episodes"], info["total_tasks"])
   
   # v3.0
   print(pd.read_parquet(f"{root}/meta/tasks.parquet"))
   # v2.1 instead:
   # print(open(f"{root}/meta/tasks.jsonl").read())

3. 3
   Convert down if you are training GR00T

   Isaac-GR00T ships the downconverter. It rewrites the v3.0 layout into the v2 layout the GR00T loader expects, task metadata included.

   bash

   uv run --project scripts/lerobot_conversion \
     python scripts/lerobot_conversion/convert_v3_to_v2.py \
     --repo-id $HF_USER/pick_red_cube \
     --root examples/SO100/pick_red_cube_lerobot

The GR00T annotation key: three names that must agree

GR00T does not read the LeRobot task_index column by name. It reads an annotation key, and the naming convention is annotation..(.). The segments after the prefix are chosen by whoever built the dataset, which is why two well-known GR00T datasets use different keys for the identical thing. All three layers have to line up or nothing loads.

There is a third form in the wild. The pre-registered embodiment configs inside Isaac-GR00T use the short form for SO-100 and the two Unitree G1 entries, the long form for LIBERO, both SimplerEnv entries and RoboCasa Panda, and annotation.language.language_instruction for OXE DROID. None is more correct. What matters is that the key in your modality.json is the key your data config asks for. If it is not, the loader does not warn, it asserts: Key not found in language modality.

{
  "state":  { "single_arm": {"start": 0, "end": 5},
              "gripper":    {"start": 5, "end": 6} },
  "action": { "single_arm": {"start": 0, "end": 5},
              "gripper":    {"start": 5, "end": 6} },
  "video":  { "front": {"original_key": "observation.images.front"},
              "wrist": {"original_key": "observation.images.wrist"} },
  "annotation": {
    "human.task_description": { "original_key": "task_index" }
  }
}

original_key is the most useful line in the schema and the easiest to miss: it lets the annotation key point at a column that already exists. A dataset from the normal recorder has a task_index column and nothing named annotation.anything, so rather than rewriting every parquet file to add a duplicate, you declare that the annotation key reads task_index. The shipped SO-100 and LIBERO modality.json files both do this.

What each policy actually does with your sentence

The string does not reach the model as a string. A preprocessing chain sits between the dataset and the transformer, and each of the five policies on the policies page has a different one. Which one you are feeding changes how you should write the instruction.

The Pi0.5 template explains two behaviours people report as bugs. Underscores in your task name silently become spaces, and the proprioceptive state is discretised into 256 bins and pasted into the same prompt. That state paste is on by default: the processor sets include_state_in_prompt = not use_proprioceptive_memory, and the Pi0.5 config defaults that flag to False while capping the tokenizer at 200 tokens. Your sentence is not the only thing competing for them.

# src/lerobot/policies/pi05/processor_pi05.py, lerobot main, 23 August 2026
cleaned_text = task.strip().replace("_", " ").replace("\n", " ")
if discretized_states is None:
    full_prompt = f"Task: {cleaned_text};\nAction: "
else:
    state_str = " ".join(map(str, discretized_states[i]))
    full_prompt = f"Task: {cleaned_text}, State: {state_str};\nAction: "

# src/lerobot/processor/newline_task_processor.py - what SmolVLA does instead
if not task.endswith("\n"):
    new_complementary_data["task"] = f"{task}\n"

ACT deserves a plain statement, because it is the cheapest policy to train and the most misunderstood here. Grep modeling_act.py and its config for language and the only hit is the Apache licence header. ACT is a policy conditioned on images and joint state, nothing else. Record two tasks into one dataset, train ACT on it, and ACT averages them. That is not a language failure, it is a policy without language.

What a good instruction looks like

The SmolVLA paper is the best public evidence on what bad annotations look like, because its authors had to clean 481 community datasets into a pretraining mix of 22.9K episodes and 10.6M frames. They report substantial noise: ambiguous placeholders such as task desc, overly vague commands such as Hold or Up, and datasets that lacked instructions entirely. Their fix was to run Qwen2.5-VL-3B-Instruct over sampled frames and regenerate a short, action-oriented sentence per dataset, prompted for a maximum of 30 characters starting with a verb like Pick, Place or Open.

Thirty characters is a useful anchor: roughly the length of Pick up the cube and place it, well below SmolVLA's 48-token budget. The instruction is not where you put nuance. It is where you put the one distinguishing fact the policy cannot read off the camera.

The sandwich example is a dataset you can open right now: izuluaga/finish_sandwich is 80 episodes, 70,277 frames and exactly two tasks, and the reference evaluation in Isaac-GR00T passes --lang_instruction="finish the ham cheese olives sandwich" at inference. Character for character. That is the discipline the mechanism depends on.

What happens when the phrasing differs at inference

Until recently this was folklore. It is now measured. LIBERO-Para, published in March 2026, holds the task, the scene and the desired outcome constant and varies only the wording, across seven VLA configurations from 0.6B to 7.5B parameters. The abstract is blunt: consistent degradation of 22 to 52 percentage points under paraphrasing.

Put the last row next to the first. A GR00T N1.7 fine-tune scores in the mid-to-high nineties on LIBERO using the benchmark's own strings, and the Spatial suite is ten variants of one pick-and-place whose descriptions differ only in the phrase locating the bowl: pick up the black bowl between the plate and the ramekin, next to the ramekin, on the cookie box. The sentence is what selects the target, so the model is clearly reading it. It is reading it as a lookup key, not as a description.

A second paper, from February 2026, arrives from a different direction. Testing SmolVLA and Pi0.5 on multi-object picking with progressively randomised placement, the authors report that in the harder regimes execution of the manipulation primitive stays substantially more reliable than instruction-conditioned task success, which they take as evidence that skill acquisition is decoupled from instruction following. They recommend reporting the two separately rather than as one success number. In plain terms: the arm can be good at picking things up and still be unreliable about picking the thing you named.

# Isaac-GR00T, examples/SO100/README.md - the inference-side flag
PROMPT="finish the ham cheese olives sandwich"

uv run --no-sync python eval_so100.py \
  --robot.type=so101_follower \
  --robot.port="$ROBOT_PORT" \
  --robot.id="$ROBOT_ID" \
  --policy_host=localhost \
  --policy_port=5555 \
  --lang_instruction="$PROMPT"

Inside the GR00T Policy API the same value arrives as a nested observation dictionary with exactly three modalities: video, state and language. Language is passed as {"task": [[str]]}, shape (B, 1), a list of lists of strings, and the docs note it is normally a single timestep. Nothing validates that string against the checkpoint it is sent to.

Doing it by hand, or doing it here

Do it yourselfOn AY-Robots

Everything above runs today with two open-source repos and one GPU. The work is mostly bookkeeping, and the bookkeeping is where runs die.

1. Record with lerobot-record --dataset.single_task="...", and write the exact sentence somewhere that is not your shell history.
2. Check codebase_version and total_tasks in meta/info.json. A larger total_tasks than you intended means a typo is now part of your dataset.
3. For GR00T, convert v3.0 down to v2 with scripts/lerobot_conversion/convert_v3_to_v2.py.
4. Copy a modality.json whose annotation key matches your data config into the dataset's meta/ directory: for SO-100, human.task_description with original_key: task_index, which is exactly what examples/SO100/modality.json ships.
5. Fine-tune with examples/finetune.sh --modality-config-path examples/SO100/so100_config.py --embodiment-tag NEW_EMBODIMENT.
6. Evaluate open-loop with gr00t/eval/open_loop_eval.py, then closed-loop with --lang_instruction set to the exact recorded string.

bash

CUDA_VISIBLE_DEVICES=0 NUM_GPUS=1 uv run bash examples/finetune.sh \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path examples/SO100/pick_red_cube_lerobot/$HF_USER/pick_red_cube \
  --modality-config-path examples/SO100/so100_config.py \
  --embodiment-tag NEW_EMBODIMENT \
  --output-dir /tmp/so100_finetune

One reproducibility caveat

The Isaac-GR00T fine-tune entry point is a tyro CLI that exposes no seed, so GR00T runs are not bit-for-bit reproducible. That matters when you are trying to attribute a behaviour change to an annotation change. lerobot's trainer does expose a seed, defaulting to 1000.

A checklist before you launch the run

1. Confirm codebase_version matches what your model wants and total_tasks matches what you intended.
2. Print the task table and read every string out loud. If one is a filename, a version number or the word test, re-annotate before training.
3. For GR00T, diff the annotation block in meta/modality.json against the modality_keys in your data config. String equality is the whole check.
4. Decide whether your policy reads the field at all. ACT on a two-task dataset gives one averaged behaviour however good the annotations are.
5. Copy the exact instruction into your inference command rather than retyping it, and put it in the dataset card if anyone else will drive the policy.
6. Budget the episodes. The platform minimum is 50 for GR00T N1.7, GR00T N1.5 and Pi0.5, 30 for SmolVLA, and that is per run, not per instruction.

Where this does not help

A language annotation is a conditioning input, not a specification. It cannot fix inconsistent demonstrations, and it does not make an imitation learning run generalise to an object you never showed it. The 22 to 52 pp paraphrase gap is a property of small-data fine-tuning, and no care in writing the sentence closes it on 50 episodes.

• It does not affect inference latency noticeably. The 20 to 485 ms per action step is set by the architecture, not your sentence.
• It does not make remote inference viable for fast reactive motion. Public-internet round trips on top of a 152 ms GR00T step turn a working policy into a hesitant one, whatever the instruction says.
• Rich, compositional instructions of the kind RT-2 demonstrated come from internet-scale co-training, not a 50-episode fine-tune on your bench.

Related reading

The RT-2 walkthrough covers how actions became text tokens, the VLA overview puts the five policies in context, and the data collection guide covers the demonstrations these annotations attach to. Benchmarks for 85 models live in the arena; the SO-100 page covers the arm.