Multi-Task Datasets for One Policy: Language and Task Balance

Almost every LeRobot dataset you record is already multi-task in the format sense: it has a task field, a task index, and the trainer reads both. Whether that becomes one policy that does several things, or one policy that does the average of several things badly, comes down to which task string you write, how many frames each task contributes, and whether your model can read language at all.

This page covers all three: the file layout, the CLI flags on lerobot main and Isaac-GR00T main as of 23 August 2026, and the published numbers from BC-Z, RT-1 and SmolVLA. It also says where multi-task training makes things worse, and where ACT on one task still beats a vision-language-action model trained on five.

What a multi-task LeRobot dataset actually is

Here is the on-disk shape you are working with. Everything about tasks lives in two places: one metadata file and one column.

my_dataset/
  data/
    chunk-000/
      file-000.parquet        # one row per frame, includes task_index
      file-001.parquet
  meta/
    info.json                 # fps, features, total_tasks
    stats.json
    tasks.parquet             # the task strings, indexed by task_index
    episodes/
      chunk-000/file-000.parquet   # per-episode metadata: tasks, length, ...
  videos/
    observation.images.front/chunk-000/file-000.mp4
    observation.images.wrist/chunk-000/file-000.mp4

The lerobot source describes that metadata in one sentence: tasks contains the prompts for each task of the dataset, which can be used for task-conditioned training. Nothing forces one task per dataset. 120 episodes over four task strings is a legal v3.0 dataset.

Where the task string comes from

The recorder writes one task string per run. Recording a dataset with lerobot-record takes --dataset.single_task, singular, and every frame in that invocation gets it. The natural unit of collection is one task per run, so a multi-task dataset is something you assemble afterwards.

# one run, one task string, 25 episodes
lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/tty.usbmodem585A0076841 \
    --robot.id=my_follower_arm \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/tty.usbmodem58760431551 \
    --teleop.id=my_leader_arm \
    --dataset.repo_id=${HF_USER}/cubes_pick \
    --dataset.num_episodes=25 \
    --dataset.single_task="Pick up the red cube and put it in the box" \
    --dataset.streaming_encoding=true \
    --dataset.encoder_threads=2

What multi-task training actually buys you

Three papers give usable numbers, and they do not all point the same way.

BC-Z: pooling across tasks rescued a task that single-task training could not learn

BC-Z (Jang et al., CoRL 2021) collected 25,877 demonstrations across 100 tasks, 125 hours of robot time, 12 robots and 7 operators, plus 18,726 human videos. The policy is a ResNet-18 conditioned through FiLM layers on a 512-dimensional task embedding, running closed-loop at 10 Hz.

Its sharpest ablation is on place the bottle in the ceramic bowl, the task with the most data at 1000 demonstrations. A single-task policy on those 1000 demos reached 5 percent; the multi-task language-conditioned policy trained on all 25,877 reached 52. The paper's own reading: collecting over several robots and operators likely makes any one task harder to learn, and only pooling across tasks recovered it.

Three things fall out. Language matches a one-hot task id on the 21 training tasks, 42 against 40, so the embedding is not the bottleneck. Language is the only conditioning that transfers to unseen tasks, 32 percent against 4 for video. And the bottom row is the warning: an unrelated processing choice, the adaptive state-diff, dropped the same multi-task policy from 45 to 3 percent.

RT-1: task diversity beat task quantity

RT-1 (Brohan et al., 2022) trained a 35M-parameter transformer at 3 Hz on roughly 130k episodes covering over 700 instructions, gathered with 13 robots over 17 months. Its data ablation is the most useful result in this area because it separates how much data from how many tasks.

Read the columns carefully, because the summary column is easy to misquote. Generalization, all is the mean of the three columns to its right, not one of them: 14, 31 and 41 average to 29 on the 22-percent row. Seen-task success is also not monotonic in dataset size, 55 at 37 percent of the data against 59 at 22 percent.

SmolVLA: multi-task on an SO-100, with the awkward row included

SmolVLA is the closest published result to what most readers will actually run: a 450M-parameter VLA, three real tasks, an SO-100 arm. It was pretrained on 481 community datasets totalling 22.9K episodes and 10.6M frames, then fine-tuned.

The averages give the quotable line: pretraining on the community mixture lifts the multi-task average from 51.7 to 78.3. The per-task columns give a second one. Without pretraining, going multi-task raised Pick-Place from 55 to 80 but lowered Stacking from 45 to 40. And single-task ACT scored 50 on Stacking against multi-task Pi0's 40, Pi0 at 3.5 B parameters against ACT's roughly 80 M.

Which policies can even read a task string

This is where a lot of multi-task datasets quietly become pointless. Before you spend a week writing careful instructions, check whether your model has a text encoder at all.

Latency figures are the ones AY-Robots publishes on the policy comparison page. The ACT row catches people: on lerobot main, configuration_act.py declares no tokenizer, no text encoder and no task field, and modeling_act.py never mentions a tokenizer. ACT is an imitation learning architecture mapping images and joint state to an action chunk, and nothing in it reads text.

Assembling several tasks into one dataset

Three runs, one task each. Now you need one dataset: merge, relabel, check.

1. 1
   Record each task as its own run

   Keep camera positions, arm id and fps identical across runs. Merge requires identical features, and aggregate_datasets validates fps, robot type and features before concatenating anything.

   bash

   lerobot-record --dataset.repo_id=${HF_USER}/cubes_pick   --dataset.single_task="Pick up the red cube and put it in the box"   --dataset.num_episodes=25 ...
   lerobot-record --dataset.repo_id=${HF_USER}/cubes_stack  --dataset.single_task="Stack the red cube on the blue cube"       --dataset.num_episodes=25 ...
   lerobot-record --dataset.repo_id=${HF_USER}/cubes_sort   --dataset.single_task="Sort the cubes by colour into the two bins" --dataset.num_episodes=25 ...

2. 2
   Merge them into a single repo id

   Episodes are concatenated in the order you list the repo ids. The task strings survive and become separate entries in meta/tasks.parquet.

   bash

   lerobot-edit-dataset \
       --repo_id ${HF_USER}/cubes_multitask \
       --operation.type merge \
       --operation.repo_ids "['${HF_USER}/cubes_pick', '${HF_USER}/cubes_stack', '${HF_USER}/cubes_sort']"

3. 3
   Fix the task strings in place

   modify_tasks rewrites meta/tasks.parquet, the task_index column, the episode metadata and total_tasks in info.json. It edits in place, so --new_repo_id and --new_root are ignored. Back up first.

   bash

   # standardise wording across the merged dataset
   lerobot-edit-dataset \
       --repo_id ${HF_USER}/cubes_multitask \
       --operation.type modify_tasks \
       --operation.task_replacements '{"Sort the cubes by colour into the two bins": "Sort the cubes by color into the two bins"}'
   
   # or relabel specific episodes
   lerobot-edit-dataset \
       --repo_id ${HF_USER}/cubes_multitask \
       --operation.type modify_tasks \
       --operation.episode_tasks '{"0": "Pick up the red cube and put it in the box", "63": "Stack the red cube on the blue cube"}'

4. 4
   Check the task counts before you rent a GPU

   Count frames per task, not episodes per task. Frames are what the sampler uses.

   python

   from lerobot.datasets.lerobot_dataset import LeRobotDataset
   
   ds = LeRobotDataset("your-user/cubes_multitask")
   print(ds.meta.total_tasks)
   print(ds.meta.tasks)          # task string -> task_index
   
   import collections
   frames = collections.Counter()
   eps = ds.meta.episodes
   for i in range(ds.num_episodes):
       task = eps["tasks"][i][0] if eps["tasks"][i] else ""
       frames[task] += eps["length"][i]
   for task, n in frames.most_common():
       print(f"{n:>8}  {task}")

Task balance: what the sampler is really doing

The word balance does a lot of unexamined work here, so: lerobot trains with EpisodeAwareSampler, which its docstring calls a sampler over episode frames. It keeps only the per-episode frame boundaries, then shuffles each epoch with one torch.randperm over every frame, seeded from the run seed and the epoch number. No per-task weighting anywhere in it, and no per-task argument to give it.

A task's effective weight is therefore its share of the total frames. Twenty-five episodes of a 40-second stack and twenty-five of a 12-second pick contribute frames at roughly 10:3, so the stack gets about three times the gradient updates even though the episode counts match.

Two deserve a note. lerobot's eval_split is stratified by task by design: make_train_eval_datasets groups episodes by task string and holds out the tail of each group, so eval_split 0.2 on five tasks gives an eval set containing all five.

GR00T's ds_weights_alpha is the only mixture control the fine-tune CLI exposes. Its docstring: each dataset's sampling weight is len(dataset)^alpha and per-dataset mix_ratio values are ignored. Alpha 1 is plain pooling by length, alpha 0 weights every dataset equally regardless of size. Either way launch_finetune.py pools every path you pass into a single entry at mix_ratio 1.0, so genuine per-dataset ratios have to come from a config file loaded by launch_train.py, not from a flag.

Writing task strings the model can use

SmolVLA's authors hit this at scale. Across 481 community datasets they found substantially noisy annotations: placeholders like task desc, commands as vague as Hold or Up, and datasets with no instruction at all. Their fix was to run Qwen2.5-VL-3B-Instruct over representative frames to write a short, action-oriented sentence, and to rename cameras by hand to OBS_IMAGE_1 through OBS_IMAGE_3.

• Write the instruction the way a person would say it out loud: "Pick up the red cube and put it in the box", not "pick_red_cube_v2".
• Keep it short. SmolVLA sets tokenizer_max_length = 48, so anything past roughly 48 tokens never reaches the model.
• Separate the strings by content words, not punctuation or casing. "Pick the cube" and "pick the cube." are two different task_index values.
• Name the object and the destination when they vary. If two tasks differ only in which bin, the string is the only signal the policy has.
• Use exactly one string per task across all runs, standardised afterwards with --operation.type modify_tasks.
• Do not encode the episode number, the date or the operator. Every unique string is a separate task as far as the format is concerned.

lerobot-rollout \
  --strategy.type=base \
  --policy.path=${HF_USER}/my_smolvla_multitask \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM1 \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --task="Stack the red cube on the blue cube" \
  --duration=60

Two routes to the same multi-task policy

When a single-task ACT still beats a multi-task VLA

The honest version is not "VLAs win". VLAs win on the axes a VLA is built for and lose on the one a small from-scratch model is built for. Latency is the axis nobody argues with: on this platform ACT runs at 20 ms per action step, GR00T N1.7 at 152 ms, SmolVLA at 245 ms and Pi0.5 at 485 ms.

Language conditioning also has a measured cost. Octo, evaluated zero-shot on the WidowX tasks, reached a 25 percent higher success rate under goal-image conditioning than under language; the authors say this is likely because a goal image carries more information about how to achieve the task. Two tasks that differ only by a fine spatial detail may not be separable by a sentence.

Evaluating a policy that does several things

A single average across tasks is the wrong summary statistic. Report a per-task table before and after, and hold out episodes per task rather than off the end of the dataset. The training docs cover the run mechanics this section assumes.

# per-task holdout: the last 20% of episodes of EACH task go to eval
lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=${HF_USER}/cubes_multitask \
  --dataset.eval_split=0.2 \
  --steps=20000 \
  --output_dir=outputs/train/smolvla_multitask \
  --policy.device=cuda

Check the noise floor before concluding a mixture change helped. NVIDIA's GR00T README notes users may observe 5-6 percent variance between runs from non-deterministic image augmentations alone, and the Isaac-GR00T fine-tune entry point exposes no seed at all, so GR00T runs are not bit-for-bit reproducible (lerobot's default seed is 1000). A 3-point per-task difference between two checkpoints tells you nothing.

What to do next

One fast task: train an ACT and skip everything above. Three tasks on the same rig: record separately, merge, count frames per task rather than episodes, and fine-tune SmolVLA with eval_split set so the holdout contains all three. Then score each task on its own against the single-task baselines you already have. That comparison is cheap on the 24 GB tier and it is the only thing that tells you whether multi-task is better for your tasks.

Related reading: the BC-Z breakdown, how to collect high-quality VLA training data on the recording side, and the VLA overview. Benchmark data for 85 models sits in the arena, and the format details this page skipped are in the dataset docs.