Curriculum Learning for Manipulation: Does Demo Order Matter?

What curriculum learning actually claims

The idea is old and intuitive. Nobody learns integration before addition, so maybe a network fed easy examples first ends up somewhere better than one fed everything at random. Bengio, Louradour, Collobert and Weston formalised it at ICML 2009 in Curriculum Learning, noting that humans and animals learn better when examples are 'organized in a meaningful order which illustrates gradually more concepts, and gradually more complex ones'. They framed it as a continuation method: train on a smoothed, easier objective, then anneal toward the real one.

Their headline experiment is worth knowing precisely, because most people citing the paper have forgotten what it measured. On a shape recognition dataset they trained a three hidden layer network for 256 epochs and varied one number: the 'switch epoch' at which training moved from the easy set to the target set, zero meaning no curriculum. The result was that 'the best generalization is obtained by doing a 2-stage curriculum where the first half of the total allowed training time (of 256 epochs) is spent on the easier examples'. That is not a per-example ordering inside a shuffled epoch. It is a two-stage schedule with a hard boundary, the shape of pre-training followed by fine-tuning.

Your recording order is destroyed before the first gradient step

This is what decides the question for most readers. If you record fifty episodes on an SO-100 and fine-tune ACT or SmolVLA on the resulting LeRobot dataset, the order you recorded them in is not an input to training. It is metadata. The dataloader shuffles.

# lerobot/src/lerobot/datasets/sampler.py (main, checked 2026-08-23)
#
# "Each epoch is shuffled with a `torch.randperm` seeded from `(seed, epoch)`,
#  so the data order is a pure function of `(seed, epoch)`"

# lerobot/src/lerobot/scripts/lerobot_train.py
sampler = EpisodeAwareSampler(
    dataset.meta.episodes["dataset_from_index"],
    dataset.meta.episodes["dataset_to_index"],
    episode_indices_to_use=dataset.episodes,
    drop_n_last_frames=getattr(active_cfg, "drop_n_last_frames", 0),
    shuffle=True,
    seed=cfg.seed if cfg.seed is not None else 0,
)

GR00T reaches the same place differently. Its fine-tuning training loop subclasses the Hugging Face Trainer and feeds it a sharded mixture dataset whose shard schedule is built from a seed. Order in, order out, no relationship, and no flag to turn it off in either trainer.

When curricula do work, and by how much

The cleanest answer is Wu, Dyer and Neyshabur, When Do Curricula Work?, ICLR 2021. Running curriculum, anti-curriculum and random order across a grid of settings, they report that 'curricula have only marginal benefits, and that randomly ordered samples perform as well or better than curricula and anti-curricula, suggesting that any benefit is entirely due to the dynamic training set size'.

They did find two regimes where a curriculum earns its keep: 'with limited training time budget or in existence of noisy data'. Hold onto both, because a bench-scale robot dataset is often exactly that. A 50-episode SO-100 set is a limited budget, and human teleoperation data is noisy by construction. The honest reading is still that the effect is small, easily confounded, and not where your success rate is hiding.

• Soviany, Ionescu, Rota and Sebe's IJCV survey (arXiv v1, January 2021) is candid about the cost: you need 'a way to rank the samples from easy to hard, as well as the right pacing function'. For robot demonstrations, nobody has a defensible difficulty score.
• Narvekar and colleagues' JMLR 2020 survey covers the RL side, where the framing is task sequencing rather than example ordering: experience from one task 'can be leveraged when starting to learn the next, harder task'.
• Hou, Hindriks, Eiben and Baraka's Active Robot Curriculum Learning from Online Human Demonstrations (March 2025) comes closest to a demonstration curriculum, guiding the human to demonstrate 'in situations of gradually increasing difficulty'. Its 26-participant study reports less demonstrator time and fewer failed attempts, so much of the win is on the human side.

Four places where order genuinely survives

1. Stage boundaries

Bengio's switch epoch has a direct descendant in every modern vision-language-action model. Physical Intelligence describe pi0's recipe as mirroring 'the pre-training/post-training separation' from language models: 'the pre-training dataset should cover as many tasks as possible', while 'the post-training dataset should instead cover behaviors that are conducive to effective task execution, which should exhibit a consistent and fluent strategy'. Diverse and rough first, clean and consistent last.

pi0.5 hardens the split: it 'is trained in two stages', pre-training on discrete action tokens across diverse platforms and web data, then post-training with flow matching for the target robot. GR00T N1 stacks the same idea as a 'data pyramid' where 'data quantity decreases, and embodiment-specificity increases, moving from the bottom to the top'. Fine-tune GR00T N1.7 or Pi0.5 on your own arm and you are the final stage of somebody else's curriculum. That is the ordering effect you get, and it is free.

2. Dataset mixture weights

The second survivor is how often each source gets sampled, and this is where published runs really do intervene. pi0 weights 'each task-robot combination by n^0.43, where n is the number of samples for that combination, such that over-represented combinations are down-weighted'. Octo doubles the weight of its more diverse datasets. OpenVLA adopts Octo's weights and then does something unambiguously curricular: it added DROID 'at a conservative mixture weight of 10%', saw action token accuracy on it stay low, and 'removed DROID from the data mixture for the final third of training'.

# Isaac-GR00T, main branch, checked 2026-08-23.
# Multiple dataset roots joined by the OS path separator, plus a
# power-law mixture exponent over them.

uv run python gr00t/experiment/launch_finetune.py \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path "/data/easy_variant:/data/hard_variant" \
  --embodiment-tag NEW_EMBODIMENT \
  --modality-config-path examples/SO100/so100_config.py \
  --num-gpus 1 \
  --output-dir /tmp/ft_mix \
  --max-steps 2000 \
  --global-batch-size 32 \
  --ds-weights-alpha 0.43

Alpha 1.0 is proportional: a set twice the size is seen twice as often. Alpha 0.0 is uniform, equal airtime regardless of size. In between compresses the imbalance, which is what pi0's 0.43 does. If you recorded forty easy episodes and ten hard ones and want the hard ones to count, this is the knob, not the recording order.

3. Sequential fine-tuning of one checkpoint

The moment you stop pooling data and start chaining checkpoints (train on task A, then fine-tune that on task B), order becomes load bearing, because each stage starts from the last one's weights. LIBERO, the lifelong robot learning benchmark (NeurIPS 2023 Datasets and Benchmarks), ran the same algorithms over five task orderings and found that 'different task ordering could result in very different performances for the same algorithm', statistically significant for PackNet. Their framing is blunt: 'a robot in the real world, however, often cannot choose which task to encounter first'.

4. Environment curricula, which need a simulator

The most convincing curricula in robotics are not about demonstrations. Florensa and colleagues' reverse curriculum (CoRL 2017) trains 'in reverse, gradually learning to reach the goal from a set of start states increasingly far from the goal'. Automatic Domain Randomization, from OpenAI's Solving Rubik's Cube with a Robot Hand (October 2019), widens an environment parameter's randomisation range when average performance beats a high threshold and narrows it below a low one. Both need what a bench arm cannot give you: resetting the world to an arbitrary state, thousands of times, for free.

What moves the number instead of ordering

If ordering is a rounding error, name what is not. The robomimic study (CoRL 2021) collected the same tasks from six teleoperators, fifty demonstrations each, in a 'better' group of two experienced operators, an 'okay' group of two adequate ones and a 'worse' group of two inexperienced ones. Training on different subsets of that pool moves success rates far more than reshuffling ever would. Belkhale, Cui and Sadigh's Data Quality in Imitation Learning (June 2023) names the mechanism: action divergence, 'the mismatch between the expert and learned policy at certain states', plus transition diversity. Their inconvenient finding is that 'state diversity is not always beneficial'.

Scene count is the other lever with real evidence. pi0.5 varied training locations across 3, 12, 22, 53, 82 and 104, holding unique samples seen constant at 40k steps so dataset size was not the confound, and performance on the four held-out tasks improved with more locations. DROID went the same way at collection time: 76,000 trajectories, 564 scenes, 84 tasks, 50 collectors over twelve months. RT-1's data was gathered 'over the course of 17 months with a fleet of 13 robots', then trained as one pooled shuffle. Nobody sequenced those 17 months into a curriculum.

The long version of that argument is in how to collect high-quality VLA training data, and the multi-source pooling argument in the Open X-Embodiment write-up. Both matter more than a curriculum.

A recording order that is actually worth following

There is still a right order to record in. It optimises for you and your pipeline, not for the network: catch a mislabelled camera after five episodes instead of eighty, and keep nuisance variables from lining up with task variants.

1. 1
   Calibrate, then record one throwaway episode

   Record one episode and load it before recording a second. Most wasted sessions are a moved camera index or an empty task string. Delete it afterwards.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/tty.usbmodem58760431541 \
     --robot.cameras="{front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
     --robot.id=black \
     --teleop.type=so100_leader \
     --teleop.port=/dev/tty.usbmodem58760431551 \
     --teleop.id=blue \
     --dataset.repo_id=you/canary \
     --dataset.num_episodes=1 \
     --dataset.single_task="Pick up the red cube and place it in the bowl" \
     --display_data=true

2. 2
   Record five episodes of the easiest variant and smoke-train them

   One object, one position, best lighting, then a deliberately short job. You want a loss that moves and a checkpoint on disk, not a policy.

   bash

   lerobot-train \
     --dataset.repo_id=you/smoke_5 \
     --policy.type=act \
     --steps=500 \
     --batch_size=8 \
     --seed=1000 \
     --save_freq=500 \
     --output_dir=outputs/smoke

3. 3
   Record the bulk interleaved, not blocked

   Rotate through variants inside each session instead of all of A then all of B. Position 1, 2, 3, back to 1. This is the step that matters, for the reason below.

4. 4
   Add the hard variants, then go back and record more easy ones

   Ending on the hardest variant leaves your last twenty episodes carrying your most tired teleoperation. The sampler ignores order anyway, so do not let fatigue correlate with difficulty. Close with a few easy repeats.

5. 5
   Hold out a slice before you train for real

   lerobot carves one out: eval_split is the 'fraction of episodes held out per task for offline evaluation (0.0 = disabled)'. Set it before the run, not after seeing the number.

   bash

   lerobot-train \
     --dataset.repo_id=you/bench_task \
     --dataset.eval_split=0.1 \
     --policy.type=smolvla \
     --steps=20000 \
     --seed=1000

6. 6
   Name your bad episodes, do not re-record the set

   If six episodes are unusable, exclude them by index instead of starting over, then rerun with the same seed.

   bash

   lerobot-train \
     --dataset.repo_id=you/bench_task \
     --dataset.exclude_episodes="[7,8,19,31,32,44]" \
     --policy.type=act \
     --steps=100000 \
     --seed=1000

Two experiments that settle it for your dataset

To find out whether balance matters for your task, do not shuffle episode indices and retrain, because that measures seed noise. Measure the two real things: whether one condition is carrying the policy, and whether reweighting sources helps.

# Experiment 1: leave-one-condition-out.
# Same seed, same steps, same policy. Only the episode list changes.
# If dropping one lighting condition costs you 30 points, your dataset
# is not balanced, and no amount of reordering fixes that.

lerobot-train --dataset.repo_id=you/task --policy.type=act \
  --steps=20000 --seed=1000 --output_dir=outputs/all

lerobot-train --dataset.repo_id=you/task --policy.type=act \
  --dataset.exclude_episodes="[40,41,42,43,44,45,46,47,48,49]" \
  --steps=20000 --seed=1000 --output_dir=outputs/no_cond_c

# Experiment 2: mixture weight sweep (GR00T only).
# Split the recordings into two dataset roots first, then vary alpha.
for a in 0.0 0.43 1.0; do
  uv run python gr00t/experiment/launch_finetune.py \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path "/data/bulk:/data/hard" \
    --embodiment-tag NEW_EMBODIMENT \
    --modality-config-path examples/SO100/so100_config.py \
    --ds-weights-alpha $a --max-steps 2000 \
    --output-dir /tmp/ft_alpha_$a
done

Both cost GPU hours rather than recording hours, the cheaper currency. On the 24 GB tier an ACT or SmolVLA run is about 1 to 3 USD, so a three-point sweep is lunch money. On the A100 and H100 tier GR00T and Pi0.5 need, each run is about 4 to 12 USD, still cheaper than a Saturday of re-recording.

Two routes to the same comparison

Where this platform does not help

Curriculum-shaped control is mostly absent from the hosted training path, and some of it is absent upstream too.

• The training form exposes no per-dataset mixture weight. The extra knobs it surfaces are saveSteps for the GR00T models, seed and logFreq for Pi0.5 and SmolVLA, and chunkSize, nActionSteps, seed and logFreq for ACT. For ds_weights_alpha you are on the manual path.
• Gradient accumulation is sent for Pi0.5 (16) and SmolVLA (8) but does not apply, because lerobot 0.5.1 has no such flag. Do not plan around an effective batch size you are not getting.
• There is no staged-training mode: you cannot ask for 5000 steps on set A then 15000 on set B in one job. Chaining two jobs is exactly the sequential fine-tuning LIBERO found to be order-sensitive.
• A LeRobot v3.0 dataset will not load in GR00T at all and must be converted down to v2.1 first. A format wall, not a tuning choice: see dataset rejected as v3.
• Nothing here reorders or re-balances your data. The directory lists datasets and the CLI and MCP server expose the same operations to a terminal and to agents, but the curation judgement is yours.

One more limit, unrelated to curricula but responsible for more failed projects: inference has to sit next to the servos for fast tasks. The control loop runs 20 ms per action step for ACT up to 485 ms for Pi0.5, and public-internet round trips on top turn a working policy into a hesitant one. Remote inference is fine for slow pick-and-place and wrong for fast reactive motion. No training-time ordering repairs that, so start at inference latency if the arm hesitates.

What to do on your next recording session

1. Record one canary episode. Load it. Delete it.
2. Record five easy episodes, run 500 steps of ACT, confirm the loss moves and a checkpoint lands.
3. Record the bulk interleaved across variants, noting which episode indices belong to which condition.
4. Set --dataset.eval_split before the real run, not after.
5. Run leave-one-condition-out with a fixed seed. If one condition carries the policy, record more of the others.
6. Only then, on GR00T with genuinely imbalanced sources, sweep --ds-weights-alpha across 0.0, 0.43 and 1.0.

If you have not recorded anything yet, record your first dataset and train your first policy are the shorter way in. Without an arm, /live has a physical SO-100 streaming with no signup, and the arena compares 85 VLA models with 332 benchmark results. Background on why these models transfer at all is in our VLA overview.