Long-Horizon Tasks and Subtask Decomposition

What actually breaks at step seven

A policy trained on 50 clean pick-and-place episodes will pick and place. Train the same architecture on a task with ten stages, open a drawer, take out a cup, set it down, close the drawer, pour, stir, and so on, and it will usually get three or four stages in and then stall, or repeat a stage it already completed, or reach for something that is no longer where the demonstrations left it. The training loss looks fine. The open-loop replay looks fine. The robot still does not finish.

This is not a mysterious failure. It is arithmetic, and it was measured carefully in the original ACT paper (Zhao, Kumar, Levine and Finn, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, arXiv 2304.13705). That paper reports success per stage as well as per task, across eight tasks in its Tables I and II. The per-stage columns are the most useful table in low-cost manipulation, because they show exactly where the run dies.

Read the rows sideways. Slot Battery barely degrades, because every stage is easy and the object barely moves. Thread Velcro falls from 92 to 20 across three stages, and the paper's own error analysis blames a mid-air regrasp and a low-contrast object rather than the horizon. Prep Tape loses roughly a third of its runs over four stages even though no single stage is below 64 percent. ACT collected 50 demonstrations per task, 100 for Thread Velcro, at 50 Hz, with episodes of 8 to 14 seconds, which the paper notes is 400 to 700 timesteps. A genuinely ten-step household task is several times that.

Why the horizon hurts more than linearly

Imitation learning has a known theoretical problem here. Ross, Gordon and Bagnell (arXiv 1011.0686, the DAgger paper) show that a policy trained to imitate an expert under the expert's own state distribution carries a cost bound of J(pi) <= J(pi*) + T^2 * epsilon, where T is the horizon and epsilon is the per-state loss against the expert. The paper says in as many words that this behaves as Theta(T^2 * epsilon) for small epsilon, notes that the bound is tight, and constructs a case in which the extra cost really is quadratic.

The mechanism is distribution shift. A small action error moves the arm to a state slightly off the demonstrated manifold. The next observation is therefore slightly out of distribution, so the next action is a bit worse, and so on. The ACT paper names this the compounding error problem and adds that fine manipulation makes it worse, because small differences in the predicted action can incur large differences in the state when tolerances are tight.

Action chunking buys a constant factor

Action chunking is the standard mitigation, and it is real. Instead of modelling pi(a_t | s_t), the policy models pi(a_t..t+k | s_t) and executes k actions per observation. The ACT paper states the effect plainly: this reduces the effective horizon of the task by k-fold. Every model you can train on this platform predicts chunks rather than single steps.

Do the arithmetic on your own task. LeRobot records at 30 fps by default, and its own docs call a 30 fps dataset the default when explaining how to scale frame-based settings. A 60 second sequence is therefore 1800 frames. With ACT at nActionSteps 100 that is 18 decision points. With Pi0.5 at n_action_steps 50 it is 36. Compare that with a 10 second pick and place, which is 300 frames and three to six decision points. Chunking took your horizon from 1800 to 18. It did not take it to 3. That arithmetic is mine, from published defaults, not a platform capability.

Benchmarks agree: long suites score lowest

The LIBERO benchmark (Liu, Zhu, Gao and colleagues, arXiv 2306.03310) is useful here because it isolates the variable. It defines four task suites, 130 tasks in total, and splits its LIBERO-100 set into 90 short-horizon tasks (LIBERO-90) and 10 long-horizon ones (LIBERO-LONG, which NVIDIA labels LIBERO-10). Same simulator, same robot, same demonstration pipeline. Between the short suites and the long one, mainly the horizon changes.

Those are NVIDIA's own numbers from the Isaac-GR00T LIBERO example, and all four suites were finetuned with the same recipe: 20000 max steps, global batch size 640, gradient accumulation 1, state dropout probability 0.2, on eight GPUs. The long suite comes last by a consistent margin with everything else held fixed. One caveat worth carrying: the README's percentage column does not match its own fractions exactly, so the fractions above are what to quote. And note the size of the gap. Eleven failures out of 200 in simulation with dense demonstrations is nothing like the ACT table above, where a four-stage real-hardware task with 50 episodes lost a third of its runs.

Route one: decompose the task

The first way out is to stop asking one network to hold the whole plan. Split the problem: a high-level component decides what the next subtask is, and a low-level vision-language-action model executes that one subtask. The high-level output is a short language string, which is exactly what a language-conditioned policy already accepts.

What the published results say

Two details in that table are worth more than the headline percentages. First, SayCan's long-horizon family drops to 47 percent execution while its overall average is 74. A planner does not make long tasks as easy as short ones; it makes them possible at all. Second, read Hi Robot's GPT-4o column carefully rather than as a punchline. On instruction accuracy the zero-shot GPT-4o planner scored 30 against the flat VLA's 36, so by that measure an unadapted general model was worse than no hierarchy. On task progress the same GPT-4o scored 64 against the flat baseline's 44. It was not useless, it was badly aligned with what the user actually asked for, and Hi Robot's authors had already constrained it to choose among the most common skill labels. Pi0.5's ablations put a zero-shot GPT-4 planner last of every variant they tried.

How Pi0.5 does it in one model

The Pi0.5 paper factorises the policy as pi(a, l_hat | o, l) = pi(a | o, l_hat) * pi(l_hat | o, l). The overall prompt l is something like "put away the dishes". The model first samples a subtask string l_hat such as "pick up the plate", then predicts the action chunk conditioned on l_hat. Note what is missing, and the paper says it explicitly: the action distribution does not depend on l at all, only on l_hat. The low-level half never sees the overall goal. Its whole world is one short instruction and the current image, which is why it transfers between tasks.

The action expert predicts a chunk of H = 49 actions, and the paper describes commanding arm actions and base velocities at 50 Hz, so roughly one second of motion per call. Pi0.5 was trained on about 400 hours of mobile manipulation data plus web data, cross-embodiment data and explicit subtask-prediction data. Its scaling study varied that mobile manipulation data over 3, 12, 22, 53, 82 and 104 distinct locations and reports performance improving steadily as locations are added; at 104 locations the model matched a control trained directly on the test homes it was then evaluated in. The paper does not report a plateau, so do not plan around one.

Building the decomposition yourself

The good news is that the pieces are all real CLI commands today. The plan is: record one dataset per subtask, relabel and merge them into a single multi-task LeRobot dataset, fine-tune a language-conditioned policy on the merged set, then drive it from your own loop that swaps the instruction string between stages. Commands below are from the lerobot main branch as read on 24 August 2026.

1. 1
   Record each subtask as its own dataset

   Do not record ten-step sequences yet. Record stage 1 fifty times, stage 2 fifty times, and so on, each with its own instruction string. Fifty is the platform minimum for Pi0.5, GR00T and ACT; SmolVLA's floor is 30. Start each recording from the realistic state that the previous stage leaves behind, including the messy ones. See record your first dataset for the setup.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/tty.usbmodem58760431541 \
     --robot.id=black \
     --robot.cameras="{laptop: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
     --teleop.type=so100_leader \
     --teleop.port=/dev/tty.usbmodem58760431551 \
     --teleop.id=blue \
     --dataset.repo_id=$HF_USER/kitchen_s1_open_drawer \
     --dataset.num_episodes=50 \
     --dataset.single_task="open the drawer" \
     --dataset.streaming_encoding=true

2. 2
   Fix any labels you got wrong

   --dataset.single_task writes one string for the whole dataset, so if you recorded mixed stages in one session you have to relabel per episode afterwards. The script's own help marks this operation as modifying the dataset in place, so copy it first.

   bash

   lerobot-edit-dataset \
     --repo_id $HF_USER/kitchen_mixed \
     --operation.type modify_tasks \
     --operation.episode_tasks '{"0": "open the drawer", "1": "pick up the cup", "2": "open the drawer"}'

3. 3
   Merge the per-subtask datasets into one

   One policy, many instructions. The merge keeps each episode's task string, so the result is a multi-task dataset where the language is the only thing distinguishing a stage-1 episode from a stage-7 episode.

   bash

   lerobot-edit-dataset \
     --new_repo_id $HF_USER/kitchen_subtasks \
     --operation.type merge \
     --operation.repo_ids "['$HF_USER/kitchen_s1_open_drawer', '$HF_USER/kitchen_s2_pick_cup', '$HF_USER/kitchen_s3_close_drawer']"

4. 4
   Fine-tune a language-conditioned policy

   ACT will not work for this step, see the warning below. Use Pi0.5, SmolVLA or GR00T. The command below follows the lerobot Pi0.5 training example, sized for a single 80 GB card; lerobot's default seed is 1000, while GR00T's tyro-based finetune entry point exposes no seed at all.

   bash

   lerobot-train \
     --dataset.repo_id=$HF_USER/kitchen_subtasks \
     --policy.type=pi05 \
     --policy.pretrained_path=lerobot/pi05_base \
     --policy.gradient_checkpointing=true \
     --policy.dtype=bfloat16 \
     --policy.device=cuda \
     --policy.push_to_hub=false \
     --output_dir=./outputs/kitchen_subtasks \
     --job_name=kitchen_subtasks \
     --batch_size=64 \
     --steps=30000 \
     --save_freq=5000 \
     --seed=1000

5. 5
   Serve it, then drive the instruction from your own loop

   The async client attaches the task string to every observation it sends, so you can change it mid-run. The CLI below runs one stage; the next section replaces it with a planner. Note that the docs mark --task as not necessarily defined for all policies, such as act.

   bash

   # terminal 1: policy server
   python -m lerobot.async_inference.policy_server \
     --host=127.0.0.1 \
     --port=8080
   
   # terminal 2: robot client, one stage at a time
   python -m lerobot.async_inference.robot_client \
     --server_address=127.0.0.1:8080 \
     --robot.type=so100_follower \
     --robot.port=/dev/tty.usbmodem585A0076841 \
     --robot.id=follower_so100 \
     --robot.cameras="{laptop: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
     --task="open the drawer" \
     --policy_type=pi05 \
     --pretrained_name_or_path=$HF_USER/kitchen_subtasks \
     --policy_device=cuda \
     --actions_per_chunk=50 \
     --chunk_size_threshold=0.5

For a real planner you drive the client from Python rather than the CLI, because the CLI takes one fixed --task string for the whole run. The hook you want is control_loop_observation(task), which writes the task string into the observation it ships. The trap is that control_loop(task) is not that hook: in robot_client.py it runs while self.running and only returns after stop(), so calling it once per stage leaves your planner stuck on the first stage forever. The planner has to own the loop instead, and it has to release the start barrier that the action-receiver thread is waiting on.

import threading
import time

from lerobot.async_inference.robot_client import RobotClient

SUBTASKS = [
    "open the drawer",
    "pick up the cup",
    "place the cup on the tray",
    "close the drawer",
]
STAGE_TIMEOUT_S = 45.0

client = RobotClient(client_cfg)
if not client.start():
    raise SystemExit("handshake with the policy server failed")

receiver = threading.Thread(target=client.receive_actions, daemon=True)
receiver.start()

# Both threads wait on the same barrier; control_loop() would do this for us,
# but it never returns, so the planner does it and runs the loop itself.
client.start_barrier.wait()

try:
    for subtask in SUBTASKS:
        deadline = time.perf_counter() + STAGE_TIMEOUT_S
        while client.running and not done_with_stage(subtask):
            t0 = time.perf_counter()
            if client.actions_available():
                client.control_loop_action()
            if client._ready_to_send_observation():
                # the subtask string rides along with this observation
                client.control_loop_observation(subtask)
            time.sleep(max(0.0, client.config.environment_dt - (time.perf_counter() - t0)))
            if time.perf_counter() > deadline:
                stop_and_ask_for_help(subtask)
                raise SystemExit(f"stage timed out: {subtask}")
finally:
    client.stop()
    receiver.join()

For GR00T N1.7 the equivalent runtime hook is the observation dictionary. The Isaac-GR00T policy guide shows the language field as {"language": {"task": [["pick up the cube"]]}}, a list of lists of strings passed to a policy server started with run_gr00t_server.py. Different repo, same pattern as lerobot: the subtask is data you send at inference time, not something baked into the checkpoint. That is what makes a planner possible at all, and it is worth checking on any model before you design around decomposition.

Route two: longer demonstrations and memory

The other way out is to keep the flat policy and give it what it is missing, which is more demonstrations of the full sequence and some way to remember what it already did. This is the direction the frontier labs have taken most recently, and the honest summary is that it works and that it is expensive.

MEM (Torne, Pertsch, Walke and colleagues, arXiv 2603.03596, March 2026) is the clearest recent statement of it. It equips a VLA, in the paper's case a Pi0.6-class model, with two memories: a short-horizon video memory compressed by a dedicated video encoder, and a long-horizon language memory in which a high-level policy keeps a running natural-language summary of what has already happened. The paper reports policies solving tasks that require up to fifteen minutes of memory, such as cleaning a kitchen or working through a recipe, and reports gains even on short tasks: plus 11 percent on picking up a chopstick, plus 62 percent on opening a fridge.

There is also a latency cost that is easy to overlook. MEM's own inference-time figure draws a 300 ms real-time barrier and shows that naively feeding a sequence of past frames into a VLA backbone crosses it as the frame count rises; the paper's whole video-encoder design exists to avoid that. On this platform Pi0.5 already sits at 485 ms per action step before you add anything, which is why the inference latency column deserves a look before you commit to memory as a strategy.

Which route for which task

The minimum episode counts on the policies page are 50 for GR00T N1.7, GR00T N1.5, Pi0.5 and ACT, and 30 for SmolVLA. Those floors describe one task. A merged multi-subtask dataset only helps if each subtask clears the floor on its own, so decomposing a ten-stage job into six subtasks means six times the floor before the merged set is worth a training run. That is the real cost of route one, and it is paid in teleoperation hours, not GPU hours.

The same job, two ways

The cost column is why route one is less frightening than it sounds. Six subtask policies is one merged dataset and one run, not six runs, as long as the policy reads language. Six ACT checkpoints really is six runs, which at the 4090 tier is roughly 6 to 18 USD. Neither number is the bottleneck. The teleoperation time is, and no amount of GPU budget shortens it.

Honest limits

Decomposition moves the failure rather than removing it. Pi0.5's own discussion section lists the case where high-level subtask inference is easily distracted and, for example, closes and opens a drawer repeatedly while putting items away. SayCan's error analysis found 65 percent of failures were LLM failures and 35 percent were affordance failures, that is, the value function misjudging whether a skill could succeed rather than the manipulation policy fumbling it. If your planner is wrong, a perfect low-level policy executes the wrong stage flawlessly.

The second limit is inference latency. The control loop on this platform runs at 20 to 485 ms per action step depending on the model, and adding public-internet round trips to that turns a working policy into a hesitant one. Remote inference is viable for slow pick and place and for the deliberate pace of most multi-stage household tasks. It is not viable for fast reactive motion. A ten-stage task that is slow throughout is a reasonable candidate for remote inference; a ten-stage task with one fast catching motion in the middle is not.

The third limit is diagnostic. A policy that stalls at stage four looks identical to a policy that has hit a hardware limit or a camera problem. Before you conclude that your horizon is too long, rule out the cheap explanations on the failure-mode pages, in particular policy freezes mid-motion and policy only works in one setup. Horizon problems degrade gradually across stages and vary run to run; hardware problems stop dead in the same place every time.

What to do this week

1. Write down your stages and score each one separately from your existing rollouts. If you cannot score them separately, you cannot debug them separately.
2. Multiply the per-stage rates. If the product matches your observed end-to-end rate, you have a horizon problem and not a capability problem.
3. Find the worst stage. Often one stage carries the whole loss, as in ACT's Thread Velcro dropping from 92 to 40 between stage one and stage two. Record 30 more episodes of that stage alone before touching anything else.
4. Only then decide between routes, using the table above.
5. Check the public dataset directory for a dataset with a similar stage structure before recording your own; the data collection guide covers what makes an episode reusable, and the VLA overview covers why language conditioning is the hinge this whole article turns on.

If you have no arm yet, the /live page streams a physical SO-100 you can drive with no signup, and train your first policy walks the single-stage version of everything above. There is also the flow-matching background on the Pi line of models if the pi05 architecture above went past too quickly. Get one stage to 95 percent before you plan ten of them.