DAgger Explained: Why Behavior Cloning Drifts and What Dataset Aggregation Actually Proves

There is a specific failure that everyone who trains a manipulation policy meets sooner or later. The policy reaches for the cube, gets within two centimetres, hesitates, drifts sideways, then does something unrelated to the task. Validation loss was fine. Open-loop replay against held-out episodes was fine. And yet the arm ends up in a pose that appears nowhere in the training data, and from there it has nothing sensible to say.

That failure has a name and a settled piece of theory behind it. This is the first of four articles on DAgger, and it covers the argument itself: why fitting a policy on the demonstrator's own trajectories produces an error that can grow with the square of the episode length, what dataset aggregation changes, and what the no-regret proof does not promise. The loop on real hardware is covered in running a DAgger loop on an SO-100, the human-gated variant in HG-DAgger and human-gated interventions, and the measurement question in measuring a DAgger loop.

The assumption behavior cloning quietly makes

A demonstration dataset is a pile of observation-action pairs. Behavior cloning fits a function to that pile with ordinary supervised learning and stops there. It is the oldest idea in the field. Pomerleau's ALVINN, in 1988, was a three-layer back-propagation network that took images from a camera and a laser range finder and produced the direction the vehicle should travel; it was trained on simulated road images and followed real roads under some field conditions. The recipe has not changed much; the networks have.

What gets skipped is a check on where those pairs came from. Every one of them lies on a trajectory the demonstrator produced. The policy you deploy produces its own. The moment it deviates, it is being queried about states that were not in the training distribution, and its answer moves it further out. Ross, Gordon and Bagnell open the DAgger paper with exactly this: sequential prediction violates the i.i.d. assumption underneath statistical learning, because the learner's own predictions determine the inputs it sees next.

The clearest illustration in that paper is not a robot at all. Cloning a near-optimal planner for Super Mario Bros. produced a policy that repeatedly got stuck against an obstacle instead of jumping it. The reason is the whole argument in one sentence: the expert always jumped from a comfortable distance, so the dataset contained no state in which Mario was pressed up against an obstacle, and therefore no label for what to do once he was.

Swap Mario for an SO-100 arm and the structure is identical. Your demonstrations show a clean approach and a clean grasp, not the gripper closing two centimetres short - so the policy has no idea what to do from there, and whatever it guesses takes it further out. Covariate shift is a property of the data collection procedure, not of the network architecture.

Where the quadratic term comes from

The 2010 AISTATS paper by Ross and Bagnell, Efficient Reductions for Imitation Learning, makes the compounding precise. Let T be the task horizon, let task cost be bounded in the unit interval, and let epsilon be the surrogate loss measured under the expert's state distribution - the number your validation set reports. Then the extra cost of running that policy for T steps, relative to the expert, is bounded by T squared times epsilon. Ross, Gordon and Bagnell restate this as Theorem 2.1 in the DAgger paper and add the sentence that matters: the bound is tight. Problems exist where a policy with epsilon loss on the expert's distribution really does incur extra cost growing quadratically in T.

Tight does not mean typical. The quadratic term is a worst case over a class of problems, not a prediction about your pick-and-place task. What it establishes is that more expert demonstration cannot remove the problem: it only sharpens the estimate of epsilon on a distribution the policy will not be tested on.

The escape route is in the same paper, restated as Theorem 2.2. If a policy achieves loss epsilon under its own state distribution, and a single wrong action costs at most u in cost-to-go under the expert, the extra cost is bounded by u times T times epsilon - linear in the horizon. The constant u is the interesting quantity: at most 1 for 0-1 disagreement with the expert, and O(1) whenever the expert can recover within a few steps. In the worst case it is O(T), and the linear bound is then no better than the quadratic one.

The two attempts that came before DAgger

Forward training is the honest but impractical answer. Train a separate policy for each timestep, in order, each on the state distribution induced by the policies already fixed for earlier steps, so every policy sees exactly the distribution it will face. The catch is in the description: T policies, trained sequentially, no early stopping. For a manipulation episode at 30 frames per second, T is in the hundreds.

SMILe, from the same paper, and SEARN, from Daume, Langford and Marcu's work on structured prediction, take the other route: one stationary policy, but stochastic. Each iteration trains a component and adds it to a mixture, shifting probability mass away from the expert. The result is a mixture in which some components are worse than others - on a physical arm, a controller that can sample a bad component mid-motion. That is the stated motivation for wanting a stationary deterministic policy instead.

DAgger: one idea, one box

Dataset Aggregation keeps the deterministic policy and moves the fix into data collection. Each round: roll out the current policy, record the states it visits, ask the expert what the correct action would have been in each, add those pairs to the dataset you already have, retrain on the union. The name is the algorithm - you aggregate, you never discard.

D            <- {}                      # the aggregate dataset
pi_hat_1     <- any policy in Pi

for i = 1 .. N:
    pi_i  = beta_i * expert  +  (1 - beta_i) * pi_hat_i
    roll out pi_i for T steps, record every visited state s
    D_i   = { (s, expert(s))  for every visited state s }
    D     = D  union  D_i               # aggregate, do not replace
    pi_hat_{i+1} = train on all of D

return the best pi_hat_i on a validation set

Three details carry more weight than they look like they do. The labels are for states visited by the mixed policy, but the actions come from the expert - the policy provides the questions, the expert the answers. The retraining is on the whole aggregate, which makes each round a Follow-The-Leader step: at round n you pick the best policy in hindsight over every trajectory so far. That framing is what the proof hangs on. And the algorithm ends by returning the best policy in the sequence as chosen on a validation set, because the theorems guarantee that some policy in the sequence is good, not that the last one is.

The beta schedule, and why it is not a tuning knob

The mixed policy is beta_i times the expert plus one minus beta_i times the learner. The point is practical: the first few learned policies are trained on very little data, make many mistakes, and would otherwise spend the rollout in states that become irrelevant once the policy improves.

The theory imposes exactly one condition: the running average of the betas must go to zero. The analysis works with beta_i bounded by (1 - alpha) to the power i-1, for a constant alpha independent of T.

The gap between 2980 and 3030 on a scale running to roughly 4300 is small, but the paper's explanation of it is the most useful practical note in the section. With the parameter-free schedule, Mario got stuck in the same spot early and generated a mass of near-duplicate data from that one location; letting the expert drive a fraction of the time both unstuck him and widened the variety of states. The schedule is less about the mixing ratio than about whether your data collection keeps producing new states or the same failure.

The guarantee: a reduction to no-regret online learning

Here is the move that makes the paper what it is. Treat each DAgger round as one example in an online learning problem, where the loss at round i is the surrogate loss under the state distribution of the policy used at round i. The learner commits to a policy before seeing that loss, and the sequence is non-stationary because it depends on the policies produced so far.

An algorithm is no-regret if its average loss over N rounds approaches that of the best single policy in hindsight. Follow-The-Leader on strongly convex losses is such an algorithm, with average regret shrinking on the order of 1/N - and retraining on the full aggregate is precisely Follow-The-Leader. Any other no-regret learner would serve as well: the analysis is a reduction, not a property of one optimiser.

One lemma bridges the gap between the mixed policy that collected the data and the learned policy that will be deployed: Lemma 4.1 bounds the L1 distance between their state distributions by 2 T beta_i. This is why the betas must decay - while the expert still holds appreciable control authority, the states you collect are not the states your policy will produce. Combine the lemma with the regret bound and the main result follows: after roughly T iterations, some policy in the sequence has surrogate loss under its own distribution within O(1/T) of epsilon_N. Feed that into the linear bound and you land at Theorem 3.2.

The empirical side is modest by current standards. In Super Tux Kart the supervised baseline did not improve its average falls per lap as more data arrived, DAgger reached a policy that never fell off the track after fifteen iterations, and SMILe after twenty still fell roughly twice per lap. On the handwriting benchmark, character accuracy ran 82 percent without structure, 83.6 percent supervised, 85.5 percent with DAgger. None of these is a manipulation result.

What the proof does not promise

The theorem statements are conditional, and the conditions are load-bearing.

One further result is often quoted as a refutation and is not one. Rajaraman, Yang, Jiao and Ramachandran study the minimax limits of imitation learning in episodic MDPs with a finite state space S and horizon H, and prove a suboptimality lower bound on the order of |S| H squared over N that holds even when the learner may actively query the expert at visited states. That is a worst-case rate over a class of MDPs at a fixed episode budget, and what it rules out is the idea that interaction improves the minimax rate; DAgger's theorem is a different statement, bounding the deployed policy relative to what its own policy class can achieve.

Swamy, Choudhury, Bagnell and Wu later classified these algorithms by which moments of the expert's behaviour they match, and introduced a notion of moment recoverability that delineates how well each family mitigates compounding error. The surveys by Osa and by Celemin cover the algorithmic landscape and the human-feedback interfaces.

The bill: labelling states the expert never produced

Everything above assumes an expert that can be queried anywhere. In simulation with a planner that is nearly free - the Mario experiments used a near-optimal planner with full access to game state. With a human on a robot it is the dominant cost, and a peculiar one: the human has to produce a correct action in a configuration their own competence would never have created.

Kelly, Sidrane, Driggs-Campbell and Kochenderfer state the objection directly in the HG-DAgger paper. Vanilla DAgger requires the expert to supply action labels while not being fully in control of the system. This reduces safety, and with human experts it is likely to degrade the quality of the collected labels, which they put down to perceived actuator lag. The label you get back is not the label the algorithm assumed.

Laskey and colleagues attack the problem from the other side with DART, and their framing is blunt: on-policy techniques are tedious for human supervisors, add computational burden, and may visit dangerous states during training. Their alternative injects calibrated noise into the supervisor's own demonstrations, so recovery gets demonstrated without the robot ever running an untrusted policy. On MuJoCo Humanoid they report DART decreasing the supervisor's cumulative reward by 5 percent during training, while DAgger executes policies with 80 percent less cumulative reward than the supervisor; on grasping in clutter with a Toyota HSR, an average 62 percent increase over behavior cloning.

Zhang and Cho's SafeDAgger treats queries to the reference policy as the scarce resource: a separate safety policy predicts, without querying, whether the primary policy is about to deviate from the reference beyond a threshold, and only those states are handed over. All three react to the same fact - the DAgger analysis charges nothing for expert labels, and reality charges a great deal.

What this means for an SO-100 on your desk

Translate the horizon into your own units. A twenty-second episode at 30 frames per second is 600 decision steps, and T in every bound above is that number. At T = 600, the difference between a term scaling with T and one scaling with T squared is the difference between a policy that recovers from a bad approach and one that does not.

This is part of why action chunking helps: when a policy emits a short sequence of actions per inference step, the number of decision points drops, and so does the number of chances to compound. Zhao, Kumar, Levine and Finn name compounding error as the motivation for Action Chunking with Transformers, and report 80 to 90 percent success on six difficult real-world tasks, on low-cost bimanual hardware, from ten minutes worth of demonstrations. Chunking does not remove covariate shift - the states are still the policy's own - but it shortens the effective horizon. See action chunking and the SO-100 imitation learning guide.

The second translation is the progress metric. You cannot measure epsilon under the policy's own distribution directly - that needs ground-truth expert actions for every visited state, the thing you are trying to avoid producing. What a human-gated loop gives you instead is the intervention rate: the fraction of frames in a run during which the human had taken over. It is a proxy, and it moves for reasons unrelated to the policy - a patient operator intervenes less. Used consistently, it is the one number that says whether a round was worth the afternoon.

A third translation is a data-quality warning the analysis does not cover. Mandlekar and colleagues studied six offline learning algorithms on five simulated and three real-world multi-stage manipulation tasks, and report a sensitivity to algorithmic design choices, a dependence on the quality of the demonstrations, and variability caused by the stopping criterion. Belkhale, Cui and Sadigh argue that dataset quality should be formalised through action divergence and transition diversity, and note that state diversity is not always beneficial. A DAgger round adds states nobody chose deliberately: some are the recovery data you need, some are the robot flailing while you fumble for the takeover control.

Mechanically a round is six steps: run inference with recording on, take over when the policy misbehaves, review the run and file each episode, sync the corrections, compose a mixed dataset from originals plus corrections with episode selection made explicitly per source, and continue training from the previous checkpoint rather than the base model. On ay-robots those steps exist as buttons, which removes the plumbing but not the judgement. Two caveats: continuing from a checkpoint initialises weights and is not an optimizer resume, and the leader-arm alignment move is still lightly tested on hardware. See training and datasets.

The family tree, in one table

Frequently asked questions

Where to go from here

If you have not trained a policy yet, this theory is premature: record a dataset first, starting from training your first policy and the desktop client. If you are weighing another hundred clean demonstrations against starting corrections: clean demonstrations do not fix a distribution problem. For the mechanics, continue with the human-gated variant and then the SO-100 walkthrough.