Flow Matching vs Diffusion for Robot Policies

The difference is one line in the loss

Diffusion and flow matching get described as rival paradigms. In an action head they are close to the same code. Both take an observation, a noisy action chunk and a scalar that says how noisy that chunk is, and both regress one vector per action dimension. The architecture is interchangeable: a U-Net, a diffusion transformer or a 300 M transformer expert all work under either objective. What actually differs is what that regressed vector is supposed to be, and how many forward passes you need before you have an action you can send to a servo.

Diffusion asks the network for the noise that was added. Flow matching asks it for a velocity: given a point on a straight line between noise and data, which way is data. Everything downstream follows from that choice, including the time variable, the sampler, the step count, and how hard the whole thing is on your control loop. This article walks the objective, the sampler and the numbers that ship in lerobot today, then looks at what it costs when you actually run a policy on an arm. If you want the wider family tree first, the VLA overview covers it.

Diffusion: predict the noise, then subtract it a hundred times

The denoising diffusion probabilistic model recipe (arXiv 2006.11239) corrupts data over T discrete timesteps according to a variance schedule, trains a network to predict the noise at each timestep, then walks the chain backwards. Diffusion Policy (arXiv 2303.04137, first posted 7 March 2023, revised to v5 on 14 March 2024) put that machinery behind a visuomotor imitation learning head and reported "an average improvement of 46.9%". Check which version you are quoting: the v5 PDF says 15 tasks across 4 manipulation benchmarks, while the abstract on the arXiv listing page still says 12. It is in lerobot as policy type diffusion, and it is not a vision-language-action model: no language conditioning, no web pretraining, no checkpoint to start fine-tuning from. You train it from scratch on your own data, the same way you train ACT.

n_obs_steps: int = 2
horizon: int = 64
n_action_steps: int = 32

noise_scheduler_type: str = "DDPM"
num_train_timesteps: int = 100
beta_schedule: str = "squaredcos_cap_v2"
beta_start: float = 0.0001
beta_end: float = 0.02
prediction_type: str = "epsilon"
clip_sample: bool = True
clip_sample_range: float = 1.0

# None means: fall back to num_train_timesteps
num_inference_steps: int | None = None

That last line is the one that costs you. num_inference_steps is None out of the box, and modeling_diffusion.py resolves None to num_train_timesteps, so an untouched lerobot Diffusion Policy runs 100 U-Net calls to produce one action chunk. The camera encoders are not in that loop: _prepare_global_conditioning runs them once and the result is passed into every iteration as global_cond, which is the same trick Pi0 uses at a larger scale. The paper says so in as many words, that the policy "extracts the visual representation once regardless of the denoising iterations". So the 100 steps multiply the U-Net, not the whole model. That is still 100 transformer-sized forward passes per chunk.

if config.num_inference_steps is None:
    self.num_inference_steps = self.noise_scheduler.config.num_train_timesteps
else:
    self.num_inference_steps = config.num_inference_steps

# _prepare_global_conditioning() encodes the images ONCE, outside the loop.
# conditional_sample() then reuses that vector on every step:
self.noise_scheduler.set_timesteps(self.num_inference_steps)
for t in self.noise_scheduler.timesteps:
    model_output = self.unet(sample, t, global_cond=global_cond)
    sample = self.noise_scheduler.step(model_output, t, sample, generator=generator).prev_sample

Flow matching: predict a velocity, then walk ten steps

Flow matching comes from Lipman et al., "Flow Matching for Generative Modeling" (arXiv 2210.02747, posted 6 October 2022). The idea is to regress a vector field directly instead of simulating an ODE during training, and to be free about which probability path you use. The paper argues that optimal transport paths are "more efficient than diffusion paths, provide faster training and sampling, and result in better generalization". Rectified flow (arXiv 2209.03003, Liu, Gong and Liu, 7 September 2022) reaches the same place from the straight-line side and reports "high quality results even with a single Euler discretization step". If you want the full derivation with code, the Flow Matching Guide and Code (arXiv 2412.06264, 9 December 2024) is the reference.

In Pi0 the path is literally a straight line. The Pi0 paper (arXiv 2410.24164, 31 October 2024) writes the interpolation as A_tau = tau * A + (1 - tau) * eps with eps drawn from a standard normal, and the target as u(A_tau | A) = A - eps, integrating from tau = 0 (noise) to tau = 1 (actions). That is the entire forward process. No alpha_bar, no beta schedule, no variance term to get wrong. Pi0.5 (arXiv 2504.16054, 22 April 2025) uses the same interpolation, calls tau "the flow matching time index", and then flips the sign: it trains the model "to predict the flow vector field" omega - a, noise minus action. Same straight line, mirrored axis.

dt = -1.0 / num_steps
x_t = noise
for step in range(num_steps):
    time = 1.0 + step * dt
    time_tensor = torch.tensor(time, dtype=torch.float32, device=device).expand(bsize)
    v_t = denoise_fn(x_t, time_tensor)
    x_t = x_t + dt * v_t
return x_t

That is the sampler, minus the real-time-chunking hook that wraps the velocity call. Eight lines, no scheduler object, no state. With num_steps = 10 the step size is 0.1 and the loop walks from time 1.0, which holds pure noise, down to time 0.0, which holds the action chunk. lerobot's docstring is blunt about the provenance: it calls this "the openpi sampling loop" and notes that pi0, pi05, smolvla and eo1 each used to carry their own copy. The training side of modeling_pi05.py matches it exactly, with x_t = time * noise + (1 - time) * actions and u_t = noise - actions.

Step counts across the five policies you can train here

Four of the five trainable policies on this platform use flow matching. That is not obvious from the labels, because the GR00T N1.7 model card names the objective and the architecture in the same breath. It says the model "uses a flow matching action transformer to model a chunk of actions conditioned on vision, language and proprioception", and then that "the flow matching transformer is implemented as a diffusion transformer (DiT), in which the diffusion step conditioning is implemented using adaptive layernorm (AdaLN)". DiT is the architecture. Flow matching is the objective. Reading "DiT" as "diffusion objective" is the mistake, and it is an easy one to make when the card's own timing table calls the integration steps "denoising steps". GR00T N1.7 and GR00T N1.5 use the same wording.

Read that table twice, because it breaks the obvious story. If sampler steps drove inference latency, Pi0.5 at 10 steps would sit close to GR00T N1.7 at 4 steps, and both would be far behind a 100-step Diffusion Policy. Instead Pi0.5 is three times slower than GR00T N1.7 and roughly twenty-four times slower than ACT, which does no iterative sampling at all. The step count is a multiplier on the per-step cost, and the per-step cost is not the same across these models. The head-to-head page has the rest of the comparison, and Pi0.5 against SmolVLA is the more useful one if you are choosing between two flow matching models.

Where the time actually goes

Pi0 was designed so the sampler is the cheap part. The paper is explicit: "Note that inference can be implemented efficiently by caching the attention keys and values for the prefix o_t and only recomputing the suffix corresponding to the action tokens for each integration step." The images and the language prompt go through the PaliGemma backbone once. The 10 Euler steps then only touch the action expert, which was sized for that job: "To speed up inference (which requires multiple forward passes of the action expert), we downsize the action expert to {width=1024, mlp_dim=4096}", giving the 300 M figure both Pi0 and Pi0.5 quote. Whether the design goal is actually met is a measurement, and there are exactly two published ones.

The first two rows disagree about where the money goes, and both come from the vendor. Pi0's prefix costs nearly twice its sampler on a 4090. GR00T N1.7's action head costs more than its backbone on an H100, at four steps, and it stays the larger half after TensorRT. So "the backbone dominates" is a Pi0 statement, not a law. The one thing that generalises is the third row: same model, same 4 steps, same card, 27.9 ms against 85.8 ms depending only on the serving stack. Before you attribute a latency difference to an objective, check that both numbers came off the same pipeline.

1. Encode the camera frames and the prompt through the 3 B backbone. Once per chunk.
2. Cache its attention keys and values as a prefix.
3. Run the 300 M action expert against that cached prefix, 10 times, changing only the noisy action chunk and the time scalar.
4. Emit 50 continuous actions, denormalise them, send them to the arm.

What the latency actually buys

The trade Pi0 and Pi0.5 both make is chunk size against call frequency. Both use an action horizon of 50, and the Pi0.5 appendix pins it down: the action expert "takes in a sequence of noisy action tokens for an action horizon of 50, i.e. H = 49". Physical Intelligence's own writeup calls that a "50-step (1-second)" action chunk, which puts the control rate at 50 Hz. One generation pass yields a full second of motion, all of it continuous, all of it produced jointly rather than one step conditioned on the last. That is what action chunking is for, and it is why a 485 ms generation can still drive a smooth arm: you are not paying it per servo command. Pi0 reports running inference every 0.5 seconds, after executing 25 of the 50 actions, on its 50 Hz robots.

The alternative is autoregressive action tokens, and Pi0 is direct about why that does not work at this frequency. Flow matching, it says, "allows us to handle high-frequency action chunks (up to 50 Hz) and highly dexterous tasks, which we show pose a major challenge for prior autoregressive VLAs", and notes that "OpenVLA struggles on these tasks because its autoregressive discretization architecture does not support action chunks". FAST (arXiv 2501.09747, 16 January 2025) made discrete tokens competitive by compressing action sequences with a discrete cosine transform. Pi0.5 trains against both representations, the FAST tokens and the flow matching expert, and uses only the expert at inference, because discrete representations "are less well-suited for real-time inference, because they require expensive autoregressive decoding".

Two step counts that get confused

"Steps" means two different things in this literature and mixing them up produces nonsense comparisons. Integration steps are how many times the sampler evaluates the velocity field to produce one chunk. Execution steps are how many actions of that chunk you run open-loop before you look at the cameras again. The SmolVLA paper ablates the second one on LIBERO, and the effect is much larger than anything the objective does.

Running the full 50-action chunk open-loop costs 31 points of average success against re-observing every 10. That is the SmolVLA paper's Table 13, and it is a bigger lever than the training objective, the sampler, or the step count. On this platform SmolVLA and Pi0.5 ship with n_action_steps equal to the full chunk size of 50, which is the bottom row of that ablation. lerobot's own LIBERO example overrides it to 10. Treat the shipped default as a thing to measure, not a recommendation.

The timestep distribution nobody mentions

Standard flow matching samples the training time uniformly. Pi0 does not, and Pi0.5 inherits the deviation. The Pi0.5 appendix is the clearer of the two: "we deviate from standard uniform sampling" and instead use p(tau) = Beta((s - tau) / s; alpha = 1.5, beta = 1), with timesteps above s excluded, "We use s = 0.999 in our experiments, which accommodates up to 1,000 integration steps". Pi0's Appendix B gives the same distribution and the same s. The shape puts more training mass on low tau, meaning the noisy end of the path, where the vector field is hardest to get right and where a coarse 10-step integration takes its first and largest jumps.

def sample_time_beta(bsize, device, *, alpha, beta, scale, offset):
    """Beta-distributed flow-matching timesteps: Beta(alpha, beta) * scale + offset."""
    time_beta = sample_beta(alpha, beta, bsize, device)
    return (time_beta * scale + offset).to(dtype=torch.float32, device=device)

# PI05Config defaults
num_inference_steps: int = 10
time_sampling_beta_alpha: float = 1.5
time_sampling_beta_beta: float = 1.0
time_sampling_scale: float = 0.999
time_sampling_offset: float = 0.001
chunk_size: int = 50
n_action_steps: int = 50
optimizer_lr: float = 2.5e-5

Two things worth noticing there. First, this is a genuine hyperparameter that almost never appears in a training log, so if you compare a flow matching run against a diffusion run you may be comparing timestep distributions rather than objectives. Second, lerobot's own preset learning rate for Pi0.5 is 2.5e-5, while the AY-Robots trainer sends 5e-5 with a batch size of 1 and 30000 max training steps. Neither is wrong, but they are different starting points, and you should know which one produced a curve before you read anything into it. The training docs list what the form actually sends.

Does the objective actually win?

Usually, not always, and by less than the framing suggests. The cleanest published ablation is in the SmolVLA paper (arXiv 2506.01844, 2 June 2025), which trains the same action expert two ways, once with flow matching and once with a plain L1 regression loss on the predicted chunk, and evaluates on LIBERO. Table 10 of that paper:

Five points of average, and the paper reads it as evidence that flow matching "provides better inductive bias for modeling complex, multimodal action distributions". But look at where the five points come from. Regression wins on Spatial and on Goal. The entire margin sits in LIBERO-10, the long-horizon suite, where the gap is 53 against 38. That is the shape you would expect if the benefit is about not averaging two valid strategies into one invalid one, which only bites when the task has genuine branch points. On short, single-mode tasks a regression head is competitive and much cheaper. ACT is a conditional VAE, trained with a reconstruction loss and a KL term at kl_weight 10.0, with no iterative sampling at all, and it is the fastest thing on this platform for exactly that reason.

For cross-model numbers rather than ablations, the arena lists 85 VLA models with 332 benchmark results and links every value to the paper or model card it came from, including Pi0.5, GR00T N1.5 and the transformer variant of Diffusion Policy. Benchmark numbers across papers are not directly comparable, which is the reason each one carries its source.

Running the comparison

Both paths below fine-tune Pi0.5 on your own LeRobot dataset. The manual path is what the lerobot docs give you as of 23 August 2026, against lerobot 0.6.1, the current release on PyPI. The platform path is the same job with the GPU rental and the checkpoint storage handled.

pip install -e ".[pi]"        # from a lerobot checkout
# or, from PyPI:
pip install 'lerobot[pi]'

hf auth login

lerobot-edit-dataset \
    --repo_id your_dataset \
    --new_repo_id your_dataset \
    --operation.type recompute_stats \
    --operation.overwrite true

lerobot-train \
    --dataset.repo_id=your_dataset \
    --policy.type=pi05 \
    --policy.pretrained_path=lerobot/pi05_base \
    --policy.gradient_checkpointing=true \
    --policy.dtype=bfloat16 \
    --policy.device=cuda \
    --policy.freeze_vision_encoder=false \
    --policy.train_expert_only=false \
    --output_dir=./outputs/pi05_mine \
    --job_name=pi05_mine \
    --batch_size=64 \
    --steps=30000 \
    --save_freq=5000 \
    --seed=1000

An A/B you can actually run this week

The useful experiment is not diffusion against flow matching in the abstract. It is: does the iterative sampler earn its latency on your task. Train the cheapest flow matching model and the cheapest deterministic one on one recording and drive both.

1. 1
   Record one dataset, use it twice

   Both policies need the same demonstrations. 50 episodes is the platform minimum for Pi0.5 and ACT, 30 for SmolVLA. Record with the desktop client straight from a teleop session so the joint states and camera streams line up: record your first dataset, the client download. Or skip recording and take something from the public dataset directory, which is the cheaper way to run this comparison first.

2. 2
   Train the deterministic baseline first

   ACT is about 80 M parameters, has no base model at all (it only exists after training on your task), and the platform default is 100000 steps at a cost of 1 to 3 USD on the 24 GB tier. It is your floor. If ACT solves the task, no amount of flow matching will make it more solved, and you will have spent 20 ms per action step instead of 485.

   bash

   lerobot-train \
       --dataset.repo_id=your_dataset \
       --policy.type=act \
       --policy.chunk_size=100 \
       --policy.n_action_steps=100 \
       --batch_size=8 \
       --steps=100000 \
       --seed=1000

3. 3
   Train the flow matching side

   SmolVLA if you are on your own 24 GB card, Pi0.5 if you are renting 80 GB. Both integrate 10 Euler steps by default. Keep the seed fixed so the run is repeatable; lerobot defaults to 1000, and GR00T's tyro CLI exposes no seed at all, which is worth remembering if GR00T is your third arm of the comparison.

   bash

   lerobot-train \
       --dataset.repo_id=your_dataset \
       --policy.type=smolvla \
       --policy.pretrained_path=lerobot/smolvla_base \
       --batch_size=2 \
       --steps=20000 \
       --seed=1000

4. 4
   Cut the sampler steps and see if anything breaks

   This is the measurement that tells you whether the objective is doing work. Drop the integration steps and watch success rate against latency. If 4 steps performs like 10, you just bought back part of your loop budget, bounded by the action head's share of it. If it collapses, the multimodality is real and you are paying for something.

   python

   # integration steps, flow matching side (PI05Config / SmolVLAConfig)
   num_inference_steps = 10   # pi05; try 8, 6, 4, 2
   num_steps = 10             # smolvla, same idea, different field name
   
   # the fields to change, diffusion side (DiffusionConfig)
   noise_scheduler_type = "DDIM"
   num_inference_steps = 10   # instead of the None -> 100 fallback
   
   # then re-run the checkpoint
   # lerobot-eval --policy.path=./outputs/.../checkpoints/last/pretrained_model

5. 5
   Then cut the execution steps, which is the bigger lever

   Separately from the sampler, shorten how much of the chunk you run before re-observing. SmolVLA's own ablation moves 31 points of average LIBERO success between executing 50 actions and executing 10. Change one at a time or you will not know which one moved the number.

   bash

   # generate a 50-action chunk, execute only the first 10
   lerobot-train \
       --dataset.repo_id=your_dataset \
       --policy.type=smolvla \
       --policy.chunk_size=50 \
       --policy.n_action_steps=10 \
       --seed=1000

6. 6
   Compare on the arm, not on the loss curve

   Validation loss does not rank these. Run both on the same SO-100, same lighting, same object placement, 20 trials each, and count successes. Loss falls, policy does nothing exists because this step gets skipped.

Where flow matching does not help

Being honest about the limits is more useful than another ablation table. Flow matching is a way to sample from a multimodal action distribution cheaply. If your problem is not that, it fixes nothing.

• Single-mode tasks. If there is one right way to do the motion, a regression head gets there with one forward pass. The SmolVLA ablation shows regression ahead on two of four LIBERO suites.
• Latency. Pi0.5 runs 10 steps and is the slowest model here; ACT runs zero and is the fastest. Cutting steps helps in proportion to the action head's share of the loop, which is 37 percent for Pi0 on a 4090 and 56 percent for GR00T N1.7 on an H100 in eager mode.
• Bad data. An expressive head models whatever distribution you gave it, including the inconsistencies. It will reproduce your hesitations faithfully. Data collection quality outranks the objective every time.
• Memory. Flow matching does nothing for VRAM. Pi0.5 full fine-tuning needs the 80 GB tier and openpi says "> 70 GB". See out of memory during training.
• The backbone problem. Bolting a continuous expert onto a pretrained VLM has its own cost: Knowledge Insulating Vision-Language-Action Models (arXiv 2505.23705, 29 May 2025) reports that "naively including such experts significantly harms both training speed and knowledge transfer". That is an architecture problem, not an objective problem.