Diffusion Policy (CNN-based, DiffusionPolicy-C)
Columbia University, United States · March 2023
Columbia University, with Toyota Research Institute and MIT
- Parameters
- 278 M
- About 278 M for the image based Robomimic configuration: the appendix hyperparameter table lists 256 M for the diffusion U-Net...
- GPU memory
- 0.5 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- 100 ms
- NVIDIA RTX 3080, using DDIM with 100 training denoising iterations and 10 inference iterations.
- Weights
- MIT
- downloadable checkpoint
What it is
Diffusion Policy represents the visuomotor policy as a conditional denoising diffusion process over a chunk of future actions rather than as a direct regression to one action. The CNN variant conditions a 1D temporal convolutional U-Net denoiser on the observation embedding, and the authors recommend it as the more robust of the two variants in practice. It was benchmarked on 15 tasks from 4 behaviour-cloning benchmarks with a reported average improvement of 46.9 percent over prior methods, and it is the reference architecture that most later chunk-based action heads build on. The practical constraint is the denoising loop: even with DDIM cut to 10 steps, inference costs about 0.1 s on an RTX 3080, so real deployments run the policy near 10 Hz and interpolate up to the servo rate.
Architecture
- Backbone
- No VLM and no language input. A vision encoder produces an observation embedding that conditions a 1D temporal convolutional denoising network (U-Net) operating over the action sequence.
- Action head
- Conditional denoising diffusion (DDPM) over a chunk of future actions, executed in receding-horizon fashion. At inference the authors switch to DDIM to cut the number of denoising steps from 100 to 10.
- Control rate
- 10 Hz
- Parameters
- About 278 M for the image based Robomimic configuration: the appendix hyperparameter table lists 256 M for the diffusion U-Net plus 22 M for the vision encoder. Transport uses a larger encoder (264 M plus 45 M) and the real world tasks a smaller network (67 M plus 22 M), so the number depends on the task configuration.
- Pretraining data
- None. Diffusion Policy is trained per task from demonstrations. The paper evaluates 15 tasks across 4 existing behaviour-cloning benchmarks (arXiv:2303.04137v5 abstract) and reports an average improvement of 46.9 percent over prior methods.
- Embodiments
- Robomimic simulation tasks Lift, Can, Square and ToolHang (single arm) and Transport (two arms), Push-T, simulated and real, Franka Kitchen and BlockPush simulation
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 1.0 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.5 GB | The usual way these checkpoints are served |
| int8 | 0.3 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.1 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Inference latency, reported | 100 ms on NVIDIA RTX 3080, using DDIM with 100 training denoising iterations and 10 inference iterations. Quoted: "enables 0.1s inference latency on a Nvidia 3080 GPU". |
| Control rate | 10 Hz |
The paper does not state a minimum GPU or a VRAM figure. The only hardware named is an RTX 3080 used for real-world inference. Training length is given as 4500 epochs for state-based tasks and 3000 epochs for image-based tasks.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- Robomimic Square, proficient human demonstrations, state observations93%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic Transport, proficient human demonstrations, image observations93%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic Square, proficient human demonstrations, image observations92%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Push-T state observations91%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Push-T image observations84%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic Transport, proficient human demonstrations, state observations82%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic ToolHang, proficient human demonstrations, image observations73%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic ToolHang, proficient human demonstrations, state observations30%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
The authors report real-world experiments including a real Push-T setup. The policy is queried at 10 Hz and its commands are linearly interpolated to 125 Hz for robot execution, with DDIM giving 0.1 s inference latency on an RTX 3080. The paper does not publish a consolidated real-world success-rate table in the form the simulation tables use, so the defensible real-world numbers here are the latency and control-rate figures rather than task success rates.
Fine tuning it yourself
Trained per task from a demonstration dataset in the repository's zarr format (per-episode arrays for actions, images and low-dimensional state). No pretrained trunk to adapt. Key knobs are the observation horizon, the prediction horizon and the action-execution horizon of the receding-horizon controller, plus the number of DDIM inference steps, which trades latency against action quality. The paper trains 4500 epochs for state-based tasks and 3000 for image-based tasks.
Where it helps, where it does not
Strengths
- Handles multimodal action distributions natively, which is where single-mode regression policies collapse to an averaged and unusable action.
- Consistent across a wide test set: 15 tasks over 4 benchmarks with a reported average improvement of 46.9 percent over prior state of the art.
- DDIM at inference cuts denoising from 100 to 10 steps and gets latency to 0.1 s on an RTX 3080, which is enough for 10 Hz closed-loop control.
- MIT licensed code, and the architecture is the base of many later policies, so tooling and re-implementations are widely available (LeRobot among them).
Limits
- Latency is dominated by the iterative denoising loop. 0.1 s per inference on an RTX 3080 caps the policy loop at roughly 10 Hz, which is 5 times slower than ACT on comparable hardware and rules out direct high-rate joint control without interpolation.
- In the authors' real-world setup the policy runs at 10 Hz and the commands are linearly interpolated up to 125 Hz for execution, so the smooth motion the robot shows is partly the interpolator, not the policy.
- No pretraining, no language conditioning and no cross-embodiment transfer. Every task needs its own dataset and its own trained model.
- The paper publishes no parameter count, no VRAM figures and no training wall-clock times, so hardware sizing has to be derived from the code rather than from the paper.
- Performance degrades on multi-human (mixed proficiency) demonstration splits compared with proficient-human splits.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2303.04137
- https://arxiv.org/html/2303.04137v5
- https://github.com/real-stanford/diffusion_policy
- https://huggingface.co/lerobot/diffusion_pusht
- https://journals.sagepub.com/doi/full/10.1177/02783649241273668
- https://diffusion-policy.cs.columbia.edu/
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.