Diffusion Policy (Transformer-based, DiffusionPolicy-T)
Columbia University, United States · March 2023
Columbia University, with Toyota Research Institute and MIT
- Parameters
- 31 M
- About 31 M for the image based Robomimic configuration: 9 M for the diffusion transformer plus 22 M for the vision encoder,...
- GPU memory
- 0.1 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- closed
- no public checkpoint
What it is
DiffusionPolicy-T replaces the 1D temporal convolutional denoiser of the CNN variant with a time-series diffusion transformer that cross-attends to the observation embedding. On the paper's state-based Robomimic splits it reaches the highest scores of any configuration on Lift, Can and Square, and it is the stronger option on the multi-stage BlockPush and Franka Kitchen tasks. On the harder long-horizon tasks it falls behind the CNN variant, losing 4 points on state-based ToolHang. Since the paper reports no parameter count and no latency figure for this variant, hardware sizing for it has to come from the code, not from the publication.
Architecture
- Backbone
- No VLM and no language input. A vision encoder produces an observation embedding that conditions a time-series diffusion transformer, which denoises the action sequence with cross-attention to the observation embedding.
- Action head
- Conditional denoising diffusion (DDPM) over an action chunk, transformer denoiser instead of the U-Net used by the CNN variant, DDIM at inference.
- Control rate
- 10 Hz
- Parameters
- About 31 M for the image based Robomimic configuration: 9 M for the diffusion transformer plus 22 M for the vision encoder, roughly one ninth of the CNN variant. Transport uses a larger encoder (9 M plus 45 M).
- Pretraining data
- None. Trained per task from demonstrations, same evaluation protocol as the CNN variant.
- Embodiments
- Robomimic simulation tasks Lift, Can, Square and ToolHang (single arm) and Transport (two arms), Push-T, simulated, Franka Kitchen and BlockPush simulation
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.1 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.1 GB | The usual way these checkpoints are served |
| int8 | 0.0 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.0 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Control rate | 10 Hz |
The paper does not state a minimum GPU or a VRAM figure, and the 0.1 s on RTX 3080 latency figure is reported for the real-world deployment without attributing it specifically to the transformer variant. Do not transfer the CNN variant's latency figure to this row.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- Robomimic Square, proficient human demonstrations, image observations90%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic Square, proficient human demonstrations, state observations89%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic ToolHang, proficient human demonstrations, state observations87%success rateAverage of the last 10 checkpoints. Far ahead of the CNN variant here (30), and behind it on the image split.source
- Robomimic Transport, proficient human demonstrations, state observations84%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic Transport, proficient human demonstrations, image observations81%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Push-T state observations79%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Push-T image observations66%success rateAverage of the last 10 checkpoints, the second number in each cell of the paper table.source
- Robomimic ToolHang, proficient human demonstrations, image observations47%
Real world
No results in this category are published for this model.
Fine tuned tasks
No results in this category are published for this model.
On real hardware
The paper's real-world deployment numbers (0.1 s inference on an RTX 3080, 10 Hz policy rate interpolated to 125 Hz) are reported for the real-world experiment setup without being attributed to the transformer variant specifically, so no real-world figure is claimed for this row.
Fine tuning it yourself
Same pipeline as the CNN variant: per-task training from a zarr demonstration dataset, no pretrained trunk. Selected in the repository by choosing the transformer denoiser config instead of the U-Net config.
Where it helps, where it does not
Strengths
- Reaches 100 percent average-of-last-10-checkpoints success on state-based Robomimic Lift, Can and Square, the strongest numbers in the paper's state-policy table.
- Handles high-dimensional action spaces and multi-stage tasks well, which is where the paper's BlockPush and Franka Kitchen results separate it from earlier baselines.
- Same MIT licensed codebase and same training pipeline as the CNN variant, so switching between the two is a config change.
Limits
- Less stable than the CNN variant on the harder splits. It loses 2 points on state-based Transport and 4 points on state-based ToolHang against DiffusionPolicy-C.
- The paper does not publish a latency measurement, a parameter count or a VRAM figure for this variant specifically.
- Same structural constraints as the CNN variant: no pretraining, no language conditioning, one model per task, and an iterative denoising loop at inference.
- More sensitive to hyperparameters than the CNN variant in the authors' benchmark tables, particularly on multi-human demonstration splits.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2303.04137
- https://arxiv.org/html/2303.04137v5
- https://github.com/real-stanford/diffusion_policy
- https://diffusion-policy.cs.columbia.edu/
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.