ACT (Action Chunking with Transformers)
Stanford University, United States · April 2023
Stanford University, with UC Berkeley and Meta
- Parameters
- 80 M
- The paper states "The model has around 80M parameters" and that it is trained from scratch for each task (arXiv:2304.13705,...
- GPU memory
- 0.1 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- 10 ms
- Single 11 GB NVIDIA RTX 2080 Ti, the same machine used for training.
- Weights
- MIT
- MIT (original repository).
What it is
ACT is a small imitation-learning policy, not a foundation model. It maps four camera images and current joint positions directly to a chunk of 100 future joint-position targets, trained as a conditional VAE with a transformer encoder and decoder on top of ResNet-18 image encoders. Because it predicts a chunk instead of a single step and averages overlapping chunks at inference (temporal ensembling), compounding error over long horizons is reduced without any pretraining. It is the standard baseline for low-cost bimanual manipulation and the default entry policy in LeRobot, and it runs at 50 Hz on a single consumer GPU. The cost is generality: every task needs its own dataset and its own trained model, and the hardest precision tasks in the original paper still fail four times out of five.
Architecture
- Backbone
- No VLM. Four ResNet-18 image encoders (one per camera, 480x640 inputs) feed a transformer encoder together with joint positions and a CVAE style variable z. A BERT-like transformer encoder acts as the CVAE encoder during training and is discarded at test time.
- Action head
- Transformer decoder that regresses a chunk of k absolute joint-position targets (k x 14) in one forward pass. Trained with L1 reconstruction loss plus a KL term (conditional VAE). Temporal ensembling averages overlapping chunks at inference with weights w_i = exp(-m*i).
- Action chunk
- 100 steps per forward pass
- Control rate
- 50 Hz
- Parameters
- The paper states "The model has around 80M parameters" and that it is trained from scratch for each task (arXiv:2304.13705, Section IV.C). The LeRobot re-implementation checkpoint lerobot/act_aloha_sim_transfer_cube_human is listed at 51.7M parameters on its model card, because it uses fewer camera streams than the four-camera ALOHA setup.
- Pretraining data
- None. ACT has no pretraining corpus and is trained from scratch per task. The paper uses 50 human demonstrations per real task (100 for Thread Velcro), roughly 10 to 20 minutes of data per task and 30 to 60 minutes wall clock including resets.
- Embodiments
- ALOHA bimanual setup: two ViperX 6-DoF follower arms plus grippers, 14-DoF action space, teleoperated by two WidowX leader arms, Four Logitech C922x webcams at 480x640, two wrist mounted, one front, one top, Simulated ALOHA in MuJoCo / dm_control, LeRobot documents running ACT on SO-101 follower arms
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.3 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.1 GB | The usual way these checkpoints are served |
| int8 | 0.1 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.0 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Inference latency, reported | 10 ms on Single 11 GB NVIDIA RTX 2080 Ti, the same machine used for training. The paper states the inference time is around 0.01 seconds. |
| Control rate | 50 Hz |
The paper trains ACT in about 5 hours on a single 11 GB NVIDIA RTX 2080 Ti. No VRAM consumption figure is published, so the 11 GB is the capacity of the card the authors used, not a measured requirement. The LeRobot documentation states ACT "trains in a few hours on a single GPU" and recommends starting at batch size 8.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- ALOHA simulation (MuJoCo) Cube Transfer, trained on scripted data, 3 seeds x 50 evaluations86%success rate, final subtask (Transfer)Authors' own evaluation. The same task drops to 50 percent when trained on human demonstrations instead of scripted data.source
- ALOHA simulation via LeRobot AlohaTransferCube, human demonstrations, 500 evaluation episodes83%success rateThird-party re-implementation by the LeRobot team, not the ACT authors. The same model card reports 68.0 percent for a comparable checkpoint from the original ACT repository.source
- ALOHA simulation (MuJoCo) Bimanual Insertion, trained on scripted data, 3 seeds x 50 evaluations32%success rate, final subtask (Insert)Authors' own evaluation. Drops to 20 percent when trained on human demonstrations. This is the hardest simulated task in the paper.source
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Real ALOHA hardware Slot Battery, 25 evaluations, 1 seed, human demonstrations96%
- Real ALOHA hardware Open Cup, 25 evaluations, human demonstrations84%success rate, final subtask (Open Lid)Authors' own evaluation. Best baseline (BeT) scores 0 percent on this subtask.source
- Real ALOHA hardware Thread Velcro, 25 evaluations, 100 human demonstrations20%success rate, final subtask (Insert)Authors' own evaluation. The weakest real task in the paper and the clearest evidence of the precision ceiling.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
Six real ALOHA tasks reported by the authors, each trained on 50 human demonstrations (100 for Thread Velcro), evaluated over 25 rollouts with one seed. Final-subtask success rates: Slot Battery 96 percent, Put On Shoe 92 percent, Slide Ziploc 88 percent, Open Cup 84 percent, Prep Tape 64 percent, Thread Velcro 20 percent. Demonstration episodes run 8 to 14 seconds at 50 Hz, which is 400 to 700 timesteps per episode.
Fine tuning it yourself
There is no pretrained ACT trunk to fine-tune. Training is from scratch per task. The dataset needs synchronised RGB frames from each camera plus current follower joint positions as observation, and the leader-arm absolute joint positions as the action target. The paper uses 50 demonstrations per task; LeRobot documents ACT as often reaching high success with 50 demonstrations. Reference training command in the original repo sets --chunk_size 100.
This is one of the models the AY-Robots training pool can fine tune for you on a dataset recorded with your own arm. The platform page carries the defaults, GPU tier and inference latency measured in that pool.
Platform page for ACT (Action Chunking with Transformers)Where it helps, where it does not
Strengths
- Trains from scratch on 50 demonstrations per task and reaches 80 to 90 percent on several real fine manipulation tasks (arXiv:2304.13705).
- Around 80M parameters and roughly 10 ms inference on an RTX 2080 Ti, so it sustains the 50 Hz ALOHA control rate on a single consumer GPU.
- Action chunking plus temporal ensembling shortens the effective task horizon by a factor of k and produces smooth trajectories without a separate smoothing filter.
- MIT licensed reference code, plus a maintained implementation in LeRobot that the LeRobot documentation names as the first policy to try.
Limits
- No pretraining and no language conditioning. One policy is trained per task and does not transfer to a new task, a new object set or a new instruction.
- Precision-critical insertion remains the failure mode: 20 percent on real Thread Velcro, and 32 percent (scripted) or 20 percent (human data) on simulated Bimanual Insertion.
- All reported results come from a single hardware configuration, the ALOHA setup with four fixed cameras at 480x640 and 50 Hz recording. The paper reports no cross-embodiment transfer.
- Temporal ensembling adds inference-time compute and a smoothing hyperparameter m that has to be chosen per task.
- Trained on absolute leader-arm joint positions. The paper notes degraded performance when delta joint positions are used as actions instead.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2304.13705
- https://arxiv.org/pdf/2304.13705
- https://github.com/tonyzhaozh/act
- https://tonyzhaozh.github.io/aloha/
- https://huggingface.co/docs/lerobot/en/act
- https://huggingface.co/lerobot/act_aloha_sim_transfer_cube_human
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.