CogACT-Base
Microsoft, United States · November 2024
Microsoft Research
- Parameters
- 7.6 B
- The paper states 7.6B total parameters for the default configuration, which uses the DiT-Base action module: "our model being...
- GPU memory
- 30 GB
- reported by the authors
- Inference latency
- 181 ms
- Single NVIDIA A6000 GPU.
- Weights
- MIT
- downloadable checkpoint
What it is
CogACT is a componentized VLA that keeps a Prismatic vision language model for perception and language and hands action generation to a separate diffusion transformer, which is conditioned on one cognition token rather than on quantized action tokens. The Base variant pairs that VLM with an 89M parameter DiT and emits a chunk of 16 future 7-DoF end effector deltas per forward pass. On SIMPLER it reaches 74.8 percent average success on Google Robot visual matching against 34.3 percent for OpenVLA at comparable size, and 51.3 percent on the WidowX visual matching suite. Everything is MIT licensed and all three sizes are downloadable, but the checkpoints ship as 30 GB fp32 files and the only published memory figure is about 30 GB in fp32, so running on a 24 GB card depends on a bf16 path the authors mention but never measure. Pretraining covers roughly 0.4M Open X-Embodiment trajectories, which makes this a single camera, single arm policy with no bimanual or dexterous hand evidence.
Architecture
- Backbone
- Prismatic VLM at 7B scale: DINOv2 ViT-L/14 plus SigLIP ViT-So400M/14 vision encoders with a Llama-2 language module
- Action head
- Diffusion transformer (DiT-Base, 89M parameters) conditioned on a single cognition token emitted by the language module, sampled with DDIM and classifier-free guidance
- Action chunk
- 16 steps per forward pass
- Control rate
- 5.5 Hz
- Parameters
- The paper states 7.6B total parameters for the default configuration, which uses the DiT-Base action module: "our model being much smaller, with 7.6B parameters compared to RT-2-X that has 55B parameters". Table 7 gives the action module sizes separately as DiT-Small 13M, DiT-Base 89M and DiT-Large 308M. The published checkpoint is a single 30.52 GB fp32 .pt file.
- Pretraining data
- A subset of Open X-Embodiment, about 0.4M trajectories according to the project page.
- Embodiments
- Google Robot (SIMPLER simulation), WidowX / Bridge (SIMPLER simulation), Realman single arm (real hardware), Franka Emika (real hardware)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 28 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 14 GB | The usual way these checkpoints are served |
| int8 | 7.1 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 3.5 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 24 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Inference memory, reported | 30 GB |
| Inference latency, reported | 181 ms on Single NVIDIA A6000 GPU. 181 ms per forward pass producing 16 actions, versus 307 ms for one action with OpenVLA on the same setup. The README states an action generation frequency of about 5.5 Hz using the Adaptive Action Ensemble strategy, which the action chunk then multiplies. source |
| Control rate | 5.5 Hz |
| Checkpoint download | 30.52 GB |
Inference needs roughly 30 GB in fp32 per the README and model card. Loading the VLM in bf16 is supported to reduce this, but no measured bf16 number is published. Fine-tuning reference setup is one node with 8 A100 GPUs.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- SIMPLER Google Robot, visual matching, average over 4 tasks74.8%success rateAuthors' own main results table. Baselines in the same table: OpenVLA 34.3, RT-2-X 46.3, Octo-Base 11.0.source
- SIMPLER Google Robot, variant aggregation, average over 4 tasks61.3%success rateAuthors' own main results table. Baselines: OpenVLA 39.3, RT-2-X 54.4, Octo-Base 1.2.source
- SIMPLER WidowX / Bridge, visual matching, average over 4 tasks51.3%success rateAuthors' own main results table. Baselines: OpenVLA 4.2, Octo-Small 26.7, RT-1-X 1.1.source
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Real robot (Realman arm) 3 tasks, fine-tuned on 391 demonstrations after OXE pretraining71.2%average success rateAuthors' own hardware. OpenVLA under the same protocol reaches 12.1 percent.source
- Real robot (Franka Emika) 4 tasks, fine-tuned on 400 demonstrations after OXE pretraining61.4%average success rateAuthors' own hardware. OpenVLA under the same protocol reaches 6.8 percent.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
Fine-tuned and evaluated on a Realman arm across three tasks using 391 demonstrations, reaching 71.2 percent average success versus 12.1 percent for OpenVLA, and on a Franka Emika arm across four tasks using 400 demonstrations, reaching 61.4 percent versus 6.8 percent for OpenVLA. Both were pretrained on Open X-Embodiment first. Deployment on real hardware runs at roughly 5.5 Hz policy queries on a single A6000.
Fine tuning it yourself
Fine-tuning consumes Open X-Embodiment style RLDS datasets, or custom data converted to RLDS, with 7-DoF actions expressed as end effector deltas (XYZ, roll pitch yaw, gripper open/close). Requires Python 3.10, PyTorch 2.2 or newer, CUDA 12 or newer, and Flash-Attention 2 for training. The reference configuration trains on one node with 8 A100 GPUs at global batch size 256 and per device batch size 32. Real robot results were obtained with 391 and 400 demonstrations respectively.
Where it helps, where it does not
Strengths
- Action chunk of 16 steps from a single forward pass, so one 181 ms inference on an A6000 covers many control steps
- Large measured margin over OpenVLA at similar size on SIMPLER and on two real arms
- MIT license on code, weights and data, with all three sizes actually downloadable
- Published action module scaling study (13M, 89M, 308M) that shows where the extra capacity goes
Limits
- Single third person RGB camera and single arm only. No bimanual, no dexterous hand and no wrist camera results are reported.
- The only published inference footprint is about 30 GB in fp32. No bf16, no quantized and no measured minimum GPU figure exists.
- About 5.5 Hz of policy queries on an A6000, so smooth control depends entirely on the action chunk and the adaptive action ensemble.
- Pretraining uses only about 0.4M Open X-Embodiment trajectories, far less than later foundation VLAs.
- No LIBERO, CALVIN or RoboCasa numbers are published, so cross comparison with newer models has to go through SIMPLER alone.
- Repository activity has been limited to maintenance since late 2025 and no successor model exists as of August 2026.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2411.19650
- https://arxiv.org/html/2411.19650v1
- https://cogact.github.io/
- https://github.com/microsoft/CogACT
- https://huggingface.co/CogACT/CogACT-Base
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.