CogACT and the VLM Plus Action Head Pattern in VLAs

CogACT is the clearest statement of an idea behind most current vision-language-action models: stop making the language model spell actions out as text tokens, and hang a dedicated action network off it instead. A 7B vision-language model emits one vector per timestep; a diffusion transformer turns it into a chunk of 16 actions.

The paper is arXiv:2411.19650, posted 29 November 2024 by a group from Tsinghua, Microsoft Research Asia, USTC and CAS. Still at v1. Code sits at github.com/microsoft/CogACT under MIT, with three checkpoints on Hugging Face. This describes the repo as of its last push, 30 October 2025.

What CogACT actually is

Three blocks in a row. The paper is unusually explicit about the sizes, which makes it a good reference point for reasoning about where the parameters in a policy should go.

The action space is a 7-vector: three end-effector translation offsets, three rotation changes, one gripper bit. Vision and language weights are initialised from OpenVLA's checkpoint (arXiv:2406.09246), itself built on the Prismatic VLM design (arXiv:2402.07865). Pre-training uses the Open X-Embodiment subset Octo and OpenVLA also use, about 22.5 million frames.

The interface is one token

The language module gets visual tokens, instruction tokens, and one extra learnable token the paper calls the cognition token. The output feature at that position is the whole message passed to the action module. Everything the diffusion transformer knows about the scene and task arrives through that vector.

The action module denoises 17 slots: the current action, 15 future ones, and the conditioning. Training minimises MSE between predicted and true noise over 100 diffusion timesteps on a squaredcos_cap_v2 schedule. At inference the repo uses DDIM with 10 steps and classifier-free guidance at scale 1.5.

from PIL import Image
from vla import load_vla
import torch

model = load_vla(
    'CogACT/CogACT-Base',          # or CogACT-Small / CogACT-Large
    load_for_training=False,
    action_model_type='DiT-B',     # must match the weights: DiT-S / DiT-B / DiT-L
    future_action_window_size=15,
)
# about 30 GB of memory in fp32
model.to('cuda:0').eval()

actions, _ = model.predict_action(
    image,
    "move sponge near apple",
    unnorm_key='fractal20220817_data',   # dataset the action stats come from
    cfg_scale=1.5,                       # 1.5 to 7 also works
    use_ddim=True,
    num_ddim_steps=10,
)
# actions.shape == (16, 7)

Note unnorm_key. The model predicts normalised actions in [-1, 1] and needs per-dataset statistics to get back to metres and radians. Get it wrong and the arm moves confidently in the wrong units, which looks like a bad checkpoint until you check. It shows up here as a policy that only works in one setup.

What the split buys you

Ten points from architecture, at identical parameter count

Table 7 of the paper is the one to read twice, because it holds parameters fixed and varies only the head. GR is the Google robot in SIMPLER, WR the WidowX, VM visual matching, VA variant aggregation. Success rates in percent.

Read the two 89M rows against each other: same budget, same backbone, same data, 52.5 against 62.5. Then the 13M DiT against the 89M MLP, where a head seven times smaller wins by 6 points. The gain is not parameters. It is attention over the action horizon plus a generative objective.

Predicting the future is most of the win

The second ablation varies how many future actions the head predicts, which is action chunking by another name. Zero means single-step prediction.

A third knob decides how overlapping chunks combine at inference. CogACT weights past predictions by cosine similarity to the current one, so different behaviour modes are not averaged into an action belonging to neither. Plain chunking scored 50.7, ACT's temporal ensemble (arXiv:2304.13705) 58.9, the adaptive ensemble 62.5.

The headline benchmark numbers

SIMPLER (arXiv:2405.05941) is built to correlate with real hardware. Visual matching replicates the real scene; variant aggregation perturbs background, lighting, distractors and table texture. Four tasks per setting:

On real hardware the gap is wider. Over three Realman-arm tasks after fine-tuning on 391 demonstrations: Octo-Base 4.9, OpenVLA 12.1, CogACT 71.2. On a Franka with 400 demonstrations: 5.8, 6.8 and 61.4. All three CogACT variants sit in the Arena, which carries 85 VLA models and 332 benchmark results.

What the split costs you

It is not a frozen backbone, and the hardware bill says so

The common misreading is that the VLM is frozen and only the head trains. It is not. Pre-training ran on 16 A100s for roughly 5 days under PyTorch FSDP: batch 256, constant learning rate 2e-5, 135K iterations, 8 diffusion steps per example. Fine-tuning for the real-robot experiments also used 16 A100s at batch 256, with the tested checkpoint at 10K steps, or 7.5 hours.

The data format tax

CogACT trains on RLDS, the format Open X-Embodiment uses. Your data has to be converted with rlds_dataset_builder, dropped into a custom_finetuning directory and selected with vla.data_mix="custom_finetuning". That part is mechanical. The action convention is not.

This is the general tax on the pattern, not a CogACT quirk: a head is trained against one action parameterisation and does not transfer across conventions for free. What differs is how tightly that convention is baked in. GR00T N1.7 descends from a design that puts an embodiment-specific MLP encoder in front of the action head and a matching decoder behind it, so the shared head takes whatever the dataset holds rather than a fixed 7-DoF end-effector vector. That is why a joint-space teleoperated recording works with it, and with Pi0.5, without the conversion CogACT demands.

The published numbers came from code that has since changed

On 13 January 2025 the repo landed commit 5ddf9b1: "fix a bug in forward_with_cfg, which cause a slight decrease in average performance during inference when using cfg". The diff moves two tensor slices from dimension 1 to dimension 2. Every headline number used CFG at 1.5, so reproducing the tables exactly against current main is not guaranteed. Worth knowing before you lose a week to being half a point off.

Adoption, measured rather than asserted

Papers get cited; weights get downloaded. Hugging Face API figures for the 30 days to 23 August 2026.

The weights have not moved since December 2024. The repo has, last on 30 October 2025 for Azure ML integration, and carries 431 stars and 40 forks. Treat CogACT as a reference implementation of an idea, not a maintained production stack.

Running CogACT yourself

To reproduce the SIMPLER numbers or fine-tune on your own arm, this is the path. README requirements: Python 3.10, PyTorch 2.2.0 or newer, CUDA 12.0 or newer.

1. 1
   Install the package

   The base install is inference only. The [train] extra pulls in Flash Attention 2, which is what takes the time.

   bash

   conda create --name cogact python=3.10 -y
   conda activate cogact
   
   git clone https://github.com/microsoft/CogACT
   cd CogACT
   pip install -e .
   
   # training only: adds flash-attn 2.5.5
   pip install -e .[train]

2. 2
   Pull a checkpoint

   30 GB over git-lfs, or pass the model id to the training script and let it download.

   bash

   git lfs install
   git clone https://huggingface.co/CogACT/CogACT-Base
   
   export HF_TOKEN=hf_...

3. 3
   Evaluate in SIMPLER

   Install SimplerEnv, copy the adapter in, register it in main_inference.py. One script ships per task and setting.

   bash

   cp ./sim_cogact <your_path_to_simpler>/simpler_env/policies -r
   cd <your_path_to_simpler>
   bash simpler_env/policies/sim_cogact/scripts/cogact_put_in_drawer_visual_matching.sh

4. 4
   Fine-tune on your own data

   Eight A100s in the README example; the paper used sixteen. Around 30 epochs already yields good results.

   bash

   torchrun --standalone --nnodes 1 --nproc-per-node 8 scripts/train.py \
     --pretrained_checkpoint CogACT/CogACT-Base \
     --vla.type prism-dinosiglip-224px+oxe+diffusion \
     --vla.data_mix custom_finetuning \
     --vla.expected_world_size 8 \
     --vla.global_batch_size 256 \
     --vla.per_device_batch_size 32 \
     --vla.learning_rate 2e-5 \
     --data_root_dir <path_to_dataset_dir> \
     --run_root_dir <path_to_checkpoint_dir> \
     --image_aug True \
     --repeated_diffusion_steps 8 \
     --future_action_window_size 15 \
     --action_model_type DiT-B \
     --is_resume False

5. 5
   Serve it to the robot

   scripts/deploy.py starts an HTTP server; the client needs only requests. --action_ensemble and --action_chunking are mutually exclusive, enforced by an assert.

   bash

   python scripts/deploy.py \
     --saved_model_path <your_model_path> \
     --unnorm_key custom_finetuning \
     --action_ensemble \
     --use_bf16 \
     --action_ensemble_horizon 2 \
     --adaptive_ensemble_alpha 0.1 \
     --cfg_scale 1.5 \
     --port 5500

One easily missed detail: the ensemble horizon is not a constant. The supplement gives the rule, not the numbers. It fixes K x std(action) = 0.2 across datasets, so a faster robot or a higher observation frequency gets a different K. The values that rule produced sit in the repo: sim_cogact/cogact_policy.py sets K = 2 for the Google robot setup and K = 7 for the WidowX BridgeData setup, and the deploy script defaults to 2.

Two routes to the same goal

Say the goal is a diffusion-transformer action head, conditioned on a pre-trained VLM, driving a real low-cost arm on a task you recorded. The honest comparison:

The same pattern in policies you can actually train

CogACT is one point in a design space most current VLAs occupy. Side by side, the interesting variable is not the backbone but what sits after it.

Two deserve a note. Pi0 routes tokens to two weight sets inside one transformer: images and language to the PaliGemma backbone, state and noisy actions to a 300M expert, interacting only through self-attention. Tighter coupling than CogACT's single token, and it uses flow matching rather than DDPM-style diffusion. OpenVLA-OFT (RSS 2025) is the counterexample: it drops diffusion for continuous actions with plain L1 regression and parallel decoding, reporting 26x higher throughput than OpenVLA and LIBERO success from 76.5 to 97.1 percent. A diffusion head is not the only fix.

The latency arithmetic, stated carefully

The repo reports a real measurement: CogACT-Base in bfloat16 on a single A6000, called 100 times, averages about 181 ms per inference, roughly 5.5 Hz. OpenVLA on the same card averaged 307 ms. The reason given is that one cognition token generates the whole sequence, while an OpenVLA-style model emits seven tokens for one 7-dimensional action.

The honest limit applies to all of them: inference latency has to sit next to the servos for fast tasks. The control loop is 20 to 485 ms per action step, and public-internet round trips turn a working policy into a hesitant one. Remote inference is viable for slow pick-and-place, not fast reactive motion.

What I would actually do with this

CogACT is more valuable as an argument than as a checkpoint. At fixed parameter budget the shape of the action head is worth about 10 points, chunk length about 20, the ensemble rule about 12. Large numbers for decisions that cost nothing at training time.

• Choosing a policy for a real arm: start from the head design, not backbone size.
• Do not raise chunk length past what your camera view supports. 31 future steps beat 15 in nothing.
• With 16 A100s and an RLDS pipeline, fine-tuning CogACT is a reasonable research move. With one arm and a laptop, SmolVLA on the SO-100 at 1 to 3 USD a run teaches more per day.
• Read the ablations, not the headline. "Beats OpenVLA by 35 percent" tells you less than "89M MLP scored 52.5, 89M DiT scored 62.5".
• Check adoption first. Weights untouched since December 2024 is a reference implementation, not infrastructure.

For why VLAs are structured this way, the VLA overview covers the lineage, the RT-2 article the token-based approach CogACT argues against, and the Pi-Zero article the flow-matching branch. Never trained a policy? Train your first policy is the shortest useful path, and the training docs describe what the backend does with your run.