CrossFormer
UC Berkeley, USA · August 2024
UC Berkeley (Robotic AI and Learning Lab), Carnegie Mellon University
- Parameters
- 130 M
- 130M parameters including the ResNet-26 image encoders and the four action heads (paper Appendix B and the GitHub README,...
- GPU memory
- 0.2 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- downloadable checkpoint
What it is
CrossFormer is a 130M-parameter transformer policy trained on 900K trajectories from 20 embodiments, with four action heads that emit single-arm end-effector deltas, 2D navigation waypoints, 14-dimensional bimanual joint positions and 12-dimensional quadruped joint positions. One set of weights drives all of them at their native rates, from 4 Hz navigation to 20 Hz bimanual and quadruped control, with no manual alignment of observation or action spaces. Across 116 real trials it averages 73% against 51% for the best prior method per setting. The honest caveat comes from the authors themselves: there is no significant positive transfer across embodiments yet, so the model matches specialists rather than benefiting from their combined data. No VRAM, latency or simulation-benchmark numbers exist for it, and the repository has been static since August 2024.
Architecture
- Backbone
- No VLM backbone. A 12-layer transformer with 8 attention heads, token embedding size 512, MLP dimension 2048 and a context length of 2135, fed by ResNet-26 image encoders initialized from ImageNet weights.
- Action head
- Four separate linear action heads projecting readout-token embeddings to the action dimension, with the number of readout tokens matched to each embodiment's chunk size. No diffusion and no discretization.
- Action chunk
- 4 steps per forward pass
- Control rate
- 20 Hz
- Parameters
- 130M parameters including the ResNet-26 image encoders and the four action heads (paper Appendix B and the GitHub README, which both state 130M). The Hugging Face checkpoint is a single 0.52 GB JAX file. Chunk size differs per head: 4 for the single-arm head, 4 for the navigation head, 100 for the bimanual joint head and no chunking at all for the quadruped head. The value 4 recorded here is the single-arm manipulation head.
- Pretraining data
- 900K robot trajectories across 20 embodiments, assembled from Open X-Embodiment plus the authors' own ALOHA-multi-task, GNM, Go1-walk and Franka-tabletop data, with hand-picked per-dataset sampling weights (Table 1). Training ran 300K steps at batch size 512. The paper states training took 80 hours on a TPU v5e-256 pod, while the GitHub README states 47 hours on a TPUv5-256 pod; the two official sources disagree and neither is corrected.
- Embodiments
- WidowX (BridgeData V2), Franka Emika Panda (DROID setup), ALOHA bimanual (two arms, three cameras, 14-dimensional joint positions), LoCoBot wheeled navigation, Unitree Go1 quadruped (59-dimensional proprioception, direct joint control), Tello quadcopter (zero-shot, no quadcopter data in training)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 0.5 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.2 GB | The usual way these checkpoints are served |
| int8 | 0.1 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.1 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Control rate | 20 Hz |
| Checkpoint download | 0.52 GB |
No GPU memory requirement is published in the paper, the README or the model card. The README only says the model can be finetuned with accessible compute budgets, without a number, so none is recorded here.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- CrossFormer real-robot evaluation (paper Table 3) LoCoBot obstacle avoidance, 2 trials95%proportion of subgoals reachedReported by the CrossFormer authors. Best prior method 30%. Only 2 trials, so treat as directional evidence rather than a reliable rate.source
- CrossFormer real-robot evaluation (paper Table 3) 13 tasks across 6 embodiments, 116 trials total73%average success rateReported by the CrossFormer authors. The same architecture trained only on the target robot's data reaches 68% and the best prior method per setting reaches 51%. Go1 is excluded from the best-prior average because no suitable imitation baseline exists.source
- CrossFormer real-robot evaluation (paper Table 3) ALOHA bimanual, use the knife to cut the sushi, 10 trials60%success rateReported by the CrossFormer authors. Single-robot-dataset baseline 40%, best prior method 30%. This is the 20 Hz, 100-step-chunk bimanual head with three cameras.source
Fine tuned tasks
What the model reaches after being adapted to a new task or a new robot, usually from a small number of demonstrations. This is the number that matters if you bring your own data.
- CrossFormer real-robot evaluation (paper Table 3) Tello quadcopter cornering, 3 locations, 1 trial per location82%proportion of subgoals reachedReported by the CrossFormer authors. No quadcopter data appears in training, so this is zero-shot transfer to a new embodiment via the navigation waypoint head. The trial count is only 3, so the number carries very wide uncertainty.source
- CrossFormer real-robot evaluation (paper Table 3) Franka DROID, sweep the pinecones into the dustpan, 27 trials41%success rateReported by the CrossFormer authors. Here CrossFormer ties the single-robot baseline at 41% and loses to the best prior method at 52%, which is the clearest example of the paper's own statement that positive cross-embodiment transfer is not yet visible.source
On real hardware
All reported evaluation is on real hardware: 13 tasks across 6 embodiments with 116 trials in total. WidowX (Bridge setup, 4 tasks, 12 trials each), Franka (DROID setup, 27 and 12 trials), ALOHA bimanual (2 tasks, 10 trials each, three cameras, 20 Hz), LoCoBot navigation (3 skills, 2 trials each, combined with the graph-based planner from Shah et al.), Unitree Go1 walking (reward over 25 minutes normalized by the RL expert that generated the data, scoring 1.0) and a Tello quadcopter zero-shot (3 locations). Average 73% versus 68% for the same architecture trained on target-robot data only and 51% for the best prior method per setting.
Fine tuning it yourself
JAX codebase. Data must be converted to RLDS with the rlds_dataset_mod package. Three fine-tuning modes are documented: reuse the fully pretrained weights when the observation and action spaces match, initialize new tokenizers or action heads while keeping the transformer backbone, or freeze the transformer and train only the action head. Conditioning can be a goal image or a language instruction, and during training one of the two is randomly masked so either can be used at test time. No dataset size or GPU memory guidance is given.
Where it helps, where it does not
Strengths
- The same network weights control single arms, bimanual arms, wheeled robots, quadrupeds and a quadcopter, without any manual alignment of observation or action spaces
- Four action heads with independent chunk sizes let one model serve 4 Hz navigation and 20 Hz bimanual joint control at the same time
- Zero-shot transfer to a Tello quadcopter that never appeared in training, via the shared navigation waypoint head
- Small and cheap to serve at 130M parameters and a 0.52 GB checkpoint, and it ran on an RTX 4090 attached to the robot in the manipulation experiments
- Accepts a varying number of camera views (up to three simultaneously for the bimanual setting) rather than forcing views into one input slot
Limits
- The authors state plainly that their results do not yet show significant positive transfer across embodiments; matching specialist policies is the achievement, not beating them by learning across robots
- The data mixture relies on hand-picked sampling weights to avoid over-training on repetitive datasets, so reproducing or extending the mixture is not mechanical
- On the Franka sweeping task it scores 41% and loses to the best prior method at 52%, and it only ties the single-robot baseline on 5 of 13 tasks
- Several evaluation cells have tiny trial counts (2 trials for LoCoBot skills, 3 for the quadcopter, 10 to 12 for most manipulation tasks), so per-task percentages are noisy
- The authors name inference speed as a scaling limit: applying larger models to high-frequency embodiments may not be feasible on current hardware
- No VRAM, latency or throughput figures are published at all
- The paper and the README disagree on pretraining cost (80 hours versus 47 hours on a TPU v5e-256 pod)
- No language-conditioning quality analysis and no simulation benchmark such as LIBERO or SimplerEnv is reported, so it cannot be placed on those leaderboards
- Repository last pushed August 2024, with no maintenance since
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2408.11812
- https://arxiv.org/pdf/2408.11812
- https://crossformer-model.github.io/
- https://github.com/rail-berkeley/crossformer
- https://huggingface.co/rail-berkeley/crossformer
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.