Embodied Chain-of-Thought (ECoT)
UC Berkeley, USA, Poland · July 2024
UC Berkeley, University of Warsaw, Stanford University
- Parameters
- 7.2 B
- The Hugging Face safetensors index for Embodied-CoT/ecot-openvla-7b-bridge reports 7,188,124,608 BF16 parameters, which...
- GPU memory
- 13 GB
- weights at bf16, computed. Fits a 24 GB card
- Inference latency
- not published
- per action step
- Weights
- MIT
- MIT for the code.
What it is
ECoT trains a VLA to write out an embodied reasoning chain, covering plan, sub-task, movement primitive, object bounding boxes and gripper pixel position, before emitting its action tokens. Built on OpenVLA with no architecture change and no extra robot demonstrations, it lifts real WidowX success from 44% to 66% on an in-distribution camera view and from 30% to 64% when the camera is moved, and it beats the closed 55B RT-2-X. The price is decoding about 350 tokens per timestep instead of 7, and the paper never publishes an absolute control frequency, only relative speed-ups of 24% for a 5-step reasoning freeze and 40% for asynchronous execution. It runs in 16 GB in bfloat16 or about 5 GB in 4-bit. Evaluation covers exactly one embodiment, so treat cross-robot claims as unproven on hardware. Published at the 8th Conference on Robot Learning (PMLR volume 270).
Architecture
- Backbone
- OpenVLA, that is a Prismatic-7B VLM with a Llama 2 7B decoder and pretrained vision encoders. ECoT does not change the architecture, only the training targets.
- Action head
- Autoregressive token prediction. The model first emits a reasoning chain (plan, sub-task, movement primitive, visible-object bounding boxes, gripper position) and then the 7 discrete action tokens, so about 350 tokens are decoded per timestep instead of 7.
- Action chunk
- 1 step per forward pass
- Parameters
- The Hugging Face safetensors index for Embodied-CoT/ecot-openvla-7b-bridge reports 7,188,124,608 BF16 parameters, which matches the SigLIP-only Prismatic variant used in the OpenVLA ablations rather than the 7,541M released OpenVLA checkpoint. Two checkpoints are published: ecot-openvla-7b-bridge (trained on the ECoT Bridge data) and ecot-openvla-7b-oxe (the OpenVLA Open-X checkpoint continued on a mix in which about 13% is ECoT data).
- Pretraining data
- The reasoning data was generated synthetically over the complete BridgeData V2 dataset, more than 2.5M transitions, using a pipeline of Prismatic-7B for scene description plus off-the-shelf detection and Gemini for plan and sub-task labels, run over 7 days. The released embodied_features_bridge dataset is public. The base weights carry OpenVLA's 970k-episode Open X-Embodiment pretraining.
- Embodiments
- WidowX 6-DoF arm (BridgeData V2), single third-person camera, end-effector velocity control
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 27 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 13 GB | The usual way these checkpoints are served |
| int8 | 6.7 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 3.3 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 24 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
| Checkpoint download | 14.38 GB |
The paper reports cost in tokens per timestep (7 for OpenVLA against 350 for ECoT), not in GPU memory. It shares the OpenVLA 7 B checkpoint, so size it from that.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
No results in this category are published for this model.
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- ECoT accelerated inference study 3 tasks, 25 trials total72%success rate with a 5-step reasoning freezeReported by the ECoT authors, Table 2. Naive per-step reasoning 63% at baseline speed, asynchronous execution 65% at 40% faster, 5-step freeze 72% at 24% faster. The 5-step freeze is what the main Table 1 results use.source
- ECoT BridgeData V2 evaluation suite (real WidowX) in-distribution camera view, aggregate over 13 task groups covering spatial relations, out-of-distribution objects and out-of-distribution instructions66%aggregate success rateReported by the ECoT authors, Table 1 (66% plus or minus 3.8). On the identical tasks: OpenVLA fine-tuned on Bridge 44%, naive semantic CoT 48%, the closed 55B RT-2-X 47%, Octo 21%. 314 total trials per approach across the whole study.source
- ECoT BridgeData V2 evaluation suite (real WidowX) out-of-distribution camera view, aggregate over the same 13 task groups64%aggregate success rateReported by the ECoT authors, Table 1 (64% plus or minus 3.9). OpenVLA fine-tuned on Bridge collapses to 30%, Octo to 16%, while RT-2-X and naive CoT both reach 48%. ECoT loses only 2 points when the camera moves, which is the paper's strongest robustness claim.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
All evaluation is on a real 6-DoF WidowX arm in the BridgeData V2 setup with a single third-person camera and end-effector velocity control, 314 trials per approach across 13 task groups, run on two stations: one with the in-distribution camera view and one with an out-of-distribution view. ECoT scores 66% and 64% respectively. Additional real rollouts cover the human-correction study on the three hardest tasks and the accelerated-inference study over 25 trials.
Fine tuning it yourself
Training uses OpenVLA's codebase with reasoning strings tokenized by the Llama 2 tokenizer and predicted autoregressively before the action tokens, so no new loss or head is needed. The reasoning labels are produced by an automated pipeline and released as the embodied_features_bridge dataset, so a new robot needs that pipeline rerun on its own data. The cheapest path shown is continuing a pretrained VLA on a mixture containing about 13% ECoT data, which reached near-parity within 20k steps, a 4x compute reduction, with comparable behavior visible already at 2500 steps, a 30x reduction.
Where it helps, where it does not
Strengths
- Adds 22 points of absolute success over the identical OpenVLA base model on the in-distribution view and 34 points on the out-of-distribution view, using the same robot data and no extra demonstrations
- Beats the closed 55B RT-2-X (66% versus 47%) with a network roughly 7 times smaller
- Barely degrades when the camera is moved out of distribution, dropping from 66% to 64% where plain OpenVLA drops from 44% to 30%
- Failures become inspectable: the emitted plan, sub-task, bounding boxes and gripper position show what the policy believed before it acted
- A single natural-language human correction per episode, folded back into the reasoning chain, raised success on the three hardest tasks where the unaided policy scored 32%, and the paper reports this as a 48% improvement (it does not state whether absolute or relative)
- Reasoning ability transfers to a generalist checkpoint cheaply: continuing the released OpenVLA-7B on a mix with about 13% ECoT data nearly matched the from-scratch ECoT model within 20k steps instead of 80k
- Runs in about 5 GB with 4-bit quantization
Limits
- Speed is the core cost: 350 decoded tokens per timestep instead of 7, and the paper publishes no absolute control frequency at all, only relative speed-ups. Any real deployment budget has to be measured locally.
- The published acceleration tricks are partial fixes. Freezing the reasoning for 5 steps or running an asynchronous second instance buys 24% and 40%, and the asynchronous mode doubles inference compute.
- Evaluated on exactly one embodiment, a WidowX arm with a single third-person camera. Transfer to other robots is only shown indirectly through the OXE checkpoint, without real rollouts on those robots.
- Co-training with vision-language data did not measurably improve control performance on the evaluated tasks, only anecdotal celebrity recognition.
- The frozen-bounding-box variant, adopted for later experiments because it is faster, performs worse than the base ECoT model.
- Inherits every OpenVLA structural limit: single image, no proprioception, no action chunking.
- Repository last pushed April 2025 and the paper is at v3 from March 2025, with no newer release as of August 2026.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2407.08693
- https://arxiv.org/pdf/2407.08693
- https://embodied-cot.github.io/
- https://github.com/MichalZawalski/embodied-CoT
- https://huggingface.co/Embodied-CoT/ecot-openvla-7b-bridge
- https://huggingface.co/datasets/Embodied-CoT/embodied_features_bridge
- https://proceedings.mlr.press/v270/zawalski25a.html
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.