Seer
Shanghai AI Laboratory, China · December 2024
Shanghai AI Laboratory, with Peking University and the Chinese University of Hong Kong
- Parameters
- 316 M
- The paper states 316M total parameters with 65M trainable for Seer.
- GPU memory
- 0.6 GB
- weights at bf16, computed. Fits an 8 GB card
- Inference latency
- not published
- per action step
- Weights
- closed
- no public checkpoint
What it is
Seer is a mid-size manipulation policy built around the idea that a policy should predict what it is about to see before it predicts what to do. A foresight head forecasts future visual states, and an inverse-dynamics head then regresses the actions conditioned on that forecast, with both trained end to end. This closes the loop between the vision-pretraining line of work and the behaviour-cloning line, and it produced state-of-the-art results on CALVIN ABC-D (3.98 average length for the base model) and LIBERO-LONG (87.7 percent) at ICLR 2025. It is small by VLA standards at 316M parameters with only 65M trainable, and it is Apache-2.0. What it does not have is a language model, so instruction following stays close to the training distribution and there is no reasoning layer.
Architecture
- Backbone
- MAE-pretrained ViT-B image encoder plus a frozen CLIP ViT-B/32 text encoder, a Perceiver Resampler that compresses image tokens, and an MLP state encoder, all feeding a causal transformer. 251M of the parameters are frozen.
- Action head
- Predictive inverse dynamics. The model first forecasts future visual states with a foresight head, then an inverse-dynamics head predicts the actions conditioned on that forecast, so vision prediction and action prediction are trained end to end in one closed loop.
- Parameters
- The paper states 316M total parameters with 65M trainable for Seer. The remaining 251M are frozen, which is the MAE ViT-B vision encoder and the CLIP text encoder.
- Pretraining data
- Pre-trained on large-scale robot data, with DROID named as the pre-training source for the real-world checkpoint. The authors report that pre-training for CALVIN ABC-D takes about 40 hours on eight RTX 4090 GPUs. Inference is run in bfloat16.
- Embodiments
- CALVIN simulation benchmark, LIBERO simulation benchmark, A single real robot arm in the authors' four-task real-world suite (Flip White Bowl, Stack Cups, Wipe Board, Pick Place Close)
What hardware it needs
This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.
| Precision | Weights | Note |
|---|---|---|
| fp32 | 1.2 GB | Training master weights, rarely used for inference |
| bf16 or fp16 | 0.6 GB | The usual way these checkpoints are served |
| int8 | 0.3 GB | Quantised, expect some loss of precision on fine motions |
| int4 | 0.1 GB | Aggressive quantisation, verify success rate before trusting it |
Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.
The paper states the authors used eight RTX 4090 GPUs to pre-train and fine-tune, with about 40 hours of pre-training for CALVIN ABC-D. No per-GPU VRAM figure and no single-GPU minimum is published. Inference is configured in bfloat16.
The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.
Reported results
Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.
Simulation
Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.
- LIBERO LIBERO-LONG87.7%average success rateAuthors' own evaluation. Baselines in the same table: OpenVLA 54.0, MPI 77.3, MVP 68.2, MTACT 41.0.source
- LIBERO LIBERO-LONG, Seer trained from scratch without pre-training78.7%
- CALVIN ABC to D, long-horizon chains of 5 language instructions3.98 tasksaverage successful sequence lengthAuthors' own evaluation. Baselines in the same table: GR-1 3.06, 3D Diffuser Actor 3.27, CLOVER 3.53, SuSIE 2.69, RoboFlamingo 2.47.source
- CALVIN ABC to D, Seer trained from scratch without pre-training3.64 tasksaverage successful sequence lengthAuthors' own ablation. Isolates the contribution of pre-training at 0.34 tasks.source
Real world
Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.
- Real-world (authors' own four-task suite) Fine-tuned from the DROID pre-trained checkpoint, 100 demonstrations per task78.4%average success rateAuthors' own evaluation on their own hardware. The same policy trained from scratch reaches 60.0 percent, and the accumulated score rises from 32.8 to 39.5.source
Fine tuned tasks
No results in this category are published for this model.
On real hardware
The authors report a four-task real-world suite (Flip White Bowl, Stack Cups, Wipe Board, Pick Place Close) fine-tuned from a DROID pre-trained checkpoint with 100 demonstrations per task on 224x224 inputs. Pre-training raises the average success rate from 60.0 percent (from scratch) to 78.4 percent and the accumulated score from 32.8 to 39.5. The paper also claims generalisation to novel objects, changed lighting and changed environments under high-intensity disturbances, but reports no separate quantitative table for those conditions.
Fine tuning it yourself
Fine-tune from a released checkpoint (CALVIN ABC-D, CALVIN ABC-D Large, or the DROID pre-trained real-world checkpoint), all linked as Google Drive folders from the GitHub README. The authors used 100 demonstrations per task for real-world fine-tuning at 224x224 image resolution. The repository also ships simpleseer, a stripped-down scratch training and deployment path. Eight RTX 4090 GPUs were used for both pre-training and fine-tuning.
Where it helps, where it does not
Strengths
- State of the art on CALVIN ABC-D and LIBERO-LONG at the time of publication, with the largest margin on the long-horizon chained-instruction splits where most policies collapse.
- Only 65M of 316M parameters are trainable, so fine-tuning is far cheaper than the total size suggests.
- Apache-2.0 for both code and released checkpoints, and pre-training plus fine-tuning fit on eight RTX 4090 cards rather than datacenter GPUs.
- The pre-training benefit is quantified rather than asserted: 3.64 to 3.98 on CALVIN ABC-D and 78.7 to 87.7 on LIBERO-LONG.
Limits
- No language model in the loop. Language enters only through a frozen CLIP text encoder, so instruction following is limited to short commands close to the training distribution and there is no reasoning or replanning.
- Checkpoints are distributed via Google Drive links in the README rather than on Hugging Face, which complicates reproducible, scripted deployment.
- The paper publishes no inference latency, no control rate on real hardware and no VRAM figure, so real-time feasibility on a given robot has to be measured by the integrator.
- Real-world evidence is a four-task suite on the authors' own setup with 100 demonstrations per task. There is no cross-lab replication.
- The released Seer-Large CALVIN checkpoint scores 4.30 average length in the repository while the paper's table reports 4.28, a small but real discrepancy between the published and the shipped artifact.
Sources
Everything on this page was taken from these documents. Where they disagree with what you read here, they win.
- https://arxiv.org/abs/2412.15109
- https://arxiv.org/html/2412.15109v1
- https://github.com/InternRobotics/Seer
- https://raw.githubusercontent.com/InternRobotics/Seer/main/README.md
- https://openreview.net/forum?id=meRCKuUpmc
- https://iclr.cc/virtual/2025/poster/28455
Entry last checked 2026-08-11.
Try a policy on a real arm
A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.