SAI

Seer

Shanghai AI Laboratory, China · December 2024

Shanghai AI Laboratory, with Peking University and the Chinese University of Hong Kong

Action policyClosed weightsAlso written Predictive Inverse Dynamics Model, PIDM, Seer-Base
Parameters
316 M
The paper states 316M total parameters with 65M trainable for Seer.
GPU memory
0.6 GB
weights at bf16, computed. Fits an 8 GB card
Inference latency
not published
per action step
Weights
closed
no public checkpoint

What it is

Seer is a mid-size manipulation policy built around the idea that a policy should predict what it is about to see before it predicts what to do. A foresight head forecasts future visual states, and an inverse-dynamics head then regresses the actions conditioned on that forecast, with both trained end to end. This closes the loop between the vision-pretraining line of work and the behaviour-cloning line, and it produced state-of-the-art results on CALVIN ABC-D (3.98 average length for the base model) and LIBERO-LONG (87.7 percent) at ICLR 2025. It is small by VLA standards at 316M parameters with only 65M trainable, and it is Apache-2.0. What it does not have is a language model, so instruction following stays close to the training distribution and there is no reasoning layer.

Architecture

Backbone
MAE-pretrained ViT-B image encoder plus a frozen CLIP ViT-B/32 text encoder, a Perceiver Resampler that compresses image tokens, and an MLP state encoder, all feeding a causal transformer. 251M of the parameters are frozen.
Action head
Predictive inverse dynamics. The model first forecasts future visual states with a foresight head, then an inverse-dynamics head predicts the actions conditioned on that forecast, so vision prediction and action prediction are trained end to end in one closed loop.
Parameters
The paper states 316M total parameters with 65M trainable for Seer. The remaining 251M are frozen, which is the MAE ViT-B vision encoder and the CLIP text encoder.
Pretraining data
Pre-trained on large-scale robot data, with DROID named as the pre-training source for the real-world checkpoint. The authors report that pre-training for CALVIN ABC-D takes about 40 hours on eight RTX 4090 GPUs. Inference is run in bfloat16.
Embodiments
CALVIN simulation benchmark, LIBERO simulation benchmark, A single real robot arm in the authors' four-task real-world suite (Flip White Bowl, Stack Cups, Wipe Board, Pick Place Close)

What hardware it needs

This is the section most people came for, so it is worth being precise about which numbers are measured and which are arithmetic.

Computed from 316 M parameters. Weights only, so treat it as a floor: activations, image encoder intermediates and the CUDA context come on top, in practice 30 to 50 percent more for inference.
PrecisionWeightsNote
fp321.2 GBTraining master weights, rarely used for inference
bf16 or fp160.6 GBThe usual way these checkpoints are served
int80.3 GBQuantised, expect some loss of precision on fine motions
int40.1 GBAggressive quantisation, verify success rate before trusting it

Smallest real card that fits the bf16 weights plus a 40 percent runtime allowance: 8 GB. That is an estimate for inference, not for fine tuning. Full fine tuning also holds optimiser state and gradients, which typically costs several times the weights unless you use LoRA or a comparable adapter.

The paper states the authors used eight RTX 4090 GPUs to pre-train and fine-tune, with about 40 hours of pre-training for CALVIN ABC-D. No per-GPU VRAM figure and no single-GPU minimum is published. Inference is configured in bfloat16.

The authors publish no memory or latency figure for this model. Everything above is computed from the parameter count.

Reported results

Grouped by what the evaluation actually asked. Values from different suites are not comparable with each other, so each one keeps its suite and split.

Simulation

Reproducible benchmark suites. Everyone runs the same episodes, so the numbers can be compared inside a suite.

  • LIBERO LIBERO-LONG
    87.7%
    average success rateAuthors' own evaluation. Baselines in the same table: OpenVLA 54.0, MPI 77.3, MVP 68.2, MTACT 41.0.source
  • LIBERO LIBERO-LONG, Seer trained from scratch without pre-training
    78.7%
    average success rateAuthors' own ablation, 9 points below the pre-trained model.source
  • CALVIN ABC to D, long-horizon chains of 5 language instructions
    3.98 tasks
    average successful sequence lengthAuthors' own evaluation. Baselines in the same table: GR-1 3.06, 3D Diffuser Actor 3.27, CLOVER 3.53, SuSIE 2.69, RoboFlamingo 2.47.source
  • CALVIN ABC to D, Seer trained from scratch without pre-training
    3.64 tasks
    average successful sequence lengthAuthors' own ablation. Isolates the contribution of pre-training at 0.34 tasks.source

Real world

Rollouts on physical hardware. The setups differ, so read these as evidence, not as a ranking.

  • Real-world (authors' own four-task suite) Fine-tuned from the DROID pre-trained checkpoint, 100 demonstrations per task
    78.4%
    average success rateAuthors' own evaluation on their own hardware. The same policy trained from scratch reaches 60.0 percent, and the accumulated score rises from 32.8 to 39.5.source

Fine tuned tasks

No results in this category are published for this model.

On real hardware

The authors report a four-task real-world suite (Flip White Bowl, Stack Cups, Wipe Board, Pick Place Close) fine-tuned from a DROID pre-trained checkpoint with 100 demonstrations per task on 224x224 inputs. Pre-training raises the average success rate from 60.0 percent (from scratch) to 78.4 percent and the accumulated score from 32.8 to 39.5. The paper also claims generalisation to novel objects, changed lighting and changed environments under high-intensity disturbances, but reports no separate quantitative table for those conditions.

Fine tuning it yourself

Fine-tune from a released checkpoint (CALVIN ABC-D, CALVIN ABC-D Large, or the DROID pre-trained real-world checkpoint), all linked as Google Drive folders from the GitHub README. The authors used 100 demonstrations per task for real-world fine-tuning at 224x224 image resolution. The repository also ships simpleseer, a stripped-down scratch training and deployment path. Eight RTX 4090 GPUs were used for both pre-training and fine-tuning.

Where it helps, where it does not

Strengths

  • State of the art on CALVIN ABC-D and LIBERO-LONG at the time of publication, with the largest margin on the long-horizon chained-instruction splits where most policies collapse.
  • Only 65M of 316M parameters are trainable, so fine-tuning is far cheaper than the total size suggests.
  • Apache-2.0 for both code and released checkpoints, and pre-training plus fine-tuning fit on eight RTX 4090 cards rather than datacenter GPUs.
  • The pre-training benefit is quantified rather than asserted: 3.64 to 3.98 on CALVIN ABC-D and 78.7 to 87.7 on LIBERO-LONG.

Limits

  • No language model in the loop. Language enters only through a frozen CLIP text encoder, so instruction following is limited to short commands close to the training distribution and there is no reasoning or replanning.
  • Checkpoints are distributed via Google Drive links in the README rather than on Hugging Face, which complicates reproducible, scripted deployment.
  • The paper publishes no inference latency, no control rate on real hardware and no VRAM figure, so real-time feasibility on a given robot has to be measured by the integrator.
  • Real-world evidence is a four-task suite on the authors' own setup with 100 demonstrations per task. There is no cross-lab replication.
  • The released Seer-Large CALVIN checkpoint scores 4.30 average length in the repository while the paper's table reports 4.28, a small but real discrepancy between the published and the shipped artifact.

Sources

Everything on this page was taken from these documents. Where they disagree with what you read here, they win.

Entry last checked 2026-08-11.

Try a policy on a real arm

A physical SO-100 is online and free to drive in the browser, and five of the models in this arena can be fine tuned on a dataset you record with your own.