Cosmos Reason as a Planner for Manipulation

Put a vision-language model in front of the robot policy, let it work out what to do next, and let a smaller network do the moving. That is the planner and actor split, and NVIDIA ships a model built for the planner half. Cosmos Reason is a VLM post-trained on physical common sense and embodied reasoning, which NVIDIA describes as a planning model that reasons about what steps an embodied agent might take next. It emits text. It never emits joint angles.

A second, quieter use of the same model causes most of the confusion here. GR00T N1.7 does not sit behind a Cosmos Reason planner; it has Cosmos-Reason2-2B fused in as its vision-language-action model backbone. Two architectures, two sets of failure modes. If you fine-tune GR00T N1.7 on an SO-100 today, you are already running Cosmos Reason without having built a planner. This article covers both, then where the split breaks on a small arm.

What Cosmos Reason actually is

The line starts with the Cosmos-Reason1 paper, submitted on 18 March 2025. It introduced Cosmos-Reason1-7B and Cosmos-Reason1-56B, and the two are not the same architecture: the 7B takes Qwen2.5-VL as its pretrained model and keeps a dense Transformer backbone, while the 56B pairs an InternViT-300M-V2.5 vision encoder with a Nemotron-H hybrid Mamba-MLP-Transformer backbone. Both were trained in two stages, Physical AI supervised fine-tuning then Physical AI reinforcement learning, against ontologies the paper defines for space, time and physics.

Read those numbers as a warning, not a sales pitch. The 56B reaches 60.2 average on the physical common sense benchmark and 63.7 on embodied reasoning, the latter a gain of 10.2 points over its own backbone. On the individual embodied splits it scores 80.0 on RoboVQA, 65.0 on BridgeData V2, 66.2 on RoboFail, 57.8 on HoloAssist and 47.6 on Agibot. A model that gets under half of Agibot right will occasionally hand your arm a nonsense subtask.

Those counts are the BF16 tensor totals Hugging Face reports per repository; NVIDIA's own 32B card rounds to 32,000,000,000, and the repository GPU table lists only the 2B and the 8B. Cosmos Reason 2 is the version you would deploy today, a straight post-train of Qwen3-VL-Instruct checkpoints, so it inherits the Qwen3-VL tooling including the vLLM reasoning parser. It takes up to 256K input tokens, wants video at 4 fps to match its training setup, and wants an output budget of 4096 tokens or more so the chain of thought is not cut off. Those last two numbers decide whether this can plan for your robot.

The Cosmos Reason 2 card scores all three sizes against the Qwen3-VL checkpoint each was post-trained from. The two ends of that range show where the physical post-training pays and where it does not.

Read the 2B column carefully, because 2B is the size you can afford to run next to a hobby arm. Embodied reasoning jumps from 48.85 to 58.03, a real gain. ERQA is identical at 37.75, Where2Place identical at 32.00, and at 32B ERQA is worse than the base Qwen3-VL checkpoint. The post-training buys embodied narration, not spatial placement, and placement is what a manipulation planner keeps needing.

Planner and actor: what the split actually is

A hierarchical stack has three layers, and the boundaries move depending on which paper you read. The version that matters for a desk robot looks like this.

The reason to bother is not that the planner is smarter. It is that the actor's training data is expensive. A policy trained on 50 episodes of one long task learns that one long task. The same 50 episodes cut into three named subtasks give you a policy you can recombine, with the recombination logic in text you can log and fix without touching a GPU.

Where the idea comes from

Two camps fall out of that table. SayCan, Hi Robot and the Gemini Robotics-ER stack keep the planner separate. ECoT and Pi0.5 fold the reasoning into the actor. GR00T N1.7 is a third case that is easy to misfile: it swallows a reasoning VLM without keeping the reasoning. The Hi Robot project page has the clearest evidence for the separate-planner camp: 74 percent instruction accuracy on table bussing against 36 for a flat VLA and 35 for a GPT-4o high-level baseline, and 83 against 34 and 13 on sandwich making. Those are kitchen-scale tasks on dual-arm and mobile platforms, so read them as direction, not forecast.

The other Cosmos Reason: backbone, not planner

Train GR00T on AY-Robots and you are already shipping Cosmos Reason weights to the GPU. The Isaac-GR00T repository states the change plainly: the VLM backbone moved from the vendored Eagle model, nvidia/Eagle-Block2A-2B-v2, to nvidia/Cosmos-Reason2-2B via Qwen3-VL. This is not a planner. There is no prompt budget, no chain of thought at rollout time, and no place in the CLI to ask it a question: the VLM is an encoder feeding a diffusion transformer head that denoises continuous actions.

One more naming trap. NVIDIA uses Cosmos for three things and only one of them is a planner. Cosmos Reason is the VLM. Cosmos Predict is the video world model. Cosmos Policy is a robot policy post-trained from Cosmos-Predict2 that reports 98.5 percent average success on LIBERO and 67.1 percent on RoboCasa; it is an actor, not a planner, and it has its own entry at /arena/cosmos-policy.

So GR00T N1.7 follows language the way a policy does, not the way a planner does: a short string plus pixels in, a chunk of joint targets out, and no account of why. For a readable plan you still need a second model. Compare Pi0.5, whose recipe co-trains semantic subtask prediction with low-level actions, or read our write-up of flow-matching policies with VLM initialization.

Building the planner loop yourself

Nothing below is exotic: two servers and a loop. The commands are from the cosmos-reason2 repository and the Isaac-GR00T repository as they stood on 2026-08-23.

1. 1
   Serve the planner

   Cosmos-Reason2 is in transformers 4.57.0 and later; for serving, NVIDIA recommends vllm 0.11.0 or newer. Log in to Hugging Face first, because the 2B repository is gated. First startup takes a couple of minutes for model loading and CUDA graph compilation.

   bash

   git clone https://github.com/nvidia-cosmos/cosmos-reason2.git
   cd cosmos-reason2
   uv sync --extra cu128
   source .venv/bin/activate
   uvx hf auth login
   
   vllm serve nvidia/Cosmos-Reason2-2B \
     --allowed-local-media-path "$(pwd)" \
     --max-model-len 16384 \
     --media-io-kwargs '{"video": {"num_frames": -1}}' \
     --reasoning-parser qwen3 \
     --port 8000

2. 2
   Check that it reasons about your scene, not a demo scene

   The repository ships an embodied reasoning prompt. Point it at a real frame from your own wrist camera before you build anything on top. If the model cannot describe your workspace correctly from a still image, it will not plan for it from video.

   bash

   cosmos-reason2-inference online -v --port 8000 \
     -i prompts/embodied_reasoning.yaml \
     --reasoning --images /path/to/your_wrist_frame.png

3. 3
   Freeze the instruction vocabulary

   The step everyone skips. Your actor understands the task strings from the dataset it was fine-tuned on and nothing else. Read them out and make that list the only thing the planner may emit. The modality config decides which key those strings arrive under; the SO100 example config declares a single language key.

   bash

   # GR00T LeRobot layout: language task descriptions live in meta/tasks.jsonl
   jq -r '.task' my_dataset/meta/tasks.jsonl | sort -u

4. 4
   Serve the actor

   Isaac-GR00T ships a policy server and a matching client. Start it on the machine with the checkpoint; the client is a thin RPC wrapper.

   bash

   uv run python gr00t/eval/run_gr00t_server.py \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path /path/to/checkpoint \
       --device cuda:0 \
       --host 0.0.0.0 --port 5555

5. 5
   Write the loop, and keep the two rates apart

   The actor runs every step. The planner runs every few seconds. Mixing the two rates is the single most common design error here. The key you write the instruction into is whatever your modality config declares as its language modality: the SO100 config shipped with Isaac-GR00T declares annotation.human.task_description.

   python

   from gr00t.policy.server_client import PolicyClient
   
   # exactly the strings that came out of meta/tasks.jsonl
   SUBTASKS = [
       "pick up the red cube",
       "put the cube in the bowl",
       "move the arm back to rest",
   ]
   
   # the language modality key declared in examples/SO100/so100_config.py
   LANG_KEY = "annotation.human.task_description"
   
   policy = PolicyClient(host="localhost", port=5555)
   plan = SUBTASKS[0]
   step = 0
   
   while not done:
       obs = robot.observe()             # video.* frames + state.* joints
   
       # seconds, not milliseconds
       if step % REPLAN_EVERY_N_STEPS == 0:
           plan = ask_planner(obs, SUBTASKS)   # must return a member of SUBTASKS
   
       obs[LANG_KEY] = plan
   
       action, info = policy.get_action(obs)
       robot.execute(action)
       step += 1

6. 6
   Prove the actor follows language before you trust the planner

   Open-loop evaluation compares predicted actions against ground truth and writes a plot with MSE metrics. Run it once per instruction string. If the same start state gives the same trajectory whichever string you pass, the language channel is dead and no planner will revive it.

   bash

   uv run python gr00t/eval/open_loop_eval.py \
       --dataset-path /path/to/dataset \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path /path/to/checkpoint \
       --traj-ids 0 \
       --execution-horizon 16

Where it breaks

1. The latency budget does not close

This is the objection that kills most planner projects, and it is arithmetic rather than opinion.

A chain of thought NVIDIA sizes at four thousand tokens is not a component of a 152 ms loop. The planner sits beside the control loop, not inside it. And because the actor predicts chunks rather than single steps, an instruction change lands at the next chunk boundary, not the next servo command: GR00T N1.7 expanded its action horizon from 16 to 40 and executes as many of those as --execution-horizon says. See action chunking and inference latency if that distinction is new.

2. The actor may not be listening at all

A planner is a language emitter. If the actor has no language input, the planner is decoration. That is not a subtlety in LeRobot, it is visible in the configuration files: the ACT config has no tokenizer field and no language field, because ACT is vision plus proprioception, and its defaults are chunk_size 100 and n_action_steps 100. The SmolVLA config by contrast carries a tokenizer_max_length of 48 and a SmolVLM2-500M-Video-Instruct backbone, so a carefully engineered instruction gets truncated at 48 tokens.

3. Nothing reports failure upward

The actor returns actions. It does not return whether the cube is in the gripper. You write the termination check yourself: a step budget, a condition on joint state, or a second planner call asking whether the subtask is done, which costs a full chain of thought. A planner watching 4 fps video sees the aftermath of a slip a quarter of a second late. If the gripper is the unreliable part, fix that first: gripper does not close is a data problem, not a planning problem.

4. Two models, two GPUs, two bills

Cosmos-Reason2-2B needs 24 GB and the 8B needs 32 GB, on hardware NVIDIA has validated only on Hopper and Blackwell; Jetson AGX Thor is listed for Transformers inference with vLLM marked as coming. The actor tiers are fixed too: GR00T N1.7, GR00T N1.5 and Pi0.5 need an A100 80 GB or H100 80 GB, while SmolVLA and ACT fit on an RTX 4090 or any 24 GB card. A run costs roughly 4 to 12 USD on the big tier, 1 to 3 USD on the small one. The planner is a third bill, and the pod's idle watchdog does not cover a process the platform does not run. See pricing.

5. You cannot A/B it cleanly

To claim the planner improved anything you have to beat run-to-run noise. Isaac-GR00T warns of 5 to 6 percent variance between runs from non-deterministic image augmentations, and its fine-tune entry point is a tyro CLI with no seed flag, so GR00T runs are not bit-for-bit reproducible. LeRobot's default seed is 1000, and on AY-Robots the Pi0.5, SmolVLA and ACT forms expose a seed while the GR00T forms do not. Repeat every configuration three times, or report nothing.

Two ways to get to a working actor

The planner half is yours either way. What differs is how much of the actor half you build. Same goal both ways: a fine-tuned language-conditioned policy on an SO-100 that a planner can drive.

git clone https://github.com/NVIDIA/Isaac-GR00T.git
cd Isaac-GR00T
uv sync --python 3.12
uv run huggingface-cli login   # Cosmos-Reason2-2B is gated

# LeRobot v3 datasets must be converted down first
cd scripts/lerobot_conversion
uv venv && source .venv/bin/activate
uv pip install -e .
python convert_v3_to_v2.py --repo-id <DATASET_REPO_ID>
cd ../..

CUDA_VISIBLE_DEVICES=0 uv run python \
    gr00t/experiment/launch_finetune.py \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path demo_data/cube_to_bowl_5 \
    --embodiment-tag NEW_EMBODIMENT \
    --modality-config-path examples/SO100/so100_config.py \
    --num-gpus 1 \
    --output-dir /tmp/test_finetune \
    --max-steps 2000 \
    --global-batch-size 32 \
    --dataloader-num-workers 4

What the platform helps with, and what it does not

AY-Robots covers the actor half end to end: teleoperation from the browser, a desktop client that records LeRobot datasets from a teleop session, a training form that rents the GPU, and an inference pod that serves the result. The arena compares 85 VLA models across 332 benchmark results, every value linked to its paper or model card. The same operations reach a terminal at /cli and agents at /mcp. For background, see our overview of vision-language-action models and how RT-2 transfers web knowledge to robot control.

What it does not do: host a planner, expose a planning API, or make a remote actor fast. If your task needs reactive motion, the actor belongs next to the servos; the planner can sit anywhere as long as you call it every few seconds. And if your actor is ACT, a planner is not a design choice, it is a no-op.

If you are going to try this, do it in this order

1. Record with varied instruction strings from the start. Two or three named subtasks per episode beat one long unlabelled one.
2. Fine-tune one actor and prove open loop that two instructions from the same start state give two different trajectories.
3. Only then serve Cosmos-Reason2-2B and constrain its output to the exact strings in meta/tasks.jsonl.
4. Replan on a timer measured in seconds, never per action step.
5. Repeat each configuration three times before you believe any difference, because run-to-run variance is 5 to 6 percent on GR00T alone.