Run GR00T Inference Without a Local GPU

A Raspberry Pi is enough to drive an SO-100 over a serial bus and pull frames off two USB cameras. It is not enough to run a three billion parameter vision-language-action model: NVIDIA's README puts GR00T N1.7 inference at one GPU with 16 GB or more of VRAM. To see what your fine-tuned GR00T N1.7 checkpoint does on the arm without buying a card, put the policy on a rented cloud GPU, keep the robot loop on the machine with the USB ports, and send observations and action chunks over the network.

It works, it is not free, and the price is not evenly spread across tasks. Below: NVIDIA's own policy server, lerobot's async stack, the arithmetic that says in advance whether your uplink is fast enough, and the platform route. All of it checked against the Isaac-GR00T main branch (N1.7 GA) and lerobot 0.6.1 on 23 August 2026.

Why the policy will not fit on the robot machine

Two of the five policies AY-Robots can train run on a workstation card, three do not. The inference latency column below is per action step, and that is the number which competes with your network round trip.

Read that as a decision, not trivia. ACT at 20 ms per step runs on the robot machine and you never think about it again. Pi0.5 at 485 ms has spent a third of a second before a packet leaves your building. The policy comparison and GR00T N1.7 vs Pi0.5 add the accuracy side.

The two client-server stacks that already exist

Isaac-GR00T ships a ZeroMQ request-reply server; lerobot ships a gRPC server built around asynchronous inference. Both accept a GR00T checkpoint. lerobot's supported policy list in async_inference/constants.py is act, smolvla, diffusion, tdmpc, vqbet, pi0, pi05 and groot; its robot list is so100_follower, so101_follower, bi_so_follower and omx_follower.

The serialization row matters more than it looks. GR00T's MsgSerializer refuses object-dtype ndarray payloads in both directions, because msgpack_numpy would otherwise hand them to pickle. lerobot pickles instead: policy_server.py calls pickle.loads on request data, robot_client.py pickles the observation it sends. Defensible on a trusted LAN, indefensible once the port is reachable from the internet.

Route A: NVIDIA's own GR00T policy server

This is the path NVIDIA documents for SO-100 and SO-101 hardware, and the one to use if your checkpoint came out of examples/finetune.sh with --embodiment-tag NEW_EMBODIMENT. The steps add what the upstream README leaves out: getting the port to the robot without exposing it to everyone else.

1. 1
   Install GR00T on the rented GPU box

   Submodules are required, and git-lfs must exist before the clone or the parquet files in demo_data arrive as pointers. flash-attn and TensorRT come with the default install. The trap on a fresh pod image: torchcodec 0.8.0 is the only supported video backend and loads FFmpeg 4 to 7 only. Ubuntu 25.10 and 26.04 ship FFmpeg 8, so GR00T fails with Could not load libtorchcodec. Install an FFmpeg below 8 and put its libraries on LD_LIBRARY_PATH.

   bash

   sudo apt install git-lfs && git lfs install
   curl -LsSf https://astral.sh/uv/install.sh | sh
   sudo apt-get update && sudo apt-get install -y ffmpeg
   
   git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T
   cd Isaac-GR00T
   uv sync --python 3.12
   uv run python -c "import gr00t; print('GR00T installed successfully')"

2. 2
   Authenticate against the gated backbone

   Every GR00T N1.7 checkpoint, including your own fine-tune, loads the gated nvidia/Cosmos-Reason2-2B on first use. Request access on the model page and log in on the pod, or loading fails with a GatedRepoError.

   bash

   uv run huggingface-cli login
   # or:  export HF_TOKEN=<your_token>

3. 3
   Start the policy server

   Point --model-path at your checkpoint directory; on that path the server ignores --modality-config-path, which is read only on the replay path. Omit --model-path and pass --dataset-path plus --execution-horizon instead for a ReplayPolicy that replays recorded actions, the cheapest way to prove the wiring works.

   bash

   uv run python gr00t/eval/run_gr00t_server.py \
     --model-path /workspace/so100_finetune/checkpoint-10000 \
     --embodiment-tag NEW_EMBODIMENT \
     --device cuda:0 \
     --host 127.0.0.1 --port 5555

4. 4
   Tunnel port 5555 to the robot machine

   Bind to loopback, as above, and carry the port over SSH or a WireGuard-style mesh. That supplies the encryption and authentication the ZeroMQ socket does not, for about a millisecond.

   bash

   # on the robot machine
   ssh -N -L 5555:127.0.0.1:5555 root@<pod-host> -p <pod-ssh-port>
   
   # sanity check that something answers
   nc -vz 127.0.0.1 5555

5. 5
   Run the robot client next to the servos

   The client needs its own uv environment: it wants lerobot's robot drivers, not the training stack. eval_so100.py imports so100_follower, so101_follower and koch_follower, so pass the --robot.type matching your arm (the upstream README uses so101_follower). Camera keys must match training: the adapter reads exactly front and wrist, and swapping them shows the policy the wrong view.

   bash

   cd gr00t/eval/real_robot/SO100
   uv sync
   uv pip install --no-deps -e ../../../../
   
   uv run --no-sync python eval_so100.py \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=orange_follower \
     --robot.cameras="{ front: {type: opencv, index_or_path: 6, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
     --policy_host=127.0.0.1 \
     --policy_port=5555 \
     --lang_instruction="pick up the red block and put it in the bin"

Route B: lerobot async inference

lerobot solves a different problem. Instead of blocking the robot while the model thinks, the client keeps stepping through the queue it already has while the server computes the next chunk. This is action chunking taken further, the asynchronous stack introduced with SmolVLA. It works with a GR00T checkpoint too.

# GPU machine
pip install -e ".[async]"
python -m lerobot.async_inference.policy_server \
     --host=127.0.0.1 \
     --port=8080

# robot machine, after tunnelling 8080
python -m lerobot.async_inference.robot_client \
    --server_address=127.0.0.1:8080 \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=follower_so100 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}}" \
    --task="pick up the red block and put it in the bin" \
    --policy_type=groot \
    --pretrained_name_or_path=<user>/my_groot_finetune \
    --policy_device=cuda \
    --actions_per_chunk=50 \
    --chunk_size_threshold=0.5 \
    --debug_visualize_queue_size=True

The server starts empty: it does not know which policy it serves until the client's first handshake tells it, which is convenient on a rented pod. The two knobs that decide whether the arm moves smoothly are actions_per_chunk and chunk_size_threshold (the lerobot docs call the second one g, after the SmolVLA paper), and the documented values and the shipped values do not agree.

The arithmetic that decides whether your link is fast enough

People skip this and then spend a day on a policy that freezes mid-motion. It takes two minutes and it is almost always decisive.

The commented observation dict in NVIDIA's eval_so100.py says what goes on the wire: two arrays of shape (480, 640, 3) in uint8, six joint floats, a language string. That is 921,600 bytes per frame, 1,843,200 bytes for two cameras, about 14.7 Mbit, and neither stack JPEG-compresses it. The chunk coming back is a few dozen steps of 6 floats. Your upload decides everything, not your download.

The budget you have to fit inside

The GR00T SO-100 client is synchronous: it calls policy.get_action(obs), executes the first action_horizon steps of the chunk at 30 FPS, then calls again. Chunk size and horizon are different numbers: NVIDIA's deployment guide recommends an action chunk size of 16, at least 32 when combined with real-time chunking, while eval_so100.py ships an execution horizon of 8. Eight steps at 30 FPS is 267 ms of motion per call, and everything else has to fit inside that.

observation upload   14.7 Mbit / 100 Mbit/s   = 147 ms
network round trip                            =  30 ms
model inference (AY-Robots figure, N1.7)      = 152 ms
action chunk return + deserialize             =  ~2 ms
                                                -------
total per call                                  331 ms

budget at action_horizon = 8   ->  267 ms   FAIL, arm pauses ~64 ms per chunk
budget at action_horizon = 16  ->  533 ms   fits, with headroom
budget at action_horizon = 32  -> 1067 ms   fits, observations now ~1 s stale

Raising the horizon is the blunt fix and not a free one: the arm acts on an observation that is now old. The principled fix is real-time chunking, which computes the next chunk while the current one runs, freezes the actions guaranteed to execute and inpaints the rest; the RTC paper reports it as robust to inference delay with no retraining. Check where that stands first. NVIDIA marks RTC experimental, a low-level model primitive reachable through action_head.get_action(..., options={"rtc_overlap_steps": ..., "rtc_frozen_steps": ...}), not wired into Gr00tPolicy or the server-client path, where options is unused, with no tests and no example. Over a policy server you get asynchronous execution, not RTC.

Measure it before you trust it

Every number above is a prediction. Four commands turn it into a measurement, worth running before committing a pod-hour to a task that was never going to work.

1. 1
   Get the raw round trip

   Against the pod, not a CDN. Watch the deviation as closely as the mean: jitter makes an arm stutter, not average latency.

   bash

   ping -c 50 <pod-host>
   # the mdev column is the number that predicts stutter

2. 2
   Measure the uplink you have, not the one you pay for

   Residential upload is usually a fraction of download, and it is the number in the bandwidth table above.

   bash

   # on the pod
   iperf3 -s
   
   # on the robot machine, -R omitted so this measures upload
   iperf3 -c <pod-host> -t 30

3. 3
   Read the client's own latency log

   The lerobot robot client logs server-to-client latency and deserialization time for every chunk. On route B you need no external tooling.

   text

   Received action chunk for step #240 | Latest action: #232 |
     Incoming actions: 240:289 |
     Network latency (server->client): 187.44ms |
     Deserialization time: 3.10ms

4. 4
   Watch the action queue drain

   Pass --debug_visualize_queue_size=True and the client plots queue size at runtime. If it repeatedly hits zero you are out of budget: lower fps, raise actions_per_chunk, or raise chunk_size_threshold so observations go out more often.

   bash

   python -m lerobot.async_inference.robot_client \
       ... \
       --debug_visualize_queue_size=True

What remote inference is actually good for

If a remote run stutters at the same point in every episode, the network is probably not the cause. A policy hesitating at the same joint angle every time is usually a data problem; see the failure-mode pages, in particular a policy that only works in one setup and loss falls but the policy does nothing.

Doing it yourself vs doing it on AY-Robots

What a remote inference session costs

Two numbers matter: the hourly rate of the card, and how long you leave it running. The first is published; the second surprises people.

Those rates were read from Runpod's pricing page on 23 August 2026, and spot markets move. AY-Robots quotes a whole run instead: 3 to 6 hours at 1.20 to 2.00 USD per hour on the A100 or H100 tier, about 4 to 12 USD for a GR00T or Pi0.5 run; 2 to 5 hours at 0.30 to 0.60 USD per hour on the 24 GB tier, 1 to 3 USD for SmolVLA or ACT. An inference session beats a training run on cost only if you stop it, which is what the idle watchdog is for. See the billing docs and the pricing page.

If you would rather not have the network in the loop

Remote inference solves a hardware problem and creates a latency problem. Sometimes the better answer is a policy that fits the hardware you have.

• ACT, roughly 80 M parameters and 20 ms per action step, 50 episodes minimum, any 24 GB card. On a repetitive single-task setup it frequently beats a remote 3 B model, because it never waits for a packet.
• SmolVLA, roughly 450 M parameters and 245 ms per action step, 30 episodes minimum. It keeps the language conditioning ACT lacks, and the lerobot docs put it at about 2 GB at inference time against roughly 14 GB for PI0.
• ACT vs GR00T N1.7 for the accuracy half of the trade.

There is also a middle path: train in the cloud, evaluate locally. Fine-tuning needs the 80 GB card and does not care about latency, so training GR00T N1.7 on an SO-100 remotely is uncontroversial. Only the evaluation loop has a real-time constraint; the training docs and model-and-arm matrix cover that half.

Frequently asked questions

The short version

Remote inference for a 3 B policy is a solved engineering problem with an unsolved physics problem attached. The engineering is two commands and an SSH tunnel. The physics is that a 1.8 MB observation has to reach a GPU in another country and come back before the arm runs out of actions. Do the arithmetic before renting anything, pick a task that tolerates a stale observation, and raise the execution horizon rather than hoping the link improves.

If you have not recorded a dataset yet, record your first dataset and the SO-100 setup guide come first, and the LeRobot dataset format entry explains what the recorder writes. Background is in vision-language-action models and the flow-matching policy work; the arena entry links every benchmark number to a source.