Edge vs Cloud Inference for Robots: The Honest Latency Math

A trained policy is a function from camera frames plus joint states to the next few joint targets. Deciding where that function runs is a deployment choice, and it is the choice that most often turns a checkpoint with a decent validation curve into an arm that hesitates, overshoots and drops the object. The honest version of the decision is arithmetic, not preference.

This page does that arithmetic with published numbers: what each policy costs per action step, what the same model costs on a Jetson instead of an H100, what a round trip adds on top, and which manipulation tasks survive it. Where the answer is run it next to the servos, it says so, including for the models this platform only serves from the cloud. Inference latency is the term, but the thing you actually care about is whether the arm is ever waiting.

Start with the control period, not with the network

Before comparing edge against cloud, write down the number you are trying to hit. A loop running at f hertz gives you 1/f seconds per step: 33 ms at 30 Hz, 20 ms at 50 Hz. The Pi0 paper reports controlling robots at frequencies of up to 50 Hz for dexterous tasks such as laundry folding, which is a 20 ms budget for everything. The five vision-language-action models you can train here have very different appetites. These are the figures published on the policies page, per action step:

Read the third column again. Pi0.5 at 485 ms per step gives you roughly two model calls per second. ACT at 20 ms gives you fifty. If you are running Pi0.5, a 40 ms network round trip adds eight percent to your loop and you will not notice. If you are running ACT, the same 40 ms triples it and the arm becomes visibly hesitant. The model you picked, not the network you have, decides whether cloud inference is a footnote or a redesign.

What the same model costs on edge hardware

NVIDIA publishes a benchmark table for the GR00T N1.7 inference pipeline across its own hardware. It is the cleanest edge-versus-datacenter comparison available for a real VLA, because it is the same model, the same export path and the same measurement on every row. End-to-end latency there covers data processing, the vision-language backbone and the diffusion action head, at 4 denoising steps with a single camera.

Two things fall out of that table. The datacenter card is about 5.4 times faster than the Jetson Orin on the identical checkpoint (27.9 ms against 150.9 ms). And TensorRT buys much less on the Jetson platforms than on the discrete GPUs: 1.28x on DGX Spark and 1.40x on AGX Thor, against 3.08x on the H100. The edge modules are memory-bandwidth-bound, and compiling the graph does not create memory bandwidth.

The bandwidth gap is the whole story. NVIDIA lists the AGX Orin 64 GB at 204.8 GB/s and the Orin Nano Super at 102 GB/s. VLA-Perf puts the LPDDR on Jetson Thor at 270 GB/s against 1 TB/s for GDDR on an RTX 4090 and 8 TB/s for HBM on a B100, and its roofline analysis finds that on Jetson Thor even the vision encoder and the language backbone become memory-bound, while on every discrete GPU they are compute-bound. That is a different bottleneck, not a slower version of the same one, which is why kernel-level optimisation pays back so differently on the two sides.

The case for keeping it on the robot

There is a serious counter-position and it deserves to be stated fairly. Google DeepMind shipped Gemini Robotics On-Device in June 2025 specifically so that the whole model runs on the robot, describing it as operating independent of a data network and, through the accompanying SDK, adaptable to new tasks with as few as 50 to 100 demonstrations. If the model is small enough for the board, the network question disappears completely. The catch is that this is a property of the model rather than of the deployment: you do not get it by moving a 3 B parameter policy onto a Jetson, you get it by using a policy that was designed for one.

What a public-internet round trip actually adds

There are three latencies stacked on top of each other and people usually only think about the third. The last mile from your bench to your provider's first hop. The transit from there to the region the GPU sits in. And the queueing, retransmission and head-of-line blocking that happens whenever either of those is busy.

For the last mile there is measured public data. The FCC's Measuring Broadband America programme reports idle round-trip times from real consumer connections to SamKnows test servers hosted at ten major public internet exchange points, which the report itself notes may be geographically distant from the communities an ISP serves. In the Twelfth Report, covering measurements taken in September 2021, the median idle latency ranged from 7 ms to 25 ms overall, breaking down to 7 ms to 13 ms for fibre ISPs, 12 ms to 22 ms for cable and 22 ms to 36 ms for DSL, with one DSL service measuring 33 ms. That is your floor before a packet has travelled anywhere interesting.

VLA-Perf is explicit that latency to cloud servers varies substantially with physical distance and routing topology, which is why it models two cloud cases rather than one. Feed those parameters into a B100 serving Pi0 synchronously and you get total end-to-end latencies of 3.3 ms over 10G Ethernet (301.4 Hz), 23.4 ms over a wired link into a fast cloud region (42.8 Hz), and 273.4 ms over 4G into a slow cloud region (3.7 Hz). Same model, same GPU, an 81-fold spread in achievable control rate purely from where the packets go. Those are roofline figures, so treat them as the best case each link could support rather than as something you will measure.

There is a fourth cost that only shows up on the first call: connection setup. A fresh TCP connection costs a round trip before any payload moves, and a TLS handshake on top of it costs one or two more, which is why the client in every one of these stacks keeps one long-lived connection open instead of issuing a request per step. LeRobot uses gRPC and reports it as roughly 5x faster than a comparable REST API; openpi uses a websocket; Isaac-GR00T holds a persistent ZMQ socket. If you find yourself writing a per-step HTTP POST, that is the bug, and no amount of GPU will hide it.

The observation payload is bigger than people expect

Latency is only half of the problem. Every model call has to ship an observation upstream, and observations are images. The openpi remote inference documentation tells you to resize images on the client side to minimise bandwidth and latency, and notes that the typical resize size for pretrained Pi0 models is 224. Here is what that actually costs on an SO-100 with two cameras.

# One observation, two cameras, at the resolution openpi recommends
# resizing to on the client side (224 x 224 RGB, uint8).
per_camera = 224 * 224 * 3            # 150,528 bytes  = 147 KiB raw
per_obs    = per_camera * 2           # 301,056 bytes  = 294 KiB raw

# Sent at 10 model calls per second, uncompressed:
bits_per_second = per_obs * 8 * 10    # 24,084,480 bit/s = 24.1 Mbit/s

# VLA-Perf models a 4G uplink at 19 Mbit/s and WiFi 6 at 560 Mbit/s.
# Uncompressed on 4G you do not fit. On Wi-Fi you fit and then some.

In practice everybody JPEG-encodes, which brings this down by roughly an order of magnitude. That costs encode time on the robot controller and decode time on the server, and it introduces compression artefacts into an input distribution that may differ from the one the model was fine-tuned on. It is a mismatch worth removing rather than measuring: encode the same way during recording and during inference. The desktop client that records LeRobot datasets and the client that runs the policy being one program is not an accident.

Measured latency is worse than modelled latency, sometimes by 10x

Everything above is either a vendor benchmark on a tuned export path or a roofline model. The LeRobot paper published at ICLR 2026 measured the stock implementations instead: 100 forward passes per policy, full fp32 at runtime, ten denoising steps for the diffusion and flow models, with a hard 5-second timeout, on four platforms.

Three things in that table are worth arguing with your instincts about. The RTX 4090 beats the A100 on every row, because fp32 PyTorch on these models is not doing what an A100 is good at. Pi0 costs 209.4 ms on a 4090 in stock fp32, while VLA-Perf's roofline for the same model on the same class of card is 31.06 ms and a Triton implementation tuned for the 4090 measured 20.0 ms with one camera. And ACT at 5.0 ms on a 4090 is faster than the 20 ms this platform plans with, which is the right direction for a planning figure to be wrong in.

The rule that falls out: the gap between a roofline and your own process is a software gap, and it is larger than the gap between two adjacent GPUs. Before renting a bigger card or buying a Jetson, find out which of those numbers your stack produces. The policies page and the comparison pages give you planning figures; only a timer around your own call gives you yours.

Action chunking is what makes remote inference survivable

Action chunking came out of the ALOHA work: given an observation, the policy generates the next k actions and executes them. The original paper introduced it to combat compounding errors in imitation learning, not to fight latency. It happens to be the single most important latency tool a robot has anyway.

The reason is simple. Without chunking you pay inference plus a round trip once per control step. With a chunk of 50 actions executed at 30 Hz, one model call covers 1.67 seconds of motion, so you pay the round trip once per 1.67 seconds instead of fifty times. The openpi documentation says the same thing in one sentence: you typically only need to call the policy every N steps and execute the remaining steps from the predicted chunk open-loop. The Pi0 authors do exactly that in production, running inference every 0.5 seconds after executing 25 actions on their 50 Hz robots, and every 0.8 seconds after 16 actions on the 20 Hz arms.

LeRobot's asynchronous inference stack turns this into two knobs. actions_per_chunk defaults to 50, and chunk_size_threshold defaults to 0.7, with the documentation recommending 0.5 to 0.6 in practice and the shipped example using 0.5. When the queue drops to that fraction, the client sends a fresh observation while the robot keeps executing what it already holds. That overlap is the whole trick, and it is why the stack is called asynchronous rather than remote.

# Robot side. Runs next to the servos, holds the action queue,
# streams observations to a policy server that may be anywhere.
python -m lerobot.async_inference.robot_client \
    --server_address=127.0.0.1:8080 \
    --robot.type=so100_follower \
    --robot.port=/dev/tty.usbmodem585A0076841 \
    --robot.id=follower_so100 \
    --robot.cameras="{ laptop: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}, phone: {type: opencv, index_or_path: 1, width: 1920, height: 1080, fps: 30}}" \
    --task="pick up the cube" \
    --policy_type=smolvla \
    --pretrained_name_or_path=lerobot/smolvla_base \
    --policy_device=cuda \
    --actions_per_chunk=50 \
    --chunk_size_threshold=0.5 \
    --aggregate_fn_name=weighted_average \
    --debug_visualize_queue_size=True

The numbers published with that stack set expectations honestly, as long as you read what was measured. Hugging Face reports sub-100 ms round-trip latency hosting SmolVLA on an RTX 4090 on their local network, and around 100 ms of inference latency for an ACT model on a 2021 MacBook Pro, which they note is roughly three frames at 30 fps. The often-quoted 2x speed-up from switching to asynchronous inference is a demonstration figure; the measured means in the SmolVLA paper are 9.7 s against 13.75 s to complete the pick-and-place task, which the paper itself calls about 30 percent faster, and 19 successful cycles against 9 in a fixed 60-second window. Average success across three tasks was 78.3 percent synchronous against 73.3 percent asynchronous, so it is comparable rather than free.

Serving research has started treating this as its own problem rather than a special case of language-model serving. The Kairos work argues that physical AI tasks have inference properties markedly different from digital AI, makes the generate-execute loop a first-class citizen in the scheduler instead of two unrelated phases, and reports 31.8 to 66.5 percent lower average end-to-end task latency at peak load against FIFO scheduling as used in LLM serving systems and fairness-based scheduling from multi-round agent serving. Its real-robot experiments run on a bimanual SO-101. The code is not released yet, so none of it is installable today, but it is a decent indication of where the remaining slack is: not in the model, in the loop wrapped around it.

Which tasks tolerate the round trip

There is no single latency threshold. What matters is how fast the world changes relative to how fast you can respond, so the answer is per-task rather than per-millisecond. The useful question is whether the scene can change during the chunk you are executing blind.

The pattern behind that table is a single question: can the scene change during the chunk you are executing blind, and does it matter if it does? Laundry folding is the interesting case, because it is the one the Pi0 paper actually ships. Fabric keeps moving while you are not looking at it, and Pi0 still runs 25 actions open-loop between inference calls, which works because 25 actions at 50 Hz is only half a second and because the controller underneath is fast. Stretch that same half second across a congested uplink into a distant region and you are no longer running the configuration that was validated. The cheap test costs nothing: record the task, replay it, and ask what would have gone wrong if the arm had been blind for the last 1.5 seconds of every chunk.

The last row is worth separating out. Teleoperation over the internet works because a human closes the loop and humans tolerate a couple of hundred milliseconds of delay by slowing down. You can see it working on the live arm, which streams a physical arm to anyone with no signup. Policy inference has no such tolerance built in, because the policy does not know it is late.

Doing it yourself, or doing it on AY-Robots

The goal is the same on both sides: a trained checkpoint serving actions to a physical arm, with the GPU somewhere other than the arm. The difference is how much plumbing you own and how much of the GPU lifecycle you have to remember.

A decision procedure you can run in ten minutes

1. 1
   Write down the control rate the task needs

   Not the rate your camera runs at, and not the rate the paper reports. The rate at which the scene stops being the scene you photographed. For a settled object on a table that can be 5 Hz. For anything with a person in it, it is not.

   python

   control_hz = 30
   budget_ms  = 1000 / control_hz      # 33.3 ms per step

2. 2
   Time your own forward pass before you blame the network

   A stock fp32 process can be 10x off a tuned one on identical hardware, which is more than routing will ever do to you.

   python

   import time
   t = time.perf_counter()
   for _ in range(100):
       policy.select_action(obs)
   print((time.perf_counter() - t) * 10, "ms per call")

3. 3
   Measure your real round trip to the endpoint you will actually use

   Not a speed test to a nearby server. The endpoint. Take enough samples to see the tail, and look at the maximum as well as the average.

   bash

   # Median and tail RTT, 200 samples.
   ping -c 200 <your-inference-endpoint> | tail -2
   
   # What a real request costs: TCP, TLS and first byte.
   curl -sS -o /dev/null \
     -w 'connect %{time_connect}s tls %{time_appconnect}s ttfb %{time_starttransfer}s\n' \
     https://<your-inference-endpoint>/health

4. 4
   Do the subtraction with the tail, not the mean

   The condition you need is that one model call finishes before the queue drains past the threshold. Use the p99 round trip and the platform inference figure for your model.

   python

   inference_ms = 152     # GR00T N1.7, per /policies
   rtt_p99_ms   = 90      # what you just measured, tail not mean
   chunk        = 50      # actions_per_chunk
   threshold    = 0.5     # chunk_size_threshold
   
   call_ms  = inference_ms + rtt_p99_ms   # 242 ms per model call
   chunk_ms = chunk * budget_ms           # 1667 ms of motion per chunk
   
   # Safe while a call finishes before the queue drains past the threshold:
   print(call_ms < chunk_ms * threshold)  # 242 < 833  ->  True

5. 5
   If it fails, shorten the chunk or move the GPU

   Those are the only two real levers. Lowering chunk_size_threshold sends observations more often and puts more pressure on the pipeline; raising actions_per_chunk buys headroom and costs reactivity. If neither closes the gap, the policy belongs on the robot, which on this platform means ACT or SmolVLA.

Run that with ACT's 20 ms instead of GR00T's 152 ms and the arithmetic barely changes, because at a 50-action chunk you are dominated by the round trip in both cases. Run it with a 5-action chunk and it collapses immediately: 110 ms against a 167 ms chunk at a 0.5 threshold leaves you 83 ms of budget and you have already spent 110. Chunk length, not model size, is what buys you distance. The head-to-head comparisons and the model arena with its 85 models and 332 benchmark results will tell you about accuracy; they will not tell you this, so do it yourself.

Where this platform does not help

Being useful here means being specific about the cases where renting a GPU is the wrong answer. There are several, and they are not edge cases.

• It cannot turn a 485 ms Pi0.5 step into a 20 ms one. A cloud GPU makes the model faster than a Jetson would; it does not make a large diffusion policy fast enough for reactive control.
• GR00T N1.7, GR00T N1.5 and Pi0.5 are cloud-only here. If your analysis says the policy must run on the robot, those three are off the table and you are choosing between ACT and SmolVLA.
• It does not fix your uplink. A pod in a fast region behind a congested home connection is still behind a congested home connection, and VLA-Perf's modelled 4G-plus-slow-cloud case (273.4 ms end to end, 3.7 Hz) is what that looks like.
• Nothing here removes the open-loop blindness of a long chunk. Latency tolerance and reactivity are the same budget spent twice.
• An idle pod stops billing because of the watchdog, but an active one bills at the published hourly rates for as long as the arm is running. Remote inference has an ongoing cost that a Jetson on the bench does not.
• There is no published end-to-end measurement of an SO-100 running a policy over the public internet, here or anywhere else. Everything on this page above the Pi0 measurement is arithmetic over vendor figures, and you should treat your own timing run as the authority.

If you want the background on why these models are shaped the way they are, and therefore why they cost what they cost per step, the VLA overview covers the architecture and the SO-100 guide covers the hardware side of the loop. For the terminology used here, the glossary has short entries, and the CLI and the MCP server expose the same operations if you would rather not click.