Latency Budget for 30 Hz Robot Control: Where the Milliseconds Go

A 30 Hz control loop gives you 33.3 milliseconds per tick. Inside that window a robot has to read its joint positions, pull a camera frame, turn both into an action and get that action onto the servo bus. Miss the window and nothing dramatic happens at first: the loop just takes longer, the commanded positions arrive late, and the arm develops a small stutter that you notice before you can name it. Miss it badly and the policy that looked fine in evaluation starts hesitating at exactly the moments that matter, like the instant before a grasp.

This is a budget article. It walks through where the milliseconds go in a real policy rollout on an SO-100, what action chunking actually buys you (it is not what most people assume), and the measured point at which running longer without a fresh observation starts costing success rate. Numbers about models come from papers, model cards and source code fetched and checked on 2026-08-24. Numbers about this platform come from the policy catalog.

What actually happens inside one tick

LeRobot's deployment entry point is lerobot-rollout. Its config defaults fps to 30.0, and the base rollout strategy wraps each part of the loop body in a named timing section. Those section names are the honest taxonomy of a control loop, so it is worth reading them straight from the source rather than inventing a model of your own.

# src/lerobot/rollout/strategies/base.py - the base strategy loop body
while not ctx.runtime.shutdown_event.is_set():
    timer.tick(new_cycle=interpolator.needs_new_action())

    with timer.section("observe"):
        obs = robot.get_observation()
    with timer.section("process_obs"):
        obs_processed = self._process_observation_and_notify(ctx.processors, obs)

    action_dict = send_next_action(obs_processed, obs, ctx, interpolator, timer)
    #   -> with section("infer"):  engine.get_action(...)
    #   -> with section("send"):   robot.send_action(...)

    with timer.section("telemetry"):
        self._log_telemetry(obs_processed, action_dict, ctx.runtime)
    with timer.section("query"):
        engine.pump_query(obs_processed)

    timer.wait()

The pacing helper deserves one line of its own. precise_sleep has a 10 ms spin threshold and a 5 ms sleep margin, but read the body before you assume what it does: only on macOS and Windows does it sleep to 5 ms before the deadline and then spin out the last stretch. On Linux it falls straight through to time.sleep, which the source calls accurate enough for most uses. Its docstring says the defaults were chosen to prioritise timing accuracy over CPU usage for the common 30 FPS case. So the busy-wait trade is real, but only on two of the three platforms.

Inference is the biggest single line item

Here is the platform's own per-action-step latency for the five trainable policies, converted into tick budgets at 30 Hz. The conversion is plain arithmetic on the catalog numbers, nothing more.

Only ACT fits inside a tick. Everything else is over budget by a factor of roughly five to fifteen, which is the entire reason the rest of this article exists. If you are choosing between them, the ACT against SmolVLA comparison and the GR00T N1.7 against Pi0.5 page put the same numbers next to accuracy and dataset requirements.

Where those milliseconds sit inside the model

NVIDIA publishes a per-stage breakdown on the GR00T N1.7 model card, measured with 4 denoising steps and one camera. It is the clearest public answer to the question of where a vision-language-action model spends its time. These rows were read from the card on 2026-08-24.

Three things fall out of that table. Data processing is noise, 6.2 to 9.5 ms across those rows. The backbone and the action head are roughly comparable in eager mode, and the action head is what compilation fixes: on an H100 it drops from 48.2 ms to 12.0 ms with torch.compile alone. And a full TensorRT pipeline is the difference between 11.7 Hz and 35.9 Hz on the same card, which is the difference between a model that cannot hold 30 Hz and one that can.

What action chunking actually buys you

Action chunking means the policy emits a sequence of future actions from a single observation instead of one action. It arrived with ACT in Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (Zhao et al., 2023), where the arms ran at 50 Hz and the chunk size was fixed at k = 100. Every modern VLA does some version of it: Pi0 uses H = 50 and reaches control frequencies up to 50 Hz, and the Diffusion Policy paper (Chi et al., 2023) built receding-horizon control around the same idea.

The misunderstanding worth clearing up: chunking does not reduce inference latency by a single millisecond. A 485 ms forward pass still takes 485 ms. What chunking changes is how often you have to pay it. One call buys you N actions, so the loop can keep issuing commands at 30 Hz while the next call is still running. It solves starvation. It does not solve staleness.

Read the fourth column again. With stock settings, an ACT policy running at 30 Hz looks at the world once, then acts blind for three and a third seconds. That is not a bug in LeRobot, it is what n_action_steps = chunk_size means. It is also almost certainly not what you want.

The point where the arm starts to hesitate

This is the part with real measurements attached. The SmolVLA paper (Shukor et al., June 2025) ran an ablation on LIBERO that varies exactly one thing: how many actions from a 50-step chunk get executed before a new observation is taken. Everything else is held constant.

Executing the full chunk costs 31 percentage points of average success against re-observing every 10 steps. The paper's own summary is that sampling new observations more frequently significantly improves performance. A companion ablation on chunk size found sizes between 10 and 50 balance reactivity against efficiency, with size 1 collapsing to 50.0% and size 100 falling back to 74.5%.

Diffusion Policy reached the same conclusion from the other direction two years earlier. Its ablation found an action horizon of 8 steps optimal for most tasks tested, with the paper noting that too long a horizon reduces performance due to slow reaction time. It also defines latency in a way worth stealing: the number of steps between the last frame of observations and the first action that can be executed. That is the quantity your budget should track, not the forward-pass time in isolation.

Telling staleness apart from starvation

These two failure modes look different on the arm and they have different fixes, so it is worth being able to name them on sight. Starvation is the loop having nothing to send. The arm holds position, or twitches, and then jumps when the next chunk lands. LeRobot counts this directly: the run summary reports ticks with no action to send, labelled as the inference engine being starved, and notes that each of those ticks commanded nothing and recorded no frame. If that counter is above zero you have a throughput problem, and the fixes are mechanical: a bigger chunk, a lower fps, a faster model, or fewer denoising steps.

Staleness is different, and worse, because it does not show up in any counter. The loop holds its rate perfectly, the queue never empties, no warning fires, and the arm moves smoothly to the wrong place. What you see is confident motion aimed at where the object was a second ago, followed by a correction once a fresh observation finally arrives. Operators usually describe it as the arm looking hesitant or second-guessing itself, which is a fair description of a policy repeatedly discovering that its plan was built on old information. The only fix is to shorten the interval between observations, which means a smaller execution horizon, and that is a decision you have to make deliberately because nothing in the logs will make it for you.

What the network adds, per action, forever

A round trip is not a one-off cost you pay at startup. It is added to the age of every observation the policy ever sees. The Real-Time Chunking paper (Black, Galliker and Levine, arXiv v2 revised December 2025) is unusually concrete about this. It notes that Pi0 running remote inference for mobile manipulation lists 13 ms of network latency in perfect conditions with a wired connection, and that in a more realistic setting the network overhead alone could easily exceed 20 ms. Their own real-world setup used remote inference over LAN, which added 10 to 20 ms.

For comparison, Hugging Face's async inference post (published 10 July 2025) reports sub-100 ms round-trip latency over gRPC on their local network with SmolVLA hosted on an RTX 4090. Both of those are LAN numbers. Neither is a public-internet number, and a public-internet number is not a fixed multiple of them, because the tail matters more than the mean. A loop with a 33 ms budget is wrecked by a 400 ms p99 even when the median is a comfortable 40 ms.

1. 1
   Measure the round trip before you design around it

   Run this against the host you plan to put the policy on, from the machine the arm is attached to. Look at the maximum and the standard deviation, not the average.

   bash

   # 200 pings, one every 200 ms, so you see the tail and not just the mean
   ping -c 200 -i 0.2 your-gpu-host.example.com
   
   # where the variance comes from
   mtr --report --report-cycles 100 your-gpu-host.example.com

2. 2
   Convert it into ticks

   At 30 Hz, divide by 33.3. A 45 ms round trip is 1.35 ticks of pure transport added to every observation, before the model has done anything at all.

   bash

   python3 -c "rtt=45; fps=30; print(f'{rtt/1000*fps:.2f} ticks of transport per action')"

3. 3
   Add it to the model number and compare against your chunk runway

   With actions_per_chunk = 50 and a request fired when the queue is half empty, you have 25 actions of runway, which at 30 Hz is 833 ms. Inference plus round trip has to fit inside that or the queue starves.

   bash

   # runway in ms = remaining_actions / fps * 1000
   python3 -c "print(25/30*1000, 'ms of runway')"   # 833.3 ms
   # Pi0.5 at 485 ms + 45 ms RTT = 530 ms. Fits, with 303 ms of margin.

Notice what that last step does and does not prove. It proves the queue will not run dry. It says nothing about the actions being good, because every action in that chunk is still a response to a picture taken 530 ms ago. This is the honest limit of remote inference, and it is why this platform states it plainly: inference has to sit next to the servos for fast tasks, and remote inference is viable for slow pick-and-place, not for fast reactive motion. See inference latency for the definition and the client guide for how the local robot client is wired up.

Two mechanisms that keep a fast loop fed by a slow model

Asynchronous inference

The async inference stack splits the system into a PolicyServer holding the model and a RobotClient holding the arm, talking over gRPC. The client keeps stepping through its action queue while the server computes the next chunk, so there are no idle frames. It works with every LeRobot policy, not only SmolVLA.

pip install -e ".[async]"

# terminal 1: the model
python -m lerobot.async_inference.policy_server \
     --host=127.0.0.1 \
     --port=8080

# terminal 2: the arm
python -m lerobot.async_inference.robot_client \
    --server_address=127.0.0.1:8080 \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=follower_so100 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --task="pick up the cube" \
    --policy_type=smolvla \
    --pretrained_name_or_path=user/my_smolvla \
    --policy_device=cuda \
    --actions_per_chunk=50 \
    --chunk_size_threshold=0.5 \
    --aggregate_fn_name=weighted_average \
    --debug_visualize_queue_size=True

Two knobs matter. actions_per_chunk is how many actions come back per call, capped at the policy's trained maximum. chunk_size_threshold is the queue fill fraction at which the client sends a fresh observation. The docs recommend 0.5 to 0.6 and note that values near 0.0 collapse to synchronous behaviour while values near 1.0 send an observation every step. Turn on --debug_visualize_queue_size and watch the queue depth. If it repeatedly touches zero, either lower your fps or raise the threshold.

Real-Time Chunking

RTC is the sharper tool. Instead of merely overlapping computation with execution, it treats the new chunk as an inpainting problem: the actions guaranteed to execute before the new chunk arrives are frozen, and the rest are generated conditioned on them. The paper's framing is that it applies to any diffusion-based or flow-matching VLA out of the box with no re-training. Their real-world setup ran Pi0.5 with H = 50 at a 20 ms timestep and 5 denoising steps, giving 76 ms of model latency for the baselines and 97 ms for RTC, and they then injected an extra 100 ms and 200 ms to simulate a distant cloud server. RTC showed no degradation across those delays while synchronous execution degraded linearly.

LeRobot ships it. lerobot-rollout takes --inference.type=rtc alongside the default sync engine.

lerobot-rollout \
    --strategy.type=base \
    --policy.path=lerobot/pi0_base \
    --inference.type=rtc \
    --inference.rtc.execution_horizon=10 \
    --inference.rtc.max_guidance_weight=10.0 \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --task="pick up cube" --duration=60

That default execution horizon of 10 is worth noticing: it lands in exactly the band the SmolVLA ablation and the Diffusion Policy ablation identified as good, 8 to 10 steps. Treat that as two independent lines of evidence rather than three, because the SmolVLA ablation and the LeRobot default come from the same team at Hugging Face; the Diffusion Policy number is the outside check. Two groups landing in the same band is about as close to a settled answer as this field gets.

Interpolation, if you want commands faster than inference

There is a third, smaller lever. RolloutConfig has interpolation_multiplier (default 1). With multiplier N the control loop runs N ticks per policy cycle: commands go out at fps times N Hz while inference and dataset recording advance once per cycle at fps Hz. The CycleTimer docstring gives the concrete case, that at 30 FPS with multiplier 2 a 25 ms policy tick followed by a 5 ms interpolated tick still fits the 33.3 ms cycle. This buys smoothness on the wire, not freshness in the observation. Do not confuse the two.

Doing it yourself against doing it here

Reading the cadence report

You do not have to guess at any of this. Every LeRobot loop that drives hardware at a fixed rate uses the same CycleTimer, and it warns when the loop body cannot fit the budget. The message is a literal string in the source, and it names the three causes in roughly the order they turn out to be the culprit.

# lerobot-rollout (records_data=False)
WARNING  Control loop is running slower (21.4 Hz) than the target FPS (30 Hz).
         Robot control might be unstable.
         Common causes are: 1) Camera FPS not keeping up
         2) Policy inference (action or text) taking too long
         3) CPU starvation

# a recording loop (records_data=True) swaps the middle clause for:
         Dataset frames might be dropped and robot control might be unstable.

At the end of a run the timer prints a full block: effective cadence against target, how many ticks went over the work budget with mean and worst values in milliseconds, ticks with no action to send (the inference engine starved), and a per-section breakdown giving mean, worst and share of work for observe, process_obs, infer, send, telemetry and query. That per-section breakdown is the whole diagnostic. If observe dominates, it is cameras or the servo bus. If infer dominates, it is the model, and you are choosing between a smaller policy, fewer denoising steps, compilation, or RTC.

A worked budget for an SO-100 at 30 Hz

Putting it together for a concrete setup: one SO-100 follower on a USB serial adapter, two 640x480 cameras at 30 fps, a policy on a local 4090. The point of the table is the structure. Replace each row with your own measured section timings from the cadence report and you have a real budget rather than a sketch.

The last row is the one to internalise. If you set your execution horizon to 10, the oldest action you execute from a chunk carries an extra 333 ms of age on top of everything above. Set it to 50 and that becomes 1.67 s, and the SmolVLA numbers tell you what happens next. Recording the episodes at the same rate you intend to run at keeps the two sides of this honest; the recording tutorial covers that part.

When 30 Hz is the wrong target

Nothing here argues that 30 Hz is sacred. ACT's original setup ran at 50 Hz, and the paper measured what dropping to 5 Hz costs: a 62% increase in teleoperation time on the same hardware. OpenVLA-OFT's real robot ran at 25 Hz, explicitly reduced from 50 Hz to make training faster while still keeping control smooth. Pi0 targets up to 50 Hz.

The rate to pick is the one your task needs and your LeRobot dataset was recorded at. Recording and inference have to agree; a policy trained on 30 fps data does not get smoother when you run it at 60 Hz, it just gets asked to predict a timescale it never learned. If your task is quasi-static, 10 Hz with a fresh observation every step will beat 30 Hz with a 50-step open-loop chunk, and it costs less GPU time. If your task involves catching, balancing or anything with contact dynamics, you need both the rate and the freshness, which in practice means a small model running locally. The model arena has 85 VLA models with 332 benchmark results, each value linked to its source, if you want to widen the search beyond the five trainable here.

Where to go next

• The complete SO-100 setup guide covers the hardware half of the loop, which is where the observe section's cost comes from.
• The VLA overview explains what these models are actually doing with the milliseconds they take.
• The Pi0 flow-matching walkthrough goes deeper on why denoising steps exist at all, which is the knob behind most of the latency numbers above.
• Run your first policy is the shortest path to having a loop you can measure, and ACT on the SO-100 is the cheapest model to measure it with.
• The training docs and the SO-100 LeRobot page cover the parts either side of deployment.