Serving One Policy to Several Arms: Batching and Its Ceiling

You have three SO-100 arms on a bench and one rented GPU. The obvious question is whether a single policy server can drive all three, and the obvious hope is that batching makes the extra arms nearly free. Batching does help, and the help is measurable. It is also not free, and the cost lands somewhere uncomfortable: every arm in a batch waits for the whole batch to finish, not for its own share of it.

The reference servers were written for one arm

Before any arithmetic about batch sizes, there is a fact worth checking in the source: the three serving stacks that most people start from do not serve several robots at once. Not badly, not slowly. They do not do it. Each one assumes a single client holding a single policy, and each one fails differently when a second client shows up.

This is not an oversight so much as a scope decision. The single-client design is what makes action chunking and asynchronous execution simple: one observation queue of depth one, one policy, one client whose action buffer the server can reason about. Adding a second robot breaks every one of those assumptions at once.

The model side is in better shape. NVIDIA's Policy API documents batched inference explicitly: pass observations with a leading batch dimension and you get actions back with shape (batch_size, action_horizon, action_dim). The tensors are ready for multi-tenancy long before the servers are.

# Isaac-GR00T, getting_started/policy.md: the batch dimension already exists
batch_size = 4
observation = {
    "video": {"wrist_cam": np.zeros((batch_size, T_video, H, W, 3), dtype=np.uint8)},
    "state": {"joints": np.zeros((batch_size, T_state, D_state), dtype=np.float32)},
    "language": {"task": [["pick up the cube"]] * batch_size},
}

action, _ = policy.get_action(observation)
# action["action_name"] has shape (batch_size, action_horizon, action_dim)

What batching actually costs, in measured milliseconds

The useful measurements come from Armory, a serving system published as Action Chunk Scheduling for Batched Robot Policy Serving (arXiv 2608.00337, 31 July 2026, Georgia Tech). The authors profiled inference latency as a function of batch size for Pi0.5 and GR00T N1.7. The last two columns below are mine: batch latency divided by batch size.

Read the two L40S columns side by side, because they behave nothing alike. Pi0.5 at batch 5 costs 2.9 times what a single request costs. GR00T N1.7 at batch 5 costs 1.4 times. The paper's explanation is that GR00T is more memory-bound than Pi0.5 in their setup, so the extra samples ride along with weight loading that has to happen anyway. The practical consequence: on the same card, five arms on GR00T are close to free, and five arms on Pi0.5 are close to a full extra card's worth of work. The paper labels the GR00T table preliminary analysis, so treat it as a strong hint rather than a specification.

Why the two curves differ

A second 2026 preprint, PhyAI (arXiv 2608.03682, August 2026), explains the mechanism by profiling execution phases. At batch size one on a Hopper-series GPU, the Pi0.5 action expert accounts for 8.8 percent of estimated FLOPs but 57.2 percent of profiled latency: many small GEMMs, poorly fed. At batch 32 its share of time falls to 13.5 percent and the vision-language path takes over. For GR00T N1.7 on an RTX 5090 they measured the same handover: the action head is 59.2 percent of the time at batch 1, and the backbone reaches 69.7 percent at batch 32. Batching does not speed up your model. It fills in the idle silicon that a batch of one leaves empty.

The Pi0.5 curve is a direct consequence of that architecture. A flow-matching policy runs an iterative action expert on top of a vision-language backbone, and at batch one the expert is starved of work while the backbone is not. The Pi0 flow-matching write-up covers the architecture itself. The serving consequence is that the denoising loop is exactly the part batching repairs, and the vision-language path is the part it cannot.

Throughput per second is not the time your arm waits

This is the single most common mistake in fleet inference planning. A batch of 32 that completes in 320 ms is often reported as ten milliseconds per request. No robot in that batch experienced ten milliseconds. Every one of them waited 320 ms.

The same document decomposes the end-to-end path into observation, transfer, queue, inference and actuation, and is blunt that the queue term is separate from the inference term: batch measurements are static-batch GPU time and exclude request queues, network transfer and tail latency. That missing queue term is exactly what a shared server introduces and what this article is about.

The arithmetic that decides how many arms fit

Three numbers determine the answer, and you can get all of them in an afternoon. Service time S(B) is how long a batch of size B takes on your card. Capacity is B divided by S(B), in requests retired per second. Demand per arm is the control frequency divided by the number of actions each arm executes before it needs a fresh chunk. Utilisation is demand divided by capacity, and it is the only one of the four you should be nervous about.

Armory's real fleet gives concrete values for demand: ten AgileX PiPER arms at 30 Hz, running a Pi0.5 checkpoint fine-tuned with a prediction horizon of 20. Their quasi-static arms execute up to 20 actions per chunk, which is 0.667 seconds of motion, or 1.5 requests per second per arm. Their dynamic arms cap the execution horizon at 10, which is 0.333 seconds, or 3.0 requests per second per arm. Halving the horizon doubles the load on the server, which is why reactive tasks are expensive to serve.

# Substitute your own measured service times. These are Armory's
# GR00T N1.7 figures on an L40S (arXiv 2608.00337, appendix A.3).
S = {1: 0.0718, 2: 0.0765, 3: 0.0835, 4: 0.0918, 5: 0.1009}  # seconds

fc = 30          # control frequency in Hz
h_exec = 20      # actions executed per chunk before a new one is needed
target_rho = 0.7 # leave 30 percent headroom, see the queueing section

per_arm = fc / h_exec                      # requests per second, per arm
for B, s in S.items():
    capacity = B / s                       # requests per second the server retires
    arms = target_rho * capacity / per_arm
    print(f"batch {B}: {s*1000:5.1f} ms/batch  {capacity:5.1f} req/s  {arms:4.1f} arms")

Arm counts assume 30 Hz control and 70 percent target utilisation, and they are my arithmetic on Armory's measured latencies, not a measured fleet size. Treat them as the ceiling you plan against, then verify with real arms.

The count is only half the check. The deadline is the other half. A reactive arm with 10 actions in hand at 30 Hz has 333 ms of motion left. Pi0.5 at batch 5 on an L40S consumes 211 ms of that before a single byte goes back on the wire. Add two 50 ms network hops and you have spent 311 ms of a 333 ms budget on a link that has not hiccupped yet. On the same card, GR00T N1.7 at batch 5 takes 101 ms and leaves genuine headroom. This is why the GR00T against Pi0.5 comparison looks different when you are serving a fleet than when you are serving one arm.

Queueing shows up as jitter, not as a slower average

When a shared server saturates, the mean round trip barely moves. What moves is the spread, and the arm is far more sensitive to spread than to level. A policy that always answers in 200 ms is easy to build around. A policy that answers in 90 ms most of the time and 400 ms sometimes is the one that makes the gripper stutter.

• The arm pauses briefly at chunk seams, which Armory calls starvation: control steps where the robot has no action left to execute.
• Two arms running the same checkpoint behave differently, and the difference tracks which one got into the batch.
• The p99 round trip separates from the median while the median stays flat.
• Adding a fourth arm degrades all four rather than just the fourth.
• Motion quality falls off faster on the dynamic task than on the slow one, at the same server load.

The reason is structural. For a single-server first-in-first-out queue with random arrivals, the classic M/M/1 model puts the average wait before service at rho/(1 - rho) service times, where rho is utilisation. At 50 percent load that is one extra service time. At 80 percent it is four. At 90 percent it is nine. Batched serving is not literally M/M/1, since the server pulls several requests at once, but the shape of the curve is the same and it explains the field report that everyone has: four arms are fine, six arms fall over, and nothing in between felt like a warning.

Armory's network sweeps add a useful correction to intuition. With a one-way median of 50 ms, they swept jitter from mild to extreme and found starvation rate and system throughput stayed nearly flat, because their delay estimator and the arm's own action buffer absorb individual late packets. Median delay, swept from 25 ms to 500 ms, had a direct monotonic effect instead, and all schedulers degraded the same way. Their conclusion is worth memorising: scheduling policy cannot compensate for raw transport delay. The jitter that hurts is the queue's, not the network's, because queue jitter is correlated with load rather than random.

Measure it before you believe it

1. 1
   Stand up a single-client server first

   Get one arm working end to end before adding anything. LeRobot's async inference server is the shortest path and supports act, smolvla, diffusion, tdmpc, vqbet, pi0, pi05 and groot.

   bash

   python -m lerobot.async_inference.policy_server \
        --host=0.0.0.0 \
        --port=8080

2. 2
   Attach the robot client and watch the action queue

   The client, not the server, is where starvation is visible. The --debug_visualize_queue_size flag plots the queue depth at runtime. Defaults that matter: chunk_size_threshold is 0.5, which means a new observation is sent when half the chunk is left.

   bash

   python -m lerobot.async_inference.robot_client \
       --server_address=127.0.0.1:8080 \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=follower_so100 \
       --task="pick up the cube" \
       --policy_type=smolvla \
       --pretrained_name_or_path=user/model \
       --policy_device=cuda \
       --actions_per_chunk=50 \
       --chunk_size_threshold=0.5 \
       --aggregate_fn_name=weighted_average \
       --debug_visualize_queue_size=True

3. 3
   Measure your own S(B) curve

   Do not import someone else's table. Time batches on the card you will rent, with your camera count and your chunk length, because both change the answer. GR00T's client takes a batched observation directly.

   python

   import time
   from gr00t.policy.server_client import PolicyClient
   
   client = PolicyClient(host="127.0.0.1", port=5555)
   
   for B in (1, 2, 3, 4, 5):
       obs = make_observation(batch_size=B)   # leading batch dimension
       for _ in range(5):
           client.get_action(obs)             # warm up, then time
       t0 = time.perf_counter()
       for _ in range(30):
           client.get_action(obs)
       print(B, round((time.perf_counter() - t0) / 30 * 1000, 1), "ms")

4. 4
   Separate server compute from queue and network

   openpi's server puts its own timing in every reply, which lets you subtract compute from the round trip and see what the wire and the queue cost you. If infer_ms is stable while your measured round trip is not, the problem is not the model.

   python

   result = client.infer(observation)
   print(result["server_timing"]["infer_ms"])     # pure model time on the server
   print(result["server_timing"]["prev_total_ms"]) # previous request, including send

5. 5
   Add arms one at a time and keep the log

   Add one arm, run twenty minutes, record p50, p99 and starvation count per arm. Repeat. The number where p99 doubles is your real ceiling, and it will be lower than the capacity arithmetic suggested.

   bash

   # GR00T's own server, for the same experiment on the NVIDIA stack
   uv run python gr00t/eval/run_gr00t_server.py \
       --model-path <CHECKPOINT_PATH> \
       --embodiment-tag NEW_EMBODIMENT \
       --device cuda:0 \
       --host 0.0.0.0 --port 5555

Doing it yourself against doing it here

Where the ceiling really is

That last row deserves a sentence of its own, because the tempting workaround is to run five server processes on one big card instead of building a batcher. The arithmetic says no. Five serialised GR00T N1.7 requests on an L40S cost five times 71.8 ms, which is 359 ms of GPU time. One batch of five costs 100.9 ms. Separate processes give you the memory isolation and none of the amortisation, plus context-switch overhead you did not have before. If you are going to share a card, share it properly.

There is also a ceiling that no amount of engineering moves: the model's own service time. The platform lists 20 ms per action step for ACT, 152 ms for GR00T N1.7, 165 ms for GR00T N1.5, 245 ms for SmolVLA and 485 ms for Pi0.5. A vision-language-action model whose forward pass is half a second is one you can serve to very few reactive arms, no matter how good your scheduler is. Choosing the model is choosing the fleet size, which is worth thinking about before training rather than after. The arena compares 85 VLA models with 332 benchmark results if you want the wider field, and the VLA overview covers why these models are shaped the way they are.

Where this does not help

Batching is a throughput technique applied to a latency problem, and it is worth being clear about which problems it leaves untouched. It does not make any single arm more responsive. It does not shorten the network path. It does not rescue a task that was already marginal at batch one. Real-time chunking, from Real-Time Execution of Action Chunking Flow Policies (Black, Galliker and Levine, June 2025), buys genuine latency tolerance, holding up under 100 ms and 200 ms of injected delay where synchronous execution degrades linearly. But PhyAI states the limit plainly: real-time chunking gives inference more time by overlapping it with execution, and it does not reduce network or queueing delay.

The same applies to the platform. An inference pod is provisioned per policy with an idle watchdog, which solves GPU provisioning and cost control. It is not a fleet batcher, and pretending otherwise would just move the surprise later. If you want to see the pieces working before committing hardware, the live arm streams a real SO-100 with no signup, and the try page lays out the three ways to start. When you are ready to record your own data, the desktop client writes LeRobot datasets straight from a teleoperation session, and the dataset directory lists public ones.

If none of this applies yet because you are still getting one arm to move, start there instead: the SO-100 setup guide walks the whole path from build to teleoperation to training, and run your first policy is the shortest version of the last step. Fleet arithmetic only pays off once a single arm does the task reliably, and most teams reach for a second arm several weeks before that is true.

Frequently asked questions