Running a Policy Server: What Goes Over the Wire

A policy server is the smallest useful idea in robot deployment: put the model on a machine with a GPU, put the motors on a machine without one, and pass observations one way and actions the other. Every serious VLA stack ships one. NVIDIA's Isaac-GR00T ships a ZeroMQ server, LeRobot ships a gRPC one, and Physical Intelligence's openpi ships a WebSocket one. They solve the same problem and they fail in three completely different ways.

This page is about the wire, not the model: what is in the payload, how many kilobytes it is, which framing limit you hit first, and what has to be true for the arm to keep moving while the server is still thinking. Every number below comes from reading the three repos on 23 August 2026 or from a benchmark run for this article.

What a policy server actually is

Strip away the framework and there is one remote procedure: here is what the robot sees, give me what it should do. The request carries one frame per camera, the current joint positions, and usually a natural-language task string. The response carries an action chunk: a matrix of shape (horizon, action_dim) that the client plays back one row per control tick.

There is a second, less obvious reason the split exists, and the GR00T deployment guide states it plainly: it keeps the full inference dependency stack off the robot. A SO-100 controller running on a laptop or a Raspberry Pi does not want CUDA, does not want transformers 4.57.3, and does not want to be restarted every time you swap a checkpoint.

Note the third row: three default ports, none of which agree. If you are switching stacks, that is the first thing that breaks and the last thing you think to check.

What goes over the wire

The clearest reference implementation for a low-cost arm is the SO-100 adapter that ships inside Isaac-GR00T. It is about forty lines and it is the entire client-side contract for a six-joint arm with two cameras.

# gr00t/eval/real_robot/SO100/eval_so100.py -- observation to model input
self.robot_state_keys = [
    "shoulder_pan.pos", "shoulder_lift.pos", "elbow_flex.pos",
    "wrist_flex.pos",   "wrist_roll.pos",    "gripper.pos",
]
self.camera_keys = ["front", "wrist"]

model_obs = {}
model_obs["video"] = {k: obs[k] for k in self.camera_keys}

state = np.array([obs[k] for k in self.robot_state_keys], dtype=np.float32)
model_obs["state"] = {
    "single_arm": state[:5],   # (5,)
    "gripper":    state[5:6],  # (1,)
}
model_obs["language"] = {"annotation.human.task_description": obs["lang"]}

# two nested calls, so every leaf ends up (B=1, T=1, ...)
model_obs = recursive_add_extra_dim(model_obs)
model_obs = recursive_add_extra_dim(model_obs)

The six floats are 24 bytes. Everything else in that dictionary is pixels, and the pixels are the whole cost model, which is why inference latency on a remote server is never just the forward pass. Here is what that observation weighs, measured with numpy and OpenCV on an Apple M3, using a deliberately busy synthetic frame so the JPEG figures are not flattered by empty background.

That last pair of rows is the argument. On a 100 Mbit/s link, raw 640x480 pixels cost 147 ms of transmission before the GPU has seen anything. The GR00T deployment guide says the same in one line: use a compressed format such as JPG. openpi goes further and resizes on the client to 224x224, the resize_size the pretrained Pi0 models expect anyway. The client guide covers the same mapping from this platform's side.

Serialisation cost, measured

The word serialisation gets blamed for a lot of latency it does not cause. Here is the same benchmark, this time reporting time rather than size. Median of 60 runs, Python 3.12, msgpack 1.1.2, numpy 2.4.3, on an M3.

For a contiguous uint8 array, both pickle and msgpack are doing a memcpy and some header bytes: under half a millisecond at 640x480, against a model that takes 20 to 485 ms depending on which of the five trainable policies you picked. Serialisation is noise. Base64 inside JSON is not: it inflates the payload by a third and costs 6.9 ms to build, a fifth of a 30 Hz control period spent on nothing.

Three transports, three failure modes

ZeroMQ REQ/REP, as Isaac-GR00T does it

GR00T N1.7's PolicyServer binds a zmq.REP socket and loops: receive bytes, deserialise, dispatch to a named endpoint, send bytes back. The registered endpoints are ping, kill, get_action, reset and get_modality_config. That is the whole protocol; there is no schema and no version negotiation.

# terminal 1, on the GPU box
uv run python gr00t/eval/run_gr00t_server.py \
    --model-path nvidia/GR00T-N1.7-3B \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
    --device cuda:0 \
    --host 0.0.0.0 --port 5555

# terminal 2, on the robot box
uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path demo_data/droid_sample \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
    --host 127.0.0.1 --port 5555 \
    --traj-ids 1 2 \
    --execution-horizon 8

gRPC streaming, as LeRobot does it

LeRobot, whose SUPPORTED_POLICIES list covers act, smolvla, diffusion, tdmpc, vqbet, pi0, pi05 and groot, splits the call in two. SendObservations is a client-streaming RPC that pushes 2 MB chunks tagged BEGIN, MIDDLE or END; the server reassembles them into a single buffer and unpickles it. GetActions is a separate unary RPC that pops the newest observation off a queue of maxsize 1, runs the policy, and returns a pickled chunk. The queue depth of one is deliberate: a stale observation is worse than no observation.

pip install -e ".[async]"

# terminal 1, GPU box: an empty server. The policy is chosen during the handshake.
python -m lerobot.async_inference.policy_server \
     --host=127.0.0.1 --port=8080 --fps=30 \
     --inference_latency=0.033 --obs_queue_timeout=1

# terminal 2, robot box
python -m lerobot.async_inference.robot_client \
    --server_address=127.0.0.1:8080 \
    --robot.type=so100_follower \
    --robot.port=/dev/tty.usbmodem58760431541 \
    --robot.id=follower_so100 \
    --task="put the cube in the bowl" \
    --policy_type=smolvla \
    --pretrained_name_or_path=lerobot/smolvla_base \
    --policy_device=cuda \
    --actions_per_chunk=50 \
    --chunk_size_threshold=0.5 \
    --aggregate_fn_name=weighted_average \
    --debug_visualize_queue_size=True

This is the path you take for SmolVLA or ACT. Two details before you debug this. The server sleeps at the end of every GetActions call to hold the reply rate down to inference_latency, so a fast model is deliberately throttled. And both ends are unencrypted: add_insecure_port on one side, grpc.insecure_channel on the other. The retry policy is real, though: 5 attempts, a 2x multiplier and a 2 s ceiling, retrying only on UNAVAILABLE and DEADLINE_EXCEEDED. The transport helper defaults to a 0.1 s initial backoff, but the async client passes the control period instead, so at 30 Hz the first retry comes after 33 ms.

WebSocket plus msgpack, as openpi does it

openpi, Physical Intelligence's own serving stack for Pi0.5, has the smallest server of the three, about ninety lines. It sends a metadata frame on connect, then loops on recv, infer, send. Two things it does are worth copying into anything you build.

# server: openpi/serving/websocket_policy_server.py
async with _server.serve(
    self._handler, self._host, self._port,
    compression=None,      # raw pixels do not deflate; this just burns CPU
    max_size=None,         # library default is 1 MiB, too small for real frames
    process_request=_health_check,   # GET /healthz -> 200 OK
) as server:
    await server.serve_forever()

# every response carries its own timing, so the client can see server-side cost
action["server_timing"] = {"infer_ms": infer_time * 1000}

The client side is four lines and has no torch dependency at all, which is the point. It also resizes and casts on the client, before the bytes leave the robot, using the same resize_with_pad helper the training pipeline uses.

from openpi_client import image_tools, websocket_client_policy

client = websocket_client_policy.WebsocketClientPolicy(host="localhost", port=8000)

observation = {
    "observation/image": image_tools.convert_to_uint8(
        image_tools.resize_with_pad(img, 224, 224)),
    "observation/wrist_image": image_tools.convert_to_uint8(
        image_tools.resize_with_pad(wrist_img, 224, 224)),
    "observation/state": state,      # unnormalised; the server normalises
    "prompt": task_instruction,
}
action_chunk = client.infer(observation)["actions"]   # (action_horizon, action_dim)

Keeping the loop stable: the queue arithmetic

This is the part people get wrong. A policy server does not have to answer within one control period; it has to answer before the client runs out of buffered actions. The SmolVLA paper states the condition exactly. Let n be the chunk length, dt the control period, and E[l] the time from sending an observation to holding the actions. The client consumes until the queue falls below a fraction g of the chunk, then sends a fresh observation. Idle frames are avoided when g >= E[l] / (n * dt).

At 30 Hz, dt is 33.3 ms and a 50-action chunk buys 1.67 seconds of motion. Put the per-step figures this platform publishes for its five policies into that inequality and you get a floor for g. The added round-trip column is illustrative, not measured: ping your own server.

LeRobot's default chunk_size_threshold is 0.5 in the code and 0.7 in the docs table, which already tells you the parameter is forgiving at n=50. Every row above clears both. The trap is the other axis.

The paper's own measurement is the honest headline: on three real tasks, async averaged 9.7 s to completion against 13.75 s synchronous, about 30 percent faster, and completed 19 cycles in a fixed window against 9. Success rates averaged 73.3 percent async against 78.3 percent sync, with sorting notably worse in async. Faster is not automatically better, and the paper does not claim it is.

There is a third option beyond tuning g. Real-time chunking (Black, Galliker and Levine, NeurIPS 2025) freezes the steps of the previous chunk that will have executed by the time the new one lands and inpaints the rest, so the two join without a seam. GR00T exposes the primitive as rtc_overlap_steps, rtc_frozen_steps and rtc_ramp_rate on the action head, but its own deployment guide says it is not wired into the policy or the server-client path and has no ready-made example. A research option, not a flag you can set.

Standing one up, end to end

1. 1
   Prove the transport before you trust the model

   Start the server in replay mode. GR00T's ReplayPolicy serves actions straight out of a recorded dataset with no checkpoint loaded, so a failure here is definitely networking and definitely not your policy.

   bash

   uv run python gr00t/eval/run_gr00t_server.py \
       --dataset-path demo_data/cube_to_bowl_5 \
       --modality-config-path examples/SO100/so100_config.py \
       --execution-horizon 8 \
       --host 0.0.0.0 --port 5555

2. 2
   Ping it from the robot machine, not from the server

   Half of all policy-server problems are a firewall rule. The ping endpoint takes no input, so it is the cheapest reachability test there is. Cameras are a separate failure mode.

   python

   from gr00t.policy.server_client import PolicyClient
   
   client = PolicyClient(host="10.0.0.42", port=5555, timeout_ms=3000)
   print(client.ping())          # True, or a 3 s stall then False
   print(client.get_modality_config())   # what the server expects to receive

3. 3
   Send one real observation and time the round trip

   Before you connect any servos, send one observation built the way your adapter builds it and time it. If that number is not comfortably inside your chunk duration, nothing downstream fixes it.

   python

   import time, numpy as np
   
   obs = adapter.obs_to_policy_inputs(fake_observation)
   t0 = time.perf_counter()
   action, info = client.get_action(obs)
   print(f"round trip {(time.perf_counter() - t0) * 1000:.1f} ms")
   print({k: v.shape for k, v in action.items()})

4. 4
   Shrink the payload before you tune anything else

   Resize on the client to whatever the checkpoint was trained on, cast to uint8, and JPEG-encode if the link is not local. In our measurement this took two 640x480 frames from 1800 KiB to 185 KiB.

   python

   from openpi_client import image_tools
   
   img = image_tools.convert_to_uint8(
       image_tools.resize_with_pad(raw_frame, 224, 224))
   
   # or, for a link where every KiB counts
   ok, buf = cv2.imencode(".jpg", img, [int(cv2.IMWRITE_JPEG_QUALITY), 85])

5. 5
   Watch the action queue, not the loss curve

   LeRobot plots the queue size at runtime. A sawtooth that touches zero is the picture of a robot that stops on every cycle. Raise actions_per_chunk first, then chunk_size_threshold.

   bash

   python -m lerobot.async_inference.robot_client \
       ... \
       --actions_per_chunk=50 \
       --chunk_size_threshold=0.6 \
       --debug_visualize_queue_size=True

6. 6
   Add the safety layer the frameworks do not give you

   None of the three servers check that the action they return is reachable. The GR00T deployment guide asks for soft joint limits, workspace bounds and a bound e-stop key. Add them before the arm moves at speed.

   python

   for step in action_chunk:
       step = np.clip(step, JOINT_MIN, JOINT_MAX)
       if np.max(np.abs(step - last)) > MAX_DELTA_PER_TICK:
           raise RuntimeError("policy jumped; stopping")
       robot.send_action(step)
       last = step
       time.sleep(1 / 30)

ssh -N -L 5555:localhost:5555 user@gpu-box   # the cheapest way to not be exposed

The failure modes you will actually hit

• Wrong or swapped camera keys. The server expects front and wrist; your config calls them laptop and phone, or the wrist view lands in the scene slot. Same shape, same dtype, no error, and the arm just drifts. Check get_modality_config() against your adapter. This is why a policy that only works in one setup is usually plumbing rather than the model.
• The client hangs for fifteen seconds. GR00T's PolicyClient defaults to timeout_ms=15000. Drop it to two or three seconds for a real loop so a dead server surfaces as an exception rather than a freeze.
• Clock skew. LeRobot logs one-way latency by subtracting the client's time.time() from the server's. If the two machines are not synchronised, that number is fiction, and it can be negative.
• Silent throttling. LeRobot's server sleeps at the end of GetActions to hold the reply rate down to inference_latency, default 1/30 s. If your model is faster than that, you are not measuring the model.
• The observation queue drops your frame. The server's queue has maxsize=1 and near-duplicate observations are filtered by joint-space distance. An arm holding still can have its observations discarded, by design.
• Version drift. Isaac-GR00T renamed --action-horizon to --execution-horizon, moved the VLM backbone from Eagle to Cosmos-Reason2-2B, and expanded the action horizon from 16 to 40 in the N1.7 release. A copied command from an older tutorial fails on an unrecognised flag if you are lucky and on a shape mismatch if you are not.

A policy server is an unauthenticated RPC endpoint

Every one of these servers deserialises attacker-controlled bytes for a living. That is not a hypothetical concern; it has already gone wrong twice in the same repository.

The current code shows the fix. GR00T's MsgSerializer now drives msgpack directly and refuses object-dtype ndarrays on encode and decode, because msgpack_numpy would otherwise hand pickle a payload straight off the wire. openpi's msgpack_numpy does the same and says why in its docstring: msgpack is secure, as opposed to pickle or dill, which allow arbitrary code execution. LeRobot's async transport still pickles both observations and actions, with a # nosec comment at every call site.

• Bind to 127.0.0.1 and tunnel. ssh -N -L 5555:localhost:5555 costs nothing and removes the entire attack surface.
• If you must bind to 0.0.0.0, put it behind WireGuard or a VPC, never a public IP with a port forward.
• GR00T's optional api_token is compared as a plaintext field inside the request dict. It stops a curious scanner, not an attacker on the path.
• openpi's Authorization: Api-Key header is a real header, but the default scheme is ws://, not wss://. Terminate TLS in front of it.
• Treat a checkpoint from an untrusted source the same way you treat the wire. Checkpoints are deserialised too.

Where the client-server split does not help

The split moves compute; it does not remove latency. If your task needs the end effector to react inside one or two control ticks, no amount of chunking or protocol tuning rescues it, because what the policy is acting on is already a full round trip old. That is a property of the loop, not the framework, which is why the honest answer for fast reactive motion is to put the GPU next to the servos.

And sometimes you do not need a server at all. If the GPU and the servo bus share a machine, instantiate the policy object directly; the GR00T deployment guide lists that as inference mode one, ahead of the ZMQ architecture. A local ACT policy at 20 ms per step on a 24 GB card gains nothing from a socket except a serialisation step and a new way to fail.

To see the loop before you build one, the live arm streams a physical SO-100 with no signup, and the three ways to start covers the options if you have no hardware yet. The VLA overview and the SO-100 setup guide cover the two ends of the same pipeline.