Action Chunking at Inference Time: Queues, Thresholds, Seams

Three integers and one clock

Training a policy is a batch job. Running one is a real-time system, and real-time systems are governed by arithmetic that has nothing to do with loss curves. Action chunking is the reason that arithmetic is tractable at all: instead of one action per forward pass, the model emits a sequence, and the robot has something to do while the next forward pass is running. Everything interesting at run time happens in the gap between the plan you are executing and the plan that is being computed.

The Physical Intelligence real-time chunking paper (arXiv 2506.07339, v2 dated 5 December 2025) gives the cleanest vocabulary for that gap, so this article borrows it. A policy emits a chunk of H actions. You execute s of them before asking for a new chunk. The controller ticks every delta t seconds. The model takes delta seconds to answer. That last quantity is what everyone means by inference latency, and expressed in control steps it is d = floor(delta / delta t). Every failure mode below is a violation of a relationship between those four numbers.

Two constraints matter. The real-time constraint from the RTC paper is d <= s <= H - d: the delay must be shorter than what you still hold, and the execution horizon must leave room for the overlap. The queue constraint from the SmolVLA paper is g >= (delta / delta t) / n, where n is the chunk length the client asked for. Below that value the queue drains faster than the server refills it, and the arm runs dry.

The synchronous loop, and what it does to the arm

Before tuning anything, it helps to see what the simple version does. This is the real-robot client that ships with Isaac-GR00T for the SO-100, trimmed to the loop body. It is the pattern most people start from, and it is fully synchronous.

while True:
    obs = robot.get_observation()
    obs["lang"] = cfg.lang_instruction

    actions = policy.get_action(obs)          # blocks on the policy server

    for i, action_dict in enumerate(_select_action_steps(actions, cfg.action_horizon)):
        tic = time.time()
        robot.send_action(action_dict)
        toc = time.time()
        if toc - tic < 1.0 / 30:
            time.sleep(1.0 / 30 - (toc - tic))

Eight actions at 30 Hz is 267 ms of motion. Then the loop returns to the top, grabs a fresh observation, and blocks. Nothing is sent to the servos during that call. If the round trip is 150 ms you have a 150 ms hole in the command stream every 267 ms, which is a duty cycle of about 64 percent. The arm does not stop dead, because position-controlled servos hold the last commanded target, but it does stop moving, and then it starts again. Whether that commanded target means what you think it means is a separate question, answered by calibration.

Queue depth is the number to watch

The asynchronous version replaces the block with a queue. lerobot's async inference stack splits into a PolicyServer that holds the model and a RobotClient that holds the robot and a queue of timed actions. The client pops one action per tick. Separately, it decides when to send the next observation. The entire decision is one line.

def _ready_to_send_observation(self):
    """Flags when the client is ready to send an observation"""
    with self.action_queue_lock:
        return self.action_queue.qsize() / self.action_chunk_size <= self._chunk_size_threshold

That is the whole control policy for the control policy. Queue depth divided by chunk length, compared against one float. Everything else in the stack is plumbing around this comparison, so it is worth understanding what each end of the range does to the robot before you touch it.

Worked numbers for an SO-100 at 30 Hz

Here is the arithmetic on the five policies AY-Robots trains. The latency column is what the platform publishes on the policies pages as the inference figure for each model. The remaining columns apply the formulas above at 30 Hz with a 50-action chunk, which is the lerobot default for Pi0.5 and SmolVLA.

Two readings fall out of that table. First, the default threshold of 0.5 is generous for every model on this list as long as the GPU sits next to the arm. Second, the margin is eaten by the network, not by the model. Adding a 200 ms public-internet round trip to Pi0.5 takes the minimum threshold from 0.29 to roughly 0.41, and the queue you thought was half full is now the only thing between you and a stall.

How much that measurement can move is worth internalising. NVIDIA publishes an end-to-end timing table for GR00T N1.7 at 4 denoising steps with a single camera: on an H100 80 GB the same model runs 85.8 ms in PyTorch eager, 48.6 ms with torch.compile and 27.9 ms with a full TensorRT pipeline. On a Jetson Orin the same three modes give 354.0 ms and 150.9 ms at the ends of the range. That is a factor of three in d from the runtime alone, before anyone touches a threshold, which is why the head-to-head model comparisons are only a starting point and not a deployment plan.

What actually happens at the seam

Now the interesting part. A new chunk arrives while the old one is still executing. The two chunks overlap in time, and they may disagree. In lerobot the merge is explicit and readable, and it turns on one field: every action carries a timestep index, assigned by the server from the timestep of the observation that produced it.

The client tracks latest_action, the index of the last action it actually sent to the servos. When a chunk lands, any incoming action whose timestep is at or below that index is dropped on the floor. Those are the actions that were overtaken by reality while the model was thinking. Actions with fresh timesteps go straight into the queue. Actions that collide with something already queued are blended.

AGGREGATE_FUNCTIONS = {
    "weighted_average": lambda old, new: 0.3 * old + 0.7 * new,
    "latest_only":      lambda old, new: new,
    "average":          lambda old, new: 0.5 * old + 0.5 * new,
    "conservative":     lambda old, new: 0.7 * old + 0.3 * new,
}

Every one of those is a linear interpolation between two trajectories, and that is the structural weakness. If the old chunk planned to go over an obstacle and the new chunk plans to go under it, the average goes through it. The RTC paper illustrates exactly this bifurcation and notes that naive smoothing is not guaranteed to produce a valid action. Blending is a smoothing filter applied to a multi-modal distribution, and smoothing a bimodal distribution gives you the trough between the modes.

Three ways across the boundary

There are three broad strategies in shipping code today, and they sit at different points on the cost curve. Pick by how much compute you can spend per control step and by whether your checkpoint is flow-based.

1. Blend the overlap

What the section above describes. Cheap, universal, and good enough for slow tasks. This is what you get by default when you run lerobot's async inference with any of the supported policy types: act, smolvla, diffusion, tdmpc, vqbet, pi0, pi05 and groot.

2. Temporal ensembling

The original ALOHA answer, from the ACT paper. Query the policy at every single timestep, so that at any moment you hold many predictions for the same future step, then take an exponentially weighted average with weights w_i = exp(-m * i), where w_0 is the weight on the oldest prediction. Smaller m incorporates new observations faster. The reference implementation uses m = 0.01.

# src/lerobot/policies/act/configuration_act.py, lerobot main
chunk_size: int = 100
n_action_steps: int = 100
temporal_ensemble_coeff: float | None = None   # ACT reference impl uses 0.01

# __post_init__ enforces this:
#   n_action_steps must be 1 when temporal_ensemble_coeff is set,
#   because ensembling needs one forward pass per control step.

Read that constraint carefully, because it is the whole story. Temporal ensembling is free of chunk boundaries because it has no chunk boundaries: it re-plans every tick. For an 80 M parameter ACT model at 20 ms that is affordable at 30 Hz. For a 3 B parameter vision-language-action model it is not affordable at any frequency you would want to drive an arm at, which is why nobody runs Pi0.5 with temporal ensembling.

3. Real-time chunking

The current best answer for flow-matching and diffusion policies, and the one that treats the boundary as a generation problem rather than a filtering problem. Freeze the first d actions of the new chunk to the values you know will execute, then inpaint the remainder so that it is consistent with that frozen prefix. It is an inference-time algorithm, requires no retraining, and lerobot ships it.

# src/lerobot/policies/rtc/configuration_rtc.py, lerobot main
@dataclass
class RTCConfig:
    enabled: bool = True
    mode: str = "guided"                 # "guided" or "trained"
    prefix_attention_schedule: RTCAttentionSchedule = RTCAttentionSchedule.LINEAR
    max_guidance_weight: float = 10.0
    execution_horizon: int = 10

The RTC authors report the method running the same real robot motion about 20 percent faster than synchronous inference while staying smoother than temporal ensembling, and holding up under inference delays above 300 ms, which was more than 30 percent of the model's prediction horizon in their setup. The constraint to remember is d <= s <= H - d. If your delay is large enough that d exceeds H - s, no inference-time trick saves you; you need a shorter delay or a longer horizon.

Run it end to end on an SO-100

Concretely, on a machine with the arm attached and a GPU either local or reachable. lerobot's async stack supports so100_follower, so101_follower, bi_so_follower and omx_follower today. This is the manual path; the platform path is in the next section.

1. 1
   Install the async extra

   The async stack needs gRPC and protobuf, which are not in the base install. Do this in the same environment your policy already runs in.

   bash

   git clone https://github.com/huggingface/lerobot.git
   cd lerobot
   pip install -e ".[async]"

2. 2
   Start the policy server

   The server starts empty. It learns which checkpoint to load during the first handshake with the client, so you do not pass a model path here.

   bash

   python -m lerobot.async_inference.policy_server \
       --host=0.0.0.0 \
       --port=8080

3. 3
   Start the client with the queue plot on

   The first run is a measurement run, not a tuning run. Leave the threshold at the default and watch what the queue does.

   bash

   python -m lerobot.async_inference.robot_client \
       --server_address=127.0.0.1:8080 \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --robot.id=follower_so100 \
       --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}}" \
       --task="put the cube in the bowl" \
       --policy_type=smolvla \
       --pretrained_name_or_path=<your-user>/<your-checkpoint> \
       --policy_device=cuda \
       --actions_per_chunk=50 \
       --chunk_size_threshold=0.5 \
       --aggregate_fn_name=weighted_average \
       --debug_visualize_queue_size=True

4. 4
   Read the queue plot, not the robot

   A healthy queue is a sawtooth that never touches zero and never sits at the top. Touching zero means the arm ran dry and you should raise the threshold or lower fps. Sitting near the top means you are paying for forward passes you are throwing away, and you can lower the threshold.

   bash

   # Same run, more detail on where the milliseconds go.
   # The server logs prepare / preprocess / inference / postprocess per chunk.
   python -m lerobot.async_inference.policy_server \
       --host=0.0.0.0 --port=8080 --fps=30 --inference_latency=0.033

5. 5
   Change exactly one thing

   Threshold first, because it is the cheapest and most reversible. Then actions_per_chunk. Then fps, which is the blunt instrument: dropping from 30 to 15 Hz doubles the wall-clock value of every action in the queue and fixes most stalls at the cost of reactivity.

   bash

   # Queue kept hitting zero: buy more headroom.
   --chunk_size_threshold=0.7
   
   # Still hitting zero at 0.7: the model is simply too slow for 30 Hz here.
   --fps=15

6. 6
   Only then reach for RTC

   If the queue is healthy and the motion is still jerky, the problem is the seam, not the depth. That is what RTC is for, and it only applies to flow-matching and diffusion checkpoints.

   bash

   lerobot-rollout \
       --strategy.type=base \
       --policy.path=<your-user>/<your-checkpoint> \
       --inference.type=rtc \
       --inference.rtc.execution_horizon=10 \
       --inference.rtc.max_guidance_weight=10.0 \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM0 \
       --task="put the cube in the bowl" \
       --duration=120 \
       --device=cuda

Doing it yourself against doing it here

The tuning above is client-side work that nobody can do for you, because the numbers depend on your network, your cameras and your GPU. What differs between the two paths is everything around it: who owns the GPU, who pays for it while you are thinking, and how long it takes to get from a checkpoint to an arm that moves.

The traps that eat a day

• actions_per_chunk larger than the model's own chunk size does nothing useful. The server truncates with chunk[:, :actions_per_chunk, :], so you get the model's length, not yours.
• Camera keys must match what the checkpoint expects, exactly. Not similar, exactly. Check config.json on the checkpoint before you argue with the queue.
• A slow camera is an inference delay. get_observation() is inside the same loop as everything else, and a 1920x1080 USB capture at 30 fps can dominate a small model's forward pass.
• Raising fps to make the motion smoother makes the stall worse, because every queued action is now worth less wall-clock time. Lowering fps is the fix for a dry queue, not raising it.
• The client blocks on nothing, which means a dead server produces a robot that keeps executing a stale plan until the queue drains. Watch the queue, not the arm.
• GR00T's fine-tuning entry point exposes no seed, so two runs of the same configuration are not bit-for-bit identical. If two checkpoints behave differently at the same threshold, that may be the reason and not your tuning.

Where none of this helps

Being honest about the boundary of the technique matters more than the technique. Queue tuning moves a fixed amount of latency around. It does not remove latency, and it cannot manufacture reactivity that the model does not have.

The last one is worth an explicit warning because it costs the most time. If the arm hesitates at the same point in every rollout regardless of threshold, the problem is upstream of inference: check the twitch-then-sag failure mode, confirm the leader-follower recordings were clean, and read policy only works in one setup before you spend another evening on hyperparameters. The failure-mode index is organised by symptom for exactly this reason.

Symptom to knob

If none of the first-column entries match what you are seeing, the problem is probably not at inference time at all. How VLA models are structured and what high-quality training data looks like are the two upstream pieces that most often explain a policy nobody can tune into working. For a hardware-first read of the same arm, the SO-100 complete guide covers the build.