WebRTC Teleoperation Latency: Where Every Millisecond Goes

Latency is not one number

A remote arm feels wrong for two separate reasons, and people routinely conflate them. One is the outbound path: you move the leader, and some milliseconds later the follower moves. The other is the return path: the follower has already moved, and some milliseconds later you see that it did. Teleoperation quality is set by the sum of the two, because your hand is closing a loop through your own eyes. A leader-follower rig on the same desk closes that loop in a few tens of milliseconds and feels like an extension of your arm. The same rig split across a continent does not, and the reason is worth being precise about.

Published work that separates the two halves is rare. The clearest recent example instrumented a teleoperated car with gyroscopes on both steering wheels, a phototransistor watching an LED inside the camera's field of view, and two GPS-synchronised Raspberry Pi 5 units, using low-pass filtering and threshold detection to spot steering motion at each end. It reported end to end latency of approximately 500 ms over commercial 4G and 5G with a stated measurement precision of plus or minus 4 ms, and found that the motion to motion component, control input to vehicle actuation, contributed up to 60 percent of the total. The video half was the smaller half. That is the opposite of what most teams assume when they start optimising.

The stage-by-stage budget, measured

The most useful published breakdown of a WebRTC robot video path was posted by Transitive Robotics on 6 May 2026. They used the infinite mirror method: point a camera at the screen showing its own stream, overlay a high precision timecode, and read the difference between two generations of the timestamp in a single screenshot. Three USB cameras were tested, a RealSense D435, a Logitech C925e and the 2 MP wide-angle camera that ships with the LeRobot SO-101 arm, each on a desktop running Ubuntu 24.04 over USB3, at 30 fps with a 60 Hz display. That third camera makes the numbers directly relevant to anyone driving an SO-100 class arm. The individual readings in their published screenshot were 132, 132, 136 and 132 ms, and the results did not change noticeably between the three cameras.

The four end to end totals were 110 ms for a direct local preview with no encoding at all, 130 ms once H.264 encode and decode were added, 130 ms again for a local WebRTC connection, and 170 ms for a remote WebRTC connection relayed through Oregon from California. Read that sequence again. Adding an entire real-time media stack over a continental link cost 60 ms on top of a local camera preview, and 30 ms of that 60 was the speed of light and the routers in between. If you have no arm of your own to measure, the three no-hardware entry points put a real one in front of you.

# Baseline: camera straight to a window, no encode, no network.
# The timeoverlay burns a millisecond timecode into the frame
# so you can point the camera at the screen and read the delta.
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  image/jpeg,width=640,height=480,framerate=30/1 ! \
  jpegdec ! timeoverlay font-desc="DejaVu Sans, 36" ! \
  autovideosink

Capture: the stage nobody instruments

Two of the terms in that table are pure arithmetic and cannot be tuned away. At 30 fps a frame that happens right after a capture waits up to 33 ms for the next one. At 60 Hz a decoded frame waits up to 17 ms for the next refresh. Halving the frame rate to 15 fps, which the same write-up notes is common in production teleoperation because it cuts bandwidth and compute on the robot, doubles the first term to 67 ms. That is a real trade and it is worth making deliberately rather than by accident.

The rest of capture latency is the camera and the bus, and it is mostly a hardware property. Cheap UVC webcams buffer internally, compress to MJPEG, and hand you frames on their own schedule. This is why teams chasing a sub-100 ms floor end up shopping for GMSL or MIPI cameras rather than for a different streaming protocol. On the browser side there is one software lever worth knowing, and it is normative rather than advisory. The MediaStreamTrack Content Hints specification defines a contentHint of "motion", "detail" or "text" on a video track, and it binds that hint to encoder behaviour: a sender transmitting a track hinted "motion" MUST use the maintain-framerate degradation preference unless the application has set one explicitly. A wrist camera watching an end effector close on an object is motion, not a slide deck, and one word tells the encoder which half of the picture to protect.

Encode, decode, and what you give up first

The measured cost of software H.264 was around 10 ms in each direction, which is small enough that switching codecs to save latency is almost always the wrong lever. Hardware encoders reduce it further, and on a robot that has an H.264 block sitting idle the useful work is making sure negotiation actually lands on it rather than on a software VP9 path. What matters far more than the codec is what the encoder throws away when it cannot hit the target rate, and that is a separate, explicit setting.

If you are building the robot side yourself, the GStreamer webrtcsink element from gst-plugins-rs is the shortest route to a working sender. Its documented defaults are worth reading before you change anything, because several of them are already the right answer for teleoperation and one of them is not.

# Robot side: one camera, H.264 only, its own signalling server.
# Property names and defaults per the gst-plugins-rs webrtcsink docs.
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-raw,width=640,height=480,framerate=30/1 ! \
  videoconvert ! webrtcsink \
    run-signalling-server=true \
    signalling-server-host=0.0.0.0 \
    signalling-server-port=8443 \
    video-caps="video/x-h264" \
    congestion-control=gcc \
    start-bitrate=2000000 \
    max-bitrate=4000000

Transport: what WebRTC is doing to your packets

The part of WebRTC that most affects how a remote arm feels is the congestion controller, because it decides when to throw quality away. The canonical description is the expired IETF draft draft-ietf-rmcat-gcc-02 from July 2016, which is still the reference everyone implements against. It runs two estimators side by side. The delay-based one watches inter-group delay variation, compares it against an adaptive over-use threshold, and signals over-use only once the estimate has stayed above that threshold for longer than overuse_time_th. The loss-based one is blunter: below 2 percent packet loss it increases its estimate by 5 percent, between 2 and 10 percent it holds it unchanged, and above 10 percent it multiplies it by (1 - 0.5p), where p is the loss ratio. The actual sending rate is the minimum of the two.

Note the asymmetry between the last three rows. One burst of cross-traffic costs you 15 percent of your bitrate in a single step, and climbing back is capped at 8 percent per second, so a couple of seconds pass before you are whole again. If your gripper camera goes soft for two seconds every time someone in the building starts a large upload, that is the controller doing its job, not a bug. The draft also bounds the estimate at 1.5 times the measured incoming bitrate, so an encoder that cannot produce the rate being asked of it will not let the estimate run away.

The other transport decision that shows up in the numbers is relaying. When ICE cannot find a direct path, traffic falls back to a TURN relay, which adds the round trip to wherever that relay lives. In the measured test the relay sat in Oregon and the operator in California, and that leg cost about 30 ms. Put the relay on the wrong continent and you have spent more on routing than on your entire encode and decode pipeline combined. If you run your own infrastructure, relay placement is the single highest-leverage thing on this list.

The jitter buffer: where latency hides

WebRTC runs over UDP, so packets can arrive out of order or not at all. The jitter buffer holds frames briefly so they can be reassembled in sequence, and its depth is adaptive: on a clean link it stays small, and on a jittery one it grows to avoid freezes. This is the one stage in the pipeline that will silently spend your latency budget on your behalf, and it is also the one the browser tells you the most about.

The W3C statistics identifiers define jitterBufferDelay as the summed time each frame spent between entering and leaving the buffer, and jitterBufferEmittedCount as the number of frames that came out. Divide the change in one by the change in the other and you have the average buffer delay per frame, which is exactly how the WebRTC specification itself says to measure it. totalProcessingDelay covers reception to decoded, freezeCount and totalFreezesDuration tell you what the buffer was protecting you from, and currentRoundTripTime on the selected candidate pair gives you the network RTT in seconds.

// Paste into the console on the page holding the peer connection.
// Stat names per the W3C webrtc-stats identifiers.
setInterval(async () => {
  const stats = await pc.getStats();
  let line = {};
  stats.forEach(r => {
    if (r.type === 'inbound-rtp' && r.kind === 'video') {
      line.jitterBufferMs =
        (r.jitterBufferDelay / r.jitterBufferEmittedCount * 1000).toFixed(1);
      line.decodeMsPerFrame =
        (r.totalDecodeTime / r.framesDecoded * 1000).toFixed(1);
      line.fps = r.framesPerSecond;
      line.freezes = r.freezeCount;
      line.frozenSec = r.totalFreezesDuration;
    }
    if (r.type === 'candidate-pair' && r.nominated) {
      line.rttMs = (r.currentRoundTripTime * 1000).toFixed(1);
      line.outKbps = Math.round(r.availableOutgoingBitrate / 1000);
    }
  });
  console.log(line);
}, 1000);

If the buffer is the problem, there is a lever. The WebRTC specification defines jitterBufferTarget on the receiver as the application's preferred hold duration in milliseconds, and requires the browser to throw a RangeError if you set it negative or above 4000. It is explicitly a target rather than a setting: the user agent keeps its own minimum and maximum allowed values based on network conditions and memory, and those can change at any time, so read the effect back out of the stats rather than assuming it took. Support is not baseline across browsers either. Setting it low on a stable wired link is a legitimate win. Setting it low on a mobile uplink converts smooth video into a stutter.

1. 1
   Establish the camera floor

   Run the local preview with a timecode overlay and no encoding. Point the camera at the screen and photograph it. The difference between the two visible timestamps is the floor that capture and display impose on everything else.

   bash

   gst-launch-1.0 v4l2src device=/dev/video0 ! \
     image/jpeg,width=640,height=480,framerate=30/1 ! \
     jpegdec ! timeoverlay font-desc="DejaVu Sans, 36" ! \
     autovideosink

2. 2
   Add encode and decode locally

   Same machine, same mirror, but now with H.264 in the path. Subtract step one. On a modern CPU expect roughly 10 ms each way, and less if you route through a hardware encoder.

   bash

   gst-launch-1.0 v4l2src device=/dev/video0 ! \
     video/x-raw,width=640,height=480,framerate=30/1 ! \
     videoconvert ! x264enc tune=zerolatency speed-preset=ultrafast ! \
     avdec_h264 ! timeoverlay font-desc="DejaVu Sans, 36" ! \
     autovideosink

3. 3
   Measure the network alone

   Before adding WebRTC, get the pure propagation cost between the two machines. This number is a physics bill and no protocol change will reduce it.

   bash

   ping -c 100 <operator-host> | tail -2
   # On a relayed path, ping the relay too. Both legs count.

4. 4
   Bring up the WebRTC link and read the stats

   Start the sender, open the viewer, then open chrome://webrtc-internals in a second tab. It shows a peer connection entry only if one actually exists, which also makes it a fast way to check what any hosted product is really doing. Run the getStats snippet above and record jitter buffer delay, decode time per frame, RTT and available outgoing bitrate.

   bash

   gst-launch-1.0 v4l2src device=/dev/video0 ! \
     video/x-raw,width=640,height=480,framerate=30/1 ! \
     videoconvert ! webrtcsink run-signalling-server=true \
       signalling-server-port=8443 video-caps="video/x-h264"

5. 5
   Close the loop with the arm

   Glass to glass is only half the answer. Drive the arm and time the motion to motion half separately: timestamp the leader reading and the follower's reported position, on clocks you trust. LeRobot's teleoperate command is the shortest way to get a real leader-follower loop running on an SO-100 class arm.

   bash

   lerobot-teleoperate \
       --robot.type=so101_follower \
       --robot.port=/dev/tty.usbmodem58760431541 \
       --robot.id=my_awesome_follower_arm \
       --teleop.type=so101_leader \
       --teleop.port=/dev/tty.usbmodem58760431551 \
       --teleop.id=my_awesome_leader_arm

Two channels, two opposite configurations

Video and control are different problems and should not share a configuration. Video benefits from retransmission and from a buffer, because a frame that arrives 40 ms late and in order is better than a frame that is missing. Joint targets are the reverse: a joint target that arrives 40 ms late is worse than useless, because a newer one has already superseded it. Delivering the stale one first is actively harmful.

RFC 8834 says this directly in its discussion of retransmission: senders are required to understand the generic NACK message but may choose to ignore it, and retransmission needs care in interactive real-time applications to ensure that a retransmitted packet still arrives in time to be useful. For a control stream carrying absolute joint positions at 30 to 50 Hz, it never does. On the browser side a data channel is configured once, at creation time, and cannot be reconfigured afterwards. The WebRTC specification defines ordered as defaulting to true, and requires the browser to throw if both maxPacketLifeTime and maxRetransmits are set, so pick exactly one. Be aware that the two references disagree on the exception: the W3C specification says TypeError, while MDN documents it as SyntaxError, so catch broadly rather than matching on the name. If the arm reacts to commands you gave it seconds ago, that is a queue somewhere, not the network, and the arm twitches then sags page covers the mechanical version of the same symptom.

// Control channel: newest sample wins, nothing is ever resent.
const control = pc.createDataChannel('joints', {
  ordered: false,        // default is true; head-of-line blocking is the enemy
  maxRetransmits: 0      // do not pass maxPacketLifeTime as well: it throws
});

// Send absolute targets, never deltas. A dropped delta corrupts state forever;
// a dropped absolute target is corrected 20 ms later by the next one.
setInterval(() => {
  if (control.readyState !== 'open') return;
  if (control.bufferedAmount > 4096) return;   // link is behind, skip a sample
  control.send(JSON.stringify({
    t: performance.now(),
    q: readLeaderJoints()   // six absolute joint angles
  }));
}, 20);   // 50 Hz, the rate ALOHA uses for non-bilateral teleoperation

What the delay actually does to an operator

The engineering question is what number to aim for. The human factors literature gives a fairly consistent answer, and the most directly useful study for manipulation is Rakita, Mutlu and Gleicher at HRI 2020. They built a mimicry-control teleoperation system, measured its baseline end to end latency at approximately 90 ms with a Vive tracking system, and then added onset delay to reach 250, 500 and 750 ms. They picked 250 ms as the lowest step because prior work reports it as the level where performance starts to degrade. They also ran a matched set of conditions where the robot was slowed down by an equivalent amount rather than delayed, by scaling its joint velocity limits.

The RoboTurk row is the honest counterweight, and it is why the answer is not a single threshold. Mandlekar and colleagues used the Cellsim network emulator to run four conditions, and none of them moved the completion time distribution: 2.4 Mbit/s with 20 ms one-way as a baseline, 500 kbit/s at 20 ms, 2.4 Mbit/s at 120 ms, and 500 kbit/s at 120 ms, with Kolmogorov-Smirnov p-values from 0.60 to 0.89 between the picking distributions. They then ran a real stress test, teleoperating from California into a data centre in China roughly 6500 miles away versus one in Oregon roughly 500 miles away. Users still completed the tasks; mean completion times rose by 24 seconds on assembly and 28 seconds on picking. Their tasks were coarse, with block lifting averaging about 16 seconds and bin picking about 110 seconds, and coarse tasks absorb delay. The finer the task, the less true this stays. That distinction is covered in more depth in our write-up of the RoboTurk platform.

Doing it yourself against doing it here

# Sender, on the robot host
gst-launch-1.0 v4l2src device=/dev/video0 ! \
  video/x-raw,width=640,height=480,framerate=30/1 ! \
  videoconvert ! webrtcsink \
    run-signalling-server=true signalling-server-port=8443 \
    video-caps="video/x-h264" congestion-control=gcc

# Control loop, on the same host, driving the arm
pip install aiortc
lerobot-teleoperate \
    --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
    --robot.id=follower --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 --teleop.id=leader

When the latency stops being a network problem

Everything above assumes a human in the loop. Swap the human for a policy and the arithmetic changes shape, because the model itself becomes a stage in the pipeline. Across the five trainable policies on this platform the per-step inference latency spans more than an order of magnitude, from ACT at 20 ms to Pi0.5 at 485 ms, and that spread decides how much network budget you have left over. The head to head numbers are laid out on the GR00T N1.7 against Pi0.5 comparison if you want them side by side.

Physical Intelligence published a breakdown of exactly this for their own serving stack, and the shape matches the video result: on a mobile manipulator the total was 139 ms, of which 97 ms was the model, 21 ms the network, 11 ms image resizing and 9.7 ms everything else. On a static robot the total was 108 ms with the same 97 ms of model and only 6.9 ms of network. They then injected 100 ms and 200 ms of extra delay and scored policies by a throughput metric, the proportion of task substeps completed divided by episode duration, averaged over 6 tasks and 10 episodes each. Their reported result is directional rather than numeric: synchronous inference suffers greatly as inference delay rises, while real-time chunking, which generates the next action chunk while the current one is still executing, was unchanged up to plus 200 ms.

LeRobot ships that method. Deployment on a real robot is lerobot-rollout --strategy.type=base --inference.type=rtc, and the behaviour is tuned through an RTCConfig whose execution_horizon and prefix attention schedule control how much of the previous chunk the new one is guided to agree with. You can rehearse it against recorded data before you point it at an arm, which is the cheap way to find out whether your chunk size and your link are compatible.

What laggy teleoperation does to the dataset

This is the part that matters most on a platform whose point is training policies, and it is the part people skip. Every millisecond of teleoperation delay is recorded into the demonstration, because the operator's compensation for the delay is part of the trajectory. ALOHA is the useful reference here: joint targets are read from the leader arms and sent to the followers at 50 Hz, the servos track those targets with an internal PID controller above 1 kHz, and on a smooth task, sliding open a ziploc bag, the measured position tracking error stayed below 2 cm with an average of 0.68 cm. The paper is careful about why the error spikes elsewhere: during high-force insertion it blames the non-bilateral design, which gives the operator no force feedback, while during sudden acceleration it points to a combination of teleoperation delay and motor torque limits.

That is a small error on a low-latency wired rig. Put 200 ms in the loop and the operator stops tracking and starts predicting, and what lands in the episode is a different behaviour: larger corrections, overshoot followed by pullback, and long pauses while the operator waits to see what happened. A policy trained on that data learns the pauses and the overshoot as if they were the task. The data collection guidance in our data quality guide applies here directly.

• Overshoot and correction cycles that the policy will faithfully reproduce, because they are in every episode.
• Move-and-wait pauses that look like deliberate hesitation to the model and add nothing to the task.
• Timestamp skew between the camera stream and the joint stream, which is worse than latency: it teaches the model that the image lags the action by a fixed amount, and that assumption breaks the moment you deploy on a different rig.
• Systematically slower episodes, which shifts the action magnitude distribution the policy is fitted to.
• Grasp timing errors specifically, because the gripper is the joint most tightly coupled to what the wrist camera is showing.

The practical rule is simple. Record locally, over USB or a LAN, with the operator in the room. Use remote teleoperation for evaluation, for demonstration, for reaching an arm you cannot physically visit, and for coarse tasks where the RoboTurk result holds. Use a local loop for anything you intend to train on. The SO-100 setup guide walks through the local path end to end, and the data collection page covers what a usable session looks like. If a recorded session comes back looking wrong rather than merely slow, the failure-mode index is the faster place to start than a network capture.

A budget you can hold yourself to

A published 5G teleoperated driving evaluation is a good worked example of reading such a number properly, and a warning about false precision. It reports mean glass to glass of 202.41 ms with a standard deviation of 31.56 ms over 100 samples, against a control round trip time of 46.63 ms over 1000 samples, on a commercial non-standalone 5G network at 3.6 GHz carrying H.264 at 30 fps with zero measured packet loss on an iPerf UDP test. On those figures the video half dominates the control half by more than four to one, which is the useful takeaway. But the same paper states that its glass to glass method, filming two clocks side by side and reading the digits by hand, is accurate only to around 50 ms, with individual readings capable of being 99 ms out. The two decimal places are arithmetic on the samples, not precision in the measurement.

That is the habit worth taking from this article: ask for the method before you believe the number, and prefer a measurement you ran yourself over anyone's published figure, including every one quoted above. A stated loop rate is not a latency, an average is not a worst case, and a figure with no described apparatus is not evidence. The teleoperation docs are the place to start if you are doing that here, and run your first policy is where the same question comes back as a control-loop budget rather than a video one.