Monitoring a Running Robot Policy: What to Log Live

A policy that worked in the lab still fails on the bench, and the postmortem always goes the same way. The arm did something odd, somebody hit the power switch, and nobody has a record of what the control loop was doing in the ten seconds before. The information was there. It got thrown away one tick at a time.

This is about what to log while a policy drives an SO-100 class arm (if the arm itself is new to you, start with the SO-100 setup guide): achieved action rate, per-joint deltas, camera frame age, action queue depth, servo telemetry. Everything below was checked against LeRobot itself, and the version you install decides how much of it you get for free.

What a running policy actually hands you

One tick of a base rollout is six timed sections and the library names them: observe reads joints and cameras, process_obs runs the observation processors, infer pulls the next action, send writes goal positions to the bus, telemetry feeds the visualiser, query services the text channel. Those are the literal section labels in rollout/strategies/base.py and core.py, and they come back as a share-of-work breakdown at the end of the run. Most of a profiler, free.

Everything worth monitoring falls out of those six steps plus the motor bus. Per-model latency figures on the policies page set the budget the loop must fit inside, and only one of the five models fits a 30 Hz tick.

Action rate: the rate you asked for versus the one you got

The most useful single number is the gap between --fps and what the loop achieved. Inference latency is not the only thing eating the budget; camera reads usually eat more.

lerobot-rollout \
    --strategy.type=base \
    --policy.path=${HF_USER}/my_act_so100 \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, \
                       wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
    --task="Put the lego brick in the box" \
    --fps=30 \
    --duration=120 \
    --display_data=true 2>&1 | tee run-$(date +%Y%m%d-%H%M%S).log

One warning matters more than the rest. It fires when summed loop-body work for a closed cycle exceeds the 1/fps budget, and it names the three usual suspects:

WARNING Control loop is running slower (24.1 Hz) than the target FPS (30 Hz).
Robot control might be unstable. Common causes are: 1) Camera FPS not keeping up
2) Policy inference (action or text) taking too long 3) CPU starvation

The diagnosis lives in the end-of-run block. It prints from the loop's finally clause, so Ctrl-C still produces it, every episode boundary emits a one-line digest, and the sink defaults to logger.info, so no log level needs raising. An example in the shape the formatter produces:

Cadence summary - whole run, 3 episodes - target 30 Hz (33.3 ms budget per tick): 1747 ticks, 1744 judged
  effective cadence: 28.91 Hz over 60.4 s measured
  ticks over the 33.3 ms work budget: 96/1744 (5.5%) - work mean 21.4 ms, worst 118.7 ms
  ticks with no action to send (inference engine starved): 12 - each commanded nothing and recorded no frame
  loop-body steps (share of measured work):
    observe      mean  12.10 ms - worst  41.30 ms -  56.5% of work - 1747 calls
    process_obs  mean   0.90 ms - worst   3.10 ms -   4.2% of work - 1747 calls
    infer        mean   6.40 ms - worst  22.10 ms -  29.9% of work - 1747 calls
    send         mean   2.00 ms - worst   9.80 ms -   9.3% of work - 1747 calls
    telemetry    mean   0.02 ms - worst   0.30 ms -   0.1% of work - 1747 calls
    query        mean   0.01 ms - worst   0.10 ms -   0.0% of work - 1747 calls
  pacing headroom: 11.9 ms slept per tick on average (max 24.1 ms) - near zero means the loop is saturated

Read it from the bottom. Pacing headroom near zero means a saturated loop. A worst an order of magnitude above mean is a stall, not a slow step, and stalls in observe are almost always a camera. Twelve starved ticks in two minutes is a policy server that cannot keep up, a different fix than a slow local loop.

Joint deltas: the cheapest anomaly detector you will ever write

A per-joint step delta is the difference between consecutive Present_Position reads, in degrees. You already have the readings, and two patterns cover most bad rollouts. A joint pinned at a limit: delta collapses to near zero while the commanded goal keeps moving, so the servo loads up against a stop. A chunk-boundary jump: delta spikes on the tick the policy hands over to a new action chunk.

# Attach to any loop that has an observation dict. Runs in microseconds.
import collections

WINDOW = 30            # one second at 30 Hz
JUMP_DEG = 6.0         # per-tick delta that should never happen on an SO-100
STUCK_DEG = 0.15       # below this the joint is not really moving

hist = collections.defaultdict(lambda: collections.deque(maxlen=WINDOW))
prev = {}

def check(obs: dict, t: float) -> list[str]:
    alerts = []
    for key, value in obs.items():
        if not key.endswith(".pos"):
            continue
        if key in prev:
            d = abs(value - prev[key])
            hist[key].append(d)
            if d > JUMP_DEG:
                alerts.append(f"{t:8.2f}s JUMP  {key:24s} {d:6.2f} deg in one tick")
            elif len(hist[key]) == WINDOW and max(hist[key]) < STUCK_DEG:
                alerts.append(f"{t:8.2f}s STUCK {key:24s} < {STUCK_DEG} deg for {WINDOW} ticks")
        prev[key] = value
    return alerts

Camera health is frame age, not device state

A USB camera that has stopped producing frames still reports as open and still returns an array. The policy sees a photograph of the past. LeRobot's OpenCV camera carries the timestamps you need: a background thread writes latest_frame and latest_timestamp together under a lock, and the read paths enforce different freshness contracts.

So log the age of every frame you hand the policy, not just whether the read succeeded. Eleven tolerated failures means a camera can degrade for roughly four tenths of a second at 30 fps before anything raises, and a frozen camera never fails a read at all:

import numpy as np

def frame_health(frame, prev_frame, capture_ts, now, fps=30):
    age_ms = (now - capture_ts) * 1e3
    mad = float(np.mean(np.abs(frame.astype(np.int16) - prev_frame.astype(np.int16))))
    mean_lum = float(frame.mean())
    return {
        "age_ms": age_ms,
        "stale": age_ms > 2.5 * (1000.0 / fps),   # ~83 ms at 30 fps
        "frozen": mad < 0.5,                       # identical frames, sensor stalled
        "black": mean_lum < 8.0,                   # exposure collapsed or lens capped
        "blown": mean_lum > 245.0,                 # auto-exposure ran away
    }

Queue depth, for anything bigger than ACT

Per the policy comparison, ACT needs about 20 ms per action step, GR00T N1.7 152 ms, SmolVLA 245 ms, Pi0.5 485 ms. Only ACT fits inside a 33.3 ms tick. Every vision-language-action model has to execute one chunk while the next is computing, which makes queue depth a first-class signal. A queue that hits zero is a robot that stops mid-motion.

Both LeRobot paths expose it. Async inference runs a PolicyServer and a RobotClient over gRPC and plots the queue for you; the RTC backend keeps chunk generation on a daemon thread, asks for a new chunk only once the queue has drained to --inference.queue_threshold or below, and reports empty pulls as starved ticks.

# Path A: async inference, gRPC policy server plus robot client
python -m lerobot.async_inference.policy_server --host=0.0.0.0 --port=8080

python -m lerobot.async_inference.robot_client \
    --server_address=10.0.0.5:8080 \
    --robot.type=so100_follower --robot.port=/dev/ttyACM0 --robot.id=follower_so100 \
    --policy_type=smolvla --pretrained_name_or_path=${HF_USER}/smolvla_task \
    --policy_device=cuda \
    --actions_per_chunk=50 \
    --chunk_size_threshold=0.5 \
    --aggregate_fn_name=weighted_average \
    --debug_visualize_queue_size=True

# Path B: RTC inside lerobot-rollout, starvation shows up in the cadence summary
lerobot-rollout --strategy.type=base --inference.type=rtc \
    --inference.rtc.execution_horizon=10 \
    --inference.rtc.max_guidance_weight=10.0 \
    --inference.queue_threshold=30 \
    --policy.path=${HF_USER}/pi05_task \
    --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
    --task="Pick up the cube" --duration=120 --device=cuda

Servo telemetry: load, voltage, temperature

The Feetech STS3215 servos expose more than position, and LeRobot's control table maps the whole SRAM block. Poll these on a slow side channel, once a second, never inside the control loop: every read costs bus bandwidth your position reads need.

One read failure here is not news: the follower retries a failed sync_read twice by default (num_read_retries = 2) because Feetech buses return a corrupted status packet now and then. ROS solved the rest of this years ago: diagnostic_updater publishes device health on its own channel at its own rate, default period 1.0 seconds. Fast loop for control, slow loop for health, never both in one thread.

The early signs of a run about to fail

Everything above is mechanical health. Whether the policy itself is going wrong is harder, and it has become a real research area over the last two years. The shared insight: none of them need failure data, because the policy's own action distribution already carries much of the warning, and they calibrate on successful rollouts alone. Sentinel is the one that also watches the video, with a VLM for the slower task-progression failures.

The version worth building on a hobby setup is narrower: keep the last two chunks that overlap in time and take the mean absolute difference across the overlap. A policy on track largely agrees with itself. One losing the plot disagrees, several hundred milliseconds before the arm does anything visibly wrong.

A monitor you can build this afternoon

A structured log line per tick, a slow health thread beside it, a script that reads the result. The point is having the trace when something goes wrong, not building a platform.

1. 1
   Record a clean baseline first

   One teleoperated episode, same cameras and arm, no policy; keep the cadence summary. lerobot-teleoperate defaults to --fps=60, so pass 30 or the baseline is not comparable, and its sections read observe / teleop / send / telemetry. See record your first dataset.

   bash

   lerobot-teleoperate \
       --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
       --teleop.type=so100_leader --teleop.port=/dev/ttyACM1 \
       --fps=30 2>&1 | tee baseline.log
   
   grep -A 12 'Cadence summary' baseline.log

2. 2
   Turn on debug logging where it pays, not everywhere

   The follower emits per-read timings at DEBUG (read state: 4.2ms) plus one line per camera; the RTC engine emits RTC inference latency=...s, queue=..., queue depth for free. Root-level DEBUG drowns you in library noise, so scope it.

   python

   import logging
   logging.getLogger("lerobot.robots").setLevel(logging.DEBUG)
   logging.getLogger("lerobot.cameras").setLevel(logging.DEBUG)
   logging.getLogger("lerobot.rollout.inference.rtc").setLevel(logging.DEBUG)

3. 3
   Write one JSON line per tick

   One line, one tick, machine readable. Never write from the control thread: queue it and let another thread hit the disk.

   python

   import json, queue, threading
   
   q: queue.Queue = queue.Queue(maxsize=10000)
   
   def writer(path):
       with open(path, "a", buffering=1) as fh:
           while (rec := q.get()) is not None:
               fh.write(json.dumps(rec) + "\n")
   
   threading.Thread(target=writer, args=("ticks.jsonl",), daemon=True).start()
   
   def log_tick(t, obs, action, frame_ages, queue_depth):
       q.put_nowait({
           "t": round(t, 4),
           "pos": {k: round(v, 3) for k, v in obs.items() if k.endswith(".pos")},
           "act": {k: round(v, 3) for k, v in action.items()},
           "cam_age_ms": {k: round(v, 1) for k, v in frame_ages.items()},
           "queue": queue_depth,
       })

4. 4
   Poll servo health on a slow thread

   Once a second, off the control thread: the ROS diagnostics pattern in twenty lines. If reads here fail, suspect the bus before the policy.

   python

   import time
   
   def health_loop(bus, motors, stop, period=1.0):
       while not stop.is_set():
           rec = {"t": time.time(), "kind": "health"}
           for m in motors:
               rec[m] = {
                   "load": bus.read("Present_Load", m),
                   "volt": bus.read("Present_Voltage", m),
                   "temp": bus.read("Present_Temperature", m),
               }
           q.put_nowait(rec)
           stop.wait(period)

5. 5
   Keep a rolling window, not the whole run

   LeRobot's highlight strategy does this: a bounded ring buffer of the last N seconds, flushed to a LeRobot dataset episode on a key press, with a second key to push it.

   bash

   lerobot-rollout \
       --strategy.type=highlight \
       --strategy.ring_buffer_seconds=30 \
       --strategy.ring_buffer_max_memory_mb=2048 \
       --strategy.save_key=s \
       --strategy.push_key=h \
       --policy.path=${HF_USER}/my_policy \
       --robot.type=so100_follower --robot.port=/dev/ttyACM0 \
       --dataset.repo_id=${HF_USER}/failure_clips \
       --dataset.single_task="Pick up the red cube"

Two ways to get this running

What monitoring will not do for you

A monitor is a witness, not a safety system, and the gap between the two has bitten people who assumed otherwise.

• It does not stop the arm. By the time a Python check fires, the goal position is on the bus. Physical limits, a low max_relative_target and a reachable power switch are the safety layer.
• It does not fix a bad dataset: a mislabelled wrist camera makes every rollout look healthy and behave wrong. See collecting high-quality VLA training data.
• It does not make remote inference viable for fast tasks. The control loop is 20 to 485 ms per action step depending on the model, and a public-internet round trip turns a working policy into a hesitant one.
• A per-tick log is not a metrics pipeline: dashboards across many arms want a time-series database.
• Alarms without a recovery policy just produce a stopped robot, and what follows the alarm is still open in the literature.