Raspberry Pi as a Robot Host: What It Can and Cannot Do

A Raspberry Pi is a very good robot host and a very bad inference machine. Those two facts are not in tension, they are the whole design. The Pi sits next to the arm, owns the serial bus and the cameras, keeps a fixed-rate loop running, and talks to something else that owns the GPU. Get that split right and a Pi 5 will drive an SO-100 all day. Get it wrong and you spend a weekend discovering that a 3 B parameter model does not fit on a board with no CUDA device.

This page covers the boring half: which USB ports exist on which Pi, how much bandwidth two cameras need, where the power comes from, and what the published hardware requirements say when you read them literally. Every number here comes from a datasheet, a repo or a doc page linked at the bottom. Where the answer is do not run it here, it says so.

What the Pi is actually doing

Split the software into jobs and the placement decision answers itself. Some jobs need to be microseconds from a UART. Some need 40 GB of VRAM. Nothing needs both, which is lucky, because no single box at this price gives you both.

That table is the article. The rest turns each yes into a working setup and each no into an architecture instead of a disappointment. If the arm has never moved at all, start with the SO-100 getting started guide and come back when the servos respond.

USB topology is the whole design

Both boards advertise the same port count: two USB 3.0 and two USB 2.0. The topology behind those ports is completely different, and that difference decides whether your second camera streams.

Before debugging anything, look at the tree. One command tells you what the kernel thinks your topology is:

# what is on which controller, at which speed
lsusb -t

# the shape to expect on a Pi 5 with one camera and the servo adapter
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Video, Driver=uvcvideo, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    |__ Port 2: Dev 3, If 0, Class=Comm, Driver=cdc_acm, 12M
    |__ Port 2: Dev 3, If 1, Class=CDC Data, Driver=cdc_acm, 12M

Camera bandwidth: do the arithmetic before you buy

USB video class cameras reserve a fixed slice of bus bandwidth when the stream starts. If the slice is not available, the stream does not start. This is the most common way a two-camera rig fails, and it is predictable from multiplication.

USB 2.0 signals at 480 Mbit/s, and the periodic schedule only gives you part of that. A single uncompressed 1080p30 stream asks for roughly twice the whole bus. Two uncompressed 640x480 streams want about 294 Mbit/s on a bus that also carries servo traffic. The MJPEG rows are estimates, since the ratio depends on sensor and scene, but the order of magnitude is not in doubt: in-camera MJPEG is the difference between two cameras working and two cameras not working.

# what formats and frame rates does this camera really support?
v4l2-ctl --device /dev/video0 --list-formats-ext

# confirm what the driver actually negotiated
v4l2-ctl -d /dev/video0 --get-fmt-video

# usbfs buffer ceiling, in MB (16 by default)
cat /sys/module/usbcore/parameters/usbfs_memory_mb

# LeRobot's own discovery helper
lerobot-find-cameras opencv

Whatever you record is what the policy expects at inference time. A wrist camera at 15 fps during recording and 30 fps during a rollout is a changed input distribution. Fix format and rate once, write them down, and use the same values in the recording run and in the rollout. If a camera vanishes mid-session, the camera-not-detected page lists the usual causes.

The servo bus: one serial port at 1 Mbaud

The arm is a daisy chain of Feetech STS3215 bus servos on a half-duplex TTL line, bridged to USB by a small adapter board. The servo spec lists a 1 Mbps bus and LeRobot's Feetech driver defaults to exactly that. At 1 Mbaud with 8N1 framing a byte is ten bit periods, so a byte costs 10 microseconds and reading six joints is a few hundred microseconds of signalling. That is not your problem. Everything around it is: USB scheduling, driver buffering, and the occasional corrupt packet.

LeRobot is explicit about the last one. The SO follower configuration carries num_read_retries: int = 2, with a comment that Feetech buses can return a corrupted status packet "especially when several joints move at once", which would otherwise abort the control loop (config_so_follower.py). Write your own loop instead and it will die on a fast move.

1. 1
   Find the port

   Run the finder with the adapter plugged in, unplug when prompted, and it tells you which device node is yours. On Linux this is normally a /dev/ttyACM* node, because the adapter enumerates as USB CDC-ACM rather than as an FTDI part.

   bash

   lerobot-find-port
   # Finding all available ports for the MotorBus.
   # ['/dev/ttyACM0', '/dev/ttyACM1']
   # Remove the usb cable from your MotorsBus and press Enter when done.
   # The port of this MotorsBus is /dev/ttyACM1

2. 2
   Give yourself access, then stop doing it by hand

   LeRobot's docs show sudo chmod 666 /dev/ttyACM0. That works once and is undone by the next replug. Add yourself to the dialout group instead and log out and in.

   bash

   sudo usermod -aG dialout $USER
   # log out and back in, then verify
   id -nG | tr ' ' '\n' | grep dialout

3. 3
   Pin the device name so it survives a reboot

   With a leader and a follower plugged in, ACM0 and ACM1 swap depending on enumeration order. Use the stable symlink under /dev/serial/by-id/, or write a udev rule keyed on the adapter's serial number.

   bash

   ls -l /dev/serial/by-id/
   
   # find the attributes to key a rule on
   udevadm info -a -n /dev/ttyACM0 | grep -E 'idVendor|idProduct|serial' | head
   
   # /etc/udev/rules.d/99-lerobot.rules  (substitute your own values)
   # SUBSYSTEM=="tty", ATTRS{idVendor}=="XXXX", ATTRS{serial}=="YYYY", SYMLINK+="robot_follower"
   
   sudo udevadm control --reload-rules && sudo udevadm trigger

4. 4
   Calibrate, on the Pi, over SSH

   Calibration writes a per-robot file that later runs load by id. Do it on the machine that will drive the arm, and give the arm a name you will still recognise in six months.

   bash

   lerobot-calibrate \
       --robot.type=so101_follower \
       --robot.port=/dev/serial/by-id/usb-...-if00 \
       --robot.id=pi_follower_01

5. 5
   Prove the loop before you add cameras

   Teleoperate with no cameras configured first. If that is smooth, the bus and the power are fine and any later stutter is video or network.

   bash

   lerobot-teleoperate \
       --robot.type=so101_follower \
       --robot.port=/dev/serial/by-id/usb-...-if00 \
       --robot.id=pi_follower_01 \
       --teleop.type=so101_leader \
       --teleop.port=/dev/serial/by-id/usb-...-if01 \
       --teleop.id=pi_leader_01

A note on serial latency, since it gets misquoted

You will find advice telling you to write 1 into latency_timer to fix a slow servo bus. The knob is real: the Linux ftdi_sio driver exposes a read-write latency_timer sysfs attribute in units of milliseconds, and the kernel header records FTDI's device default as 16 ms (ftdi_sio.h). It exists only for FTDI parts, which appear as /dev/ttyUSB*. The board LeRobot documents for the SO-100 family appears as /dev/ttyACM* under cdc_acm, where the attribute does not exist. Measure your own loop instead of copying a fix for hardware you do not have.

import time, statistics
from lerobot.robots.so_follower import SO101Follower, SO101FollowerConfig

robot = SO101Follower(SO101FollowerConfig(port="/dev/serial/by-id/usb-...-if00", id="pi_follower_01"))
robot.connect()

samples = []
for _ in range(500):
    t0 = time.perf_counter()
    robot.get_observation()
    samples.append((time.perf_counter() - t0) * 1000)

samples.sort()
print(f"median {statistics.median(samples):.2f} ms  p99 {samples[494]:.2f} ms")
robot.disconnect()

Power: the Pi is not a power supply

The servo adapter board has its own DC barrel jack, and that is where servo current comes from. The USB cable carries data. People still try to skip the brick and run the arm off the Pi. The current budget says no, and it is not close.

Waveshare publishes those currents for the 12 V ST3215, not for the 7.4 V STS3215 that the SO-100 actually uses, so take them as the order of magnitude rather than as your servo's datasheet. The conclusion survives either way: one stalled servo draws more than a Raspberry Pi 5 will hand to every USB peripheral combined, and an arm has six. Give the bus its own supply, sized for the servo voltage. Then check the Pi is on a real 5 V 5 A USB-C power delivery supply: on anything less it clamps downstream USB to 600 mA, and a camera browning out mid-episode looks exactly like a bad cable.

Why inference does not belong on the Pi

Here are the published requirements without commentary. NVIDIA's Isaac-GR00T README lists inference hardware as one GPU with 16 GB or more of VRAM, naming RTX 4090, L40, H100, Jetson AGX Thor and Orin, and DGX Spark; for fine-tuning it recommends 40 GB or more. LeRobot's async guide says Pi0 "occupies 14GB of memory at inference time, while SmolVLA requires only ~2GB". A Raspberry Pi 5 has at most 16 GB of LPDDR4X shared with the OS, the camera buffers and whatever is encoding video in software, and no CUDA device.

Read the ACT row carefully, because it is the one that tempts people. Twenty milliseconds per action step is a 50 Hz budget, and it is a GPU number. Moving the same 80 M parameter model to four Cortex-A76 cores with no accelerator is not shaving that budget, it is a different regime. Nobody has published a credible real-time ACT rollout on Pi-class CPU, so treat it as unproven rather than as a plan. Compare the five on the policies page if you are still choosing.

One genuine caveat. The paper behind SmolVLA says the model is "designed to be trained on a single GPU and deployed on consumer-grade GPUs or even CPUs". That is about x86 desktop CPUs, not a 2.4 GHz ARM board, and deployable is not the same as deployable at your control rate. Test it and publish numbers rather than assuming them. Its benchmark results sit next to 84 other models in the arena.

But what about an AI HAT?

Raspberry Pi documents the answer in a capability table. AI HAT+ is a Hailo-8L at 13 TOPS or a Hailo-8 at 26 TOPS, both INT8, and both use the Pi 5's own memory rather than carrying any. That table marks both large language model and vision-language model support as not supported for AI HAT+, and gives its use cases as object detection, camera post-processing, robotics and moderate neural workloads. The newer AI HAT+ 2 is a Hailo-10H at 40 TOPS, INT4, and the real difference is not TOPS: it carries its own 8 GB, which the docs say lets it run LLMs and VLMs "up to ~6 billion parameters".

Read that ceiling carefully. It is a real capability, and it is a capability for models Hailo's toolchain can compile and quantize. A vision-language-action model is a VLM with an action head trained on your robot's joint space, served through a runtime that expects a PyTorch device. That is not a supported path on a Hailo part today, and the checkpoint you produce by fine-tuning is not a drop-in for it. Memory stopped being the obvious blocker on that board; the toolchain became the blocker instead.

The architecture that actually works: Pi as robot client

LeRobot ships the answer, and their framing is worth reading literally. The RobotClient "streams observations to the PolicyServer, and receives action chunks obtained running inference on the server (which we assume to have better computational resources than the robot controller)". The Pi is the robot controller. Something with a GPU is the server. This is the supported path, not a workaround.

# on the GPU machine
python -m lerobot.async_inference.policy_server \
     --host=0.0.0.0 \
     --port=8080

# on the Raspberry Pi, next to the arm
python -m lerobot.async_inference.robot_client \
    --server_address=<gpu-host>:8080 \
    --robot.type=so100_follower \
    --robot.port=/dev/serial/by-id/usb-...-if00 \
    --robot.id=pi_follower_01 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \
    --task="Pick up the cube and put it in the bin" \
    --policy_type=smolvla \
    --pretrained_name_or_path=<user>/<model> \
    --policy_device=cuda \
    --actions_per_chunk=50 \
    --chunk_size_threshold=0.5 \
    --aggregate_fn_name=weighted_average \
    --debug_visualize_queue_size=True

The mechanism that makes this survivable is action chunking. The server returns a batch of future actions, the Pi steps through them locally, and a new observation goes out when the queue drops below the threshold. You pay the round trip once per chunk instead of once per step, and you pay for it in open-loop blindness for the length of the chunk. Hugging Face's async inference post reports sub-100 ms round trips on a local network with SmolVLA on an RTX 4090, and about a 2x speedup in task completion time at comparable success rates.

Note the phrase local network. A GPU box on the same switch is a different animal from a rented GPU across the public internet, and that difference in inference latency is the whole design constraint. Slow, deliberate pick and place tolerates the hop. Fast reactive motion does not. If a policy stalls part way through a motion, the freeze page walks the causes.

Recording datasets on the Pi

Recording is where a Pi is closest to its limit, because it is the only stage that is genuinely compute-bound on the host. LeRobot encodes each camera stream to MP4, and the default encoder is libsvtav1 at preset 12, crf 30, GOP 2. Software AV1 on four ARM cores while two cameras stream and a control loop runs is asking a lot, and there is no hardware encoder to fall back on: the Pi 4 brief lists H.264 1080p30 encode, the Pi 5 brief lists a 4Kp60 HEVC decoder and no encoder at all.

lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/serial/by-id/usb-...-if00 \
    --robot.id=pi_follower_01 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30} }" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/serial/by-id/usb-...-if01 \
    --teleop.id=pi_leader_01 \
    --dataset.repo_id=<user>/<dataset> \
    --dataset.num_episodes=50 \
    --dataset.single_task="Grab the cube" \
    --dataset.streaming_encoding=true \
    --dataset.encoder_threads=2 \
    --dataset.rgb_encoder.vcodec=h264 \
    --dataset.rgb_encoder.preset=fast \
    --display_data=false

Whatever you record ends up as a LeRobot dataset, and the format version matters downstream: v3.0 for Pi0.5, SmolVLA and ACT, v2.0 or v2.1 for GR00T, which crashes on a v3.0 dataset and needs it converted down. A training-time problem rather than a Pi problem, but worth knowing before you record fifty episodes. Public examples live in the dataset directory.

Two ways to get a policy running next to your arm

Same goal, two paths: a trained policy driving your SO-100 with the Pi as host. One is entirely yours, one rents the GPU parts. The Pi's job is identical in both.

Which Pi, and when to stop using a Pi

Buying today for one SO-100 with two cameras: a Raspberry Pi 5 with 8 GB, a 5 V 5 A USB-C power delivery supply and active cooling. The independent USB controllers are the reason, not the CPU. A Pi 4 works with one camera and is painful with two.

• Pi 4, one camera, MJPEG. Fine for teleoperation and a first dataset. Everything shares one VL805 uplink, so expect a fight for the second stream.
• Pi 5, two or three cameras. The sensible default. Put each camera on a different root hub and confirm with lsusb -t.
• Pi 5 plus AI HAT+. Worth it for local detection or segmentation as a preprocessing stage. Not a way to run a VLA.
• Small x86 box. Higher resolution, more than three cameras, or a fully local policy: an x86 mini PC removes the ARM wheel problem and the encoder problem at once.
• Jetson-class module. The only edge option with a CUDA device, and the one NVIDIA names in the Isaac-GR00T inference list. A different price bracket.

Whatever you pick, the split stays the same. The host owns time and I/O, the GPU owns the policy, and the wire between them is the thing you measure. To feel the loop before buying anything, the live arm is a physical SO-100 you can drive from a browser with no signup, and the try page lays out the three starting points. The SO-100 complete guide covers the assembly and calibration this page assumes you have done.