Driving an SO-100 Arm from an AI Agent over MCP

An agent that can read your code can also read a robot arm's joint angles, and if you wire it up it can write them back. This is the wiring: a coding agent on one side, a physically connected SO-100 on the other, and a local MCP server between them that exposes nine tools. No policy is trained, nothing is imitated, and no teleoperation session runs. The agent reads the state of the arm and commands single joints, and everything it is allowed to do is bounded by the same backend that the desktop client uses.

Every command, path and tool name below is taken from the shipping code rather than from a plan. Where a limit exists it is named with its actual value, and where something does not exist it says so instead of being sketched.

What Anthropic announced, and what this article actually uses

Anthropic presented the Model Hardware Standard on 27 August 2026: a standard meant to let AI agents operate physical devices in research and manufacturing safely. The first phase is a research preview rather than a general release. The applications named in the reporting are laboratory work - drug discovery at Genentech, where the interesting part is error handling at runtime, and imaging at the HHMI Janelia Research Campus, where a job that took weeks came down to a day. Hugging Face is named in the announcement among the early adopters, "adding MHS support in LeRobot, their robotics library"; no date and no arm model are given for that. What the specification covers, and what it still leaves open, is sorted out in our walkthrough of MHS; our own run inside that research preview, where an agent measured an SO-ARM101's workspace and wrote the motion itself, is in the SO-ARM101 report.

ay-robots took part in that research preview. That work is a separate thing from this article and should not be read into it: it ran on an SO-ARM101, nothing was trained, and the agent measured the workspace itself and wrote the motion from those measurements. Calibration used no checkerboard and no camera intrinsics - torque drops, a human places the closed gripper on 16 points that the software draws in the camera view, and the robot reads back where they are. The reported figures are 4.1 mm position accuracy and 3.0 mm placement accuracy; the best run placed 12 stones and formed all four colour groups. A fully hands-off run is still outstanding, and the code is not published yet.

Two MCP servers carry the ay-robots name, and only one touches hardware

This trips people up often enough that it is worth 60 seconds. There are two servers. The hosted one answers at www.ay-robots.com/api/mcp, knows the documentation, your datasets and your training jobs, and has no path to a physical arm at all. The local one is a process on the machine the arm is plugged into, started by the CLI, speaking JSON-RPC over stdin and stdout. Only the local one can move a servo.

Most people wire up both, in the same client, and let the agent pick. The finished configuration blocks live on the MCP integration page - thirteen clients for the hosted server, seven for the local one - and are not repeated here; this article stays on the local server and the arm.

Step 1: install the CLI

One command per platform. The binary carries the robot driver inside it, which is the reason the CLI exists as a separate artifact from the desktop client: a terminal, a USB cable and an arm are enough.

# macOS and Linux
curl -fsSL https://www.ay-robots.com/install.sh | sh

# Windows
irm https://www.ay-robots.com/install.ps1 | iex

The installer reads the version and the SHA-256 from the release manifest at runtime, so a saved copy of the script stays valid after the next release. It verifies the checksum every time: if it finds none of sha256sum, shasum, openssl or busybox it aborts rather than installing an unverified 100-MB download, and getting past that requires setting AY_ROBOTS_SKIP_CHECKSUM=1 deliberately. On macOS it strips the quarantine attribute. It installs to /usr/local/bin when that is writable and to ~/.local/bin otherwise, tells you if the target is not on your PATH, and links three alternative names - ayrobots, ay-robot and ayrobot - so a forgotten hyphen does not send you nowhere.

The full command reference is on the CLI page. Two names are worth memorising before anything else moves: ay-robots stop stops a running policy, and ay-robots stop-backend stops the backend process. They are different commands and the similarity has cost people time.

Step 2: pair the machine, or decide not to

Pairing links this host to an ay-robots account so the platform can see the robot and the datasets it records. It is not required for driving the arm locally, and the choice has a security consequence that is easier to make deliberately than to discover later.

Open /dashboard/client/robots/new while signed in. The page generates a code as soon as it opens - eight characters from an alphabet that leaves out 0, O, I and 1 - and it stays redeemable for 15 minutes. If a backend is already running on the same machine, the page redeems the code itself; otherwise you redeem it on the robot host:

ay-robots pair ABCD2345

# undo it later with
ay-robots unpair

The dashboard shows the equivalent curl against the local backend for hosts where you would rather not use the CLI. Both do the same thing: POST /pair with the code and the platform URL. Once redeemed, the host stores the pairing and starts sending a heartbeat every 30 seconds.

Step 3: start the backend and confirm the arm is really there

The CLI holds no hardware logic. It starts the robotapp backend - the same FastAPI application the desktop client runs as a sidecar - and talks to it over HTTP, so there is exactly one implementation of the driver, the calibration and the recording format. You can start it explicitly, and any command that needs it will start it anyway.

ay-robots start      # background, idempotent, prints the log path
ay-robots devices    # SERIAL PORTS and CAMERAS, as detected
ay-robots status     # backend reachable, mode, paired, robot name
ay-robots logs -f    # what the backend is actually saying

The first start is slow and that is expected: unpacking the bundle and loading the camera libraries takes up to a minute, the CLI prints a keep-alive line after 20 seconds so it does not look wedged, and it waits up to 180 seconds before giving up. It watches the child process while it waits, so a backend that dies immediately reports that immediately rather than burning the full timeout.

What you are checking with ay-robots status is the mode field. It reads hardware when a real arm is attached, simulation when none is, and demo in demo mode. This matters more than it looks: in simulation mode every movement call still succeeds. The state updates, the response says success, and nothing physically moves. An agent that never checks will happily report a completed task to an arm that was never connected. The backend auto-connects at startup and a hotplug watcher keeps trying every three seconds, so plugging the arm in after the fact does not need a restart.

Step 4: hand the arm to the agent

One line. The double dash separates Claude Code's own options from the command it should run.

claude mcp add ay-robots-local -- ay-robots mcp

For Claude Desktop, Cursor, VS Code with Copilot, Windsurf, Zed and the Gemini CLI the equivalent stdio blocks are on the MCP integration page - all of them run the same ay-robots binary with the single argument mcp, and Cursor is the one that additionally insists on a type field. The server announces protocol version 2025-06-18 and implements exactly what a client needs: initialize, the initialized notification, ping, tools/list and tools/call. Anything else comes back as method not found.

On connect the server also hands the model a short instruction text, and it is worth knowing exactly what your agent has been told, because it is advice to the model rather than a rule the software enforces. The whole handshake is four fields:

{"jsonrpc":"2.0","id":1,"result":{
  "protocolVersion": "2025-06-18",
  "capabilities": {"tools": {"listChanged": false}},
  "serverInfo": {"name": "ay-robots-local", "version": "1.0.0"},
  "instructions": "Controls a physically connected SO-100 robot arm on this
     machine. Motion tools move real hardware: confirm with the user before
     calling them, and prefer small movements. Recording tools produce
     imitation learning data in the LeRobot format. If a tool reports the
     backend is unreachable, the arm is not connected or not powered."}}

Check the server from a terminal before an agent touches it

The server has no initialisation state machine. It reads one JSON-RPC line from standard input, dispatches on the method and writes one line back, so a single piped line is a complete conversation and you do not need a client to test it:

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | ay-robots mcp

Back come the nine tool definitions with their input schemas. This call touches no hardware and makes no HTTP request - the tool list is a constant in the server - so it answers with the arm unplugged and the backend stopped, and a failure here is a broken install rather than a robot problem. That is the whole point of running it first: it separates "the agent cannot start the server" from "the server cannot reach the arm", and those two have nothing to do with each other.

The nine tools, and where each one stops

The route column carries more than documentation. Four of the nine tools land on routes that are declared with a token dependency - /home, /motor, /recording/start and the stop, save and discard trio behind stop_recording. The other five do not: /health, /pair, /devices, /state, /recording/status and the read side of /training/datasets are open even on a paired host. So the reading half of the tool set works without authentication and the moving half does not, and the CLI supplies the token for both out of the local pairing file. The second use of that column is diagnostic: every tool is a thin wrapper over exactly one route, so any of them can be reproduced by hand with curl, and that is the fastest way to decide whether a problem sits in the agent or in the arm.

The three that only read

get_status is the one to call first, and the tool description says so. It combines the health and pairing endpoints and returns arm_connected as a boolean derived from the mode, which spares the agent from interpreting the difference between hardware, simulation and demo. list_devices is what you reach for when the answer is no - it shows the serial ports and cameras that were actually detected, which separates a missing arm from a missing driver. get_joint_positions returns the current pose. Nothing forces the agent to call it, but move_joint takes absolute angles, so a move planned without it is a move planned from a guessed starting point.

What get_status leaves out is worth knowing, because a summary loses things. /health returns a safety block - overheating, blocked, the highest measured servo temperature and the reading of every servo - plus a busy flag that is true while teleoperation or a policy is running, and the current joint angles rounded to one decimal. The tool forwards five fields and drops all of it. In practice that means an agent cannot ask whether the arm is cooling down after an overheat, or whether something else is already driving it; it finds out when a movement call fails. The one tool that does surface the block is get_recording_status, which reports safety_blocked among its fields - an odd place for it, and the only place there is.

The two that move

Both refuse without confirm set to true. The refusal is checked before any HTTP call is made, and it comes back as a tool result rather than a protocol error, so the agent reads a sentence telling it to ask the user and call again. This is a speed bump, not a lock: the agent can set the flag itself. It exists so that moving the arm is a deliberate step in the model's reasoning rather than a side effect, and so that a user reading the transcript can see where the decision was made.

The refusal reads the same every time, and it is worth recognising in a transcript: move_joint moves a physical robot arm. Ask the user to confirm, then call again with confirm: true. It arrives wrapped as an object with an error field reading confirmation required and the sentence above as the reason - and, notably, the result is not flagged as a tool error. The isError field stays false, because the check runs before the call is dispatched and never raises. A client that only highlights failed tool calls will show a refused movement as an ordinary, successful one.

move_joint takes a motor name, an absolute angle in degrees, and the flag. The names are shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll and gripper. The response carries applied and applied_position alongside the requested value, which is the field that matters: a command that was clamped, rate-limited or dropped is visible instead of silently having no effect. An agent that ignores applied_position will believe it moved 40 degrees when it moved 18.

applied comes back false whenever the backend produced no position for that command, and it does that for four reasons that can actually reach a tool call. The response does not distinguish them, so the table is the fastest way to narrow it down.

In all four cases the backend deliberately does not update its own idea of where the joint is. That matters more than it sounds: the step clamp measures from the arm's last known position, and a phantom update would drift that base away from the physical arm until the next accepted command made up the difference in one go. It also keeps rejected commands out of a running recording, where they would otherwise be written into the action column as motion that never happened. A thermal cool-down is the one refusal that does not show up here. The /motor route checks the safety block before anything else and answers 423, so the CLI aborts and the agent reads the generic failure described further down instead of applied: false. One limit belongs in the same paragraph: move_joint sends only a motor name and a position. The route accepts a robot_id as well, but the tool never fills it in, so on a host with two arms these tools always address the primary one. The second arm shows up in get_joint_positions under arms and cannot be commanded from here.

The three that record, and the dataset that has to exist first

start_recording opens the buffer; it does not drive the arm. The backend samples at 30 Hz by default and writes LeRobot-format data. The tool defaults dataset_id to local, and this is where a first attempt usually fails: recording only works into a dataset that already exists, and a dataset literally named local is not created for you. A missing one comes back as a 409 reading that the dataset does not exist and has to be created first. Neither the CLI nor the nine tools can create one - POST /training/datasets does, and that is a call you make yourself or through the platform.

Creating one is a single call, and it is the same call the local web interface makes when you add a dataset there. It takes a name and an optional description, replaces everything outside letters, digits, underscore and hyphen in the name, and answers with the dataset's id:

# on the machine the arm is plugged into
curl -X POST http://127.0.0.1:8000/training/datasets \
  -H 'Content-Type: application/json' \
  -d '{"name": "local", "description": "agent runs"}'

# on a paired host the same route wants the token:
#   -H "Authorization: Bearer <the pairing code>"

A second call with the same name comes back 400 Dataset already exists. The id in the answer is a stable UUID rather than the folder name, and start_recording accepts either - the backend resolves a name or a UUID to the same folder - which is the reason to call list_local_datasets once instead of guessing what the dataset is called. That tool also reports the episode count, the camera keys and the frame rate that the dataset already carries, which is what tells you whether a new episode will match the ones already in it.

Two other things refuse a start, and both are 409s that read like full sentences. The first is disk space: below a free-space floor of 3 GB the backend will not open a recording at all. It first tries to clear known junk - quarantined torsos and salvage leftovers - measures again, and only then refuses, naming how much is free and where the line sits. The second is cameras: if the host's devices.json lists required cameras, every one of them has to appear in the scan, otherwise the start is refused with the missing names spelled out. The HTTP route takes allow_missing_cameras to record deliberately without one; the MCP tool has no such argument, so that override is not available to an agent. With no camera list given at all, the backend records every camera it finds. And one refusal is not a failure at all. Starting while a recording is already running returns HTTP 200 with success set to false and the message Recording already in progress. The tool call succeeds, so the agent has to read the field rather than the status. Everything else on this route is a real HTTP error.

stop_recording is two operations behind one tool: it stops, and then either saves or discards based on the save flag. Discarding failed attempts matters more than it sounds, because a saved failure teaches a policy to fail.

get_recording_status is the cheapest way to catch a camera that stopped delivering halfway through a take, and it is considerably richer than its one-line description. Beside active, frame_count, duration and the per-camera frame counts it carries three counters that each mean something different: dropped_camera_frames counts frames written without images because the capture raised, stale_camera_frames counts captures that returned fewer images than there are open cameras, and blocked_frames counts frames recorded while the safety block was on. It also reports loop_errors, spool_drops, a stop_reason if a watchdog ended the take, save_running and save_error for the background save, and safety_blocked. A camera that quietly died mid-take shows as its own count standing still while frame_count keeps climbing - that is the check worth making before the episode is saved rather than after.

One thing the tool descriptions do not spell out: while a recording is active, the commands that move_joint sends are written into the episode's action column. An agent-driven recording is therefore possible, but it will not look like a human demonstration - it is rate-limited to one command per joint every 12 milliseconds and clamped per step, so the trajectory is a staircase rather than a motion. For real demonstration data the recording path expects a human on the controls; that route is covered in recording your first dataset, and the full path from a first episode to a dataset other people can download in from first recording to a listed LeRobot dataset.

What is deliberately absent is as informative as what is there. There is no tool that overwrites calibration, sets motor IDs or removes the pairing. The stated reason in the code is that an agent should be able to use the arm, not reconfigure it. Calibration stays a human step at ay-robots calibrate - see calibration for what the zero point actually is.

A first task from start to finish

The smallest task worth doing is also the best first one: read the arm, move one joint by a known amount, verify it arrived, put it back. It exercises every part of the chain and breaks nothing if the chain is wrong.

1. 1
   Confirm the arm before anything else

   Ask the agent for the status first. get_status returns exactly five fields, and arm_connected is derived from mode - it is true only for hardware. If it comes back false, stop here and fix that; everything downstream will otherwise succeed silently against a simulated arm.

   json

   get_status
   {
     "mode": "hardware",
     "operating_mode": "idle",
     "paired": true,
     "robot": "so-100-01",
     "arm_connected": true
   }

2. 2
   Read the pose you are starting from

   move_joint is absolute, so the current angle is not context, it is the input to the arithmetic. The tool hands back the whole robot state, not a flat list of angles: each motor is an object, and on a multi-arm host the calibrated degrees per arm sit next to it under arms. Trimmed to the parts that matter:

   json

   get_joint_positions
   {
     "connected": true,
     "mode": "idle",
     "motors": {
       "shoulder_pan":  {"position": 12.4, "velocity": 0.0,
                         "torque": 0.0, "target_position": 12.4},
       "shoulder_lift": {"position": -88.9, ...},
       "elbow_flex":    {"position": 71.2, ...}
     },
     "arms": {"robot_1": {"connected": true, "positions": {...}}}
   }

3. 3
   Move one joint, small, with the flag

   Ten degrees on shoulder_pan from 12.4 means an absolute target of 22.4. Ask the human first; that is what the flag is there to make visible. robot_id comes back null because the tool never sends one - on this host that means the primary arm.

   json

   move_joint  motor=shoulder_pan  position=22.4  confirm=true
   {
     "success": true,
     "motor": "shoulder_pan",
     "position": 22.4,
     "robot_id": null,
     "applied": true,
     "applied_position": 22.4
   }

4. 4
   Verify against applied_position, not against the request

   position echoes what was asked; applied_position is what reached the servo. If they differ, the command hit the step clamp or a joint limit. If applied is false, nothing reached the servo at all - the four reasons are in the table above, and the log names which one it was.

   text

   get_joint_positions
     -> shoulder_pan: 22.3     # servos land close, not exact

5. 5
   Put it back, and check that it got there

   Home is the natural way back, and the natural place to see the clamp behaviour: call it, read the positions, call it again if they are still moving toward the pose. Do not trust its success field - it reports one whether or not anything moved.

   text

   move_home  confirm=true
   get_joint_positions        # still 18 degrees out? call move_home again

A useful habit when writing the prompt: tell the agent the working envelope in joint angles before you tell it the task. The backend enforces the mechanical limits, but it knows nothing about the mug standing next to the arm. The limits below are the outer bounds, not a description of your desk.

When it does not work

The messages below are the ones the code actually emits, which makes them worth searching for verbatim. Where they appear is the part worth knowing in advance. Inside a tool call the CLI hits the route through its api() helper, and that helper aborts on any non-2xx answer: it prints the status code and the backend's detail to standard error and exits with code 1. The MCP server catches that exit - an exiting process would otherwise take the whole server down mid-session - and turns it into a tool result reading backend call failed (exit 1). Standard output carries the protocol; standard error does not. The 409, the 423, the 400 with the motor name in it: none of them reach the agent.

That is the single most useful thing to know when this path misbehaves. The agent's report is always the same sentence, and the real reason is one terminal command away. Reproduce the call yourself - ay-robots status and ay-robots devices cover the reading side, curl against the route covers the rest - and the message appears in full. Tool errors that are raised inside the MCP server rather than by the CLI do carry their text, and those arrive flagged as errors, so the agent can read and react to them without losing the connection.

For the hardware faults underneath these messages - an arm that enumerates but does not answer, a servo that holds but will not move - the symptom-first pages are arm not detected, servo not responding and camera not detected.

What an agent can do to a real arm

The honest version, because the reassuring version gets someone's arm bent. The confirm flag is a prompt-level speed bump, not an interlock - the model sets it itself once it has decided to move. What actually constrains the hardware is the backend, and it is worth knowing exactly where that protection ends.

What the backend enforces, whatever the agent asks for

• Joint limits. Every target is clamped into the range for that joint before anything is sent, and the clamped value comes back as applied_position.
• A per-command step clamp of 18 degrees, 12 for the gripper. A single command cannot produce a large swing, whatever number was in it.
• A per-joint rate limit of one command every 12 milliseconds. Faster commands to the same joint are dropped, not queued.
• Servo speed and acceleration are capped in software rather than left at the maximum the servos accept.
• A thermal watchdog polling every two seconds. At 65 C it parks the arm in a rest pose and blocks commands; control returns below 55 C, and blocked commands are answered with 423 rather than silently ignored.
• A saturation guard that discards a target which would land far outside the raw servo scale, instead of slamming into the end stop.

What nothing enforces

• Self-collision. The collision check exists as a function and is switched off - it returns safe for every pose. Nothing stops the arm folding into itself or into the table.
• Anything about the world. There is no model of your desk, the mug, the cable or your hand. Joint limits are mechanical bounds, not a workspace.
• Two command sources at once. The health endpoint reports busy while teleoperation or a policy is running, and the movement route does not consult it. An agent can command joints into a running policy, and both will keep writing goals to the same servos.
• A safe home. If no home pose was saved, move_home drives every joint to zero degrees, which is a pose, not necessarily a safe one for your setup.
• Local access, on an unpaired host. The control routes require no token until the machine is paired.
• Anything at all once torque is off. Torque can be disabled through the backend, though not through any of the nine tools - the arm falls under its own weight when it is.

What this path deliberately does not do

The nine tools are a way to operate an arm, not a way to teach one. There is no training here and no CLI command for it either: training runs on a rented cloud GPU and is started from the web interface or, if you want an agent to do it, from the hosted server's start_training tool - which refuses to spend money without a separate confirmation flag of its own. The imitation learning route, from demonstrations through a fine-tune to a running policy, is the learn track, with the model-by-model detail under policies and the per-arm recipes under training guides. Correcting a policy that already runs but drifts is a different loop again, covered under DAgger.

Nor is this teleoperation. The recording tools open and close a buffer; they do not carry a control loop, and the CLI has no teleoperation loop either - deliberately, so that a second, divergent recording path cannot exist. Driving the arm by hand happens in the desktop client or the web interface.

What is left is worth being precise about, because it is genuinely new: an agent that has never seen a demonstration can read the arm's state, reason about it in the same context where it is reading your code, and command joints one at a time within bounds you set. For positioning tasks, fixture setup, repeatable test motions and anything you would otherwise script by hand, that is enough. For anything requiring contact, timing or recovery from being slightly wrong, it is not - that is what a trained policy is for. Which of the two a given task actually needs, and what each one costs, is worked through in scripted motion against a trained policy.

Next step, if the arm is already on the desk: install the CLI, run ay-robots devices, and only wire the local MCP server in once that command lists your arm. Debugging the agent path and the hardware path at the same time is the slowest way through.