Hand-Eye Calibration Without a Checkerboard on an SO-100 or SO-101

You point a camera at your arm's workspace, put a red block on the table, and ask the software to pick it up. The vision side does its job: the block is at pixel (612, 344). The arm does its job too: it can drive every joint to any angle you name. And then nothing works, because no part of that system knows what joint angles put the gripper at pixel (612, 344). That missing piece has a name — hand-eye calibration — and on a SO-100 or SO-101 neither LeRobot nor this platform ships it. This article explains what the missing piece is, why the textbook way of getting it fits a factory cell better than a printed arm on your desk, and what a checkerboard-free alternative looked like when ay-robots ran one.

Your arm knows angles, your camera knows pixels

A robot arm is a chain of known lengths with measured angles at every joint. Feed the six servo readings of an SO-100 through the geometry of the links and you get the position and orientation of the end effector in a coordinate frame anchored at the base of the arm. That is forward kinematics, it is exact up to how well the arm matches its own drawing, and it needs no camera at all.

That phrase, the geometry of the links, sounds like a research problem and is in fact a short list of numbers — and this platform already carries them. The browser-side solver behind the 3D teleoperation view has the SO-100 chain written into it as approximate bone lengths taken from the arm's URDF: a shoulder pivot 63.3 mm above the base, a 102.5 mm upper arm, a 113.1 mm forearm, a 59.6 mm wrist and a 134.9 mm offset out to the end effector. Six joint angles go in, the position of every link comes out. The reverse direction sits in the same file — cyclic coordinate descent to a 3 mm position tolerance, an early rejection for anything outside a 0.45 m ball around the shoulder, and up to five random restarts when it fails to converge.

Not one line of that touches a camera, and that is exactly the point. The whole chain from joint angles to a metric position in the base frame is already built, already running in a browser, and completely blind. Hand-eye calibration is the one link that would let something the camera saw enter that chain.

A camera produces a grid of brightness values. Detect the block in it and you get an image position — a column and a row — plus, if the detector is good, a size in pixels. What you do not get is a position in the arm's frame. A pixel is a direction, not a place: every point along one line of sight lands on the same pixel, and the camera has no way to tell you which of them your block is at.

Hand-eye calibration is the measurement that ties those two frames together. Once you have it, an image position becomes something the arm can act on. Without it, you have two systems that each work perfectly and cannot cooperate. Four frames are involved and it is worth naming them before anything else, because most confusion in this topic is two people using the same word for different frames:

• Base frame — fixed to the bolted-down base of the arm. Everything the arm computes ends up here.
• End-effector frame (also tool frame, gripper frame) — rides at the tip of the arm and moves with it. Forward kinematics gives you this frame relative to the base.
• Camera frame — anchored to the camera's optical centre. Anything the camera detects is expressed here, up to the unknown distance along the line of sight.
• Target frame — attached to whatever you are calibrating against: a printed board, a marker, or in the checkerboard-free case simply a point on the table.

Eye-in-hand and eye-to-hand are two different unknowns

If the camera is bolted to the wrist — the usual wrist camera in a LeRobot recording setup — it moves with the arm, and the unknown you are solving for is the fixed transform between the end-effector frame and the camera frame. This is the eye-in-hand case. If the camera sits on a tripod looking at the table, it does not move with the arm, and the unknown is the fixed transform between the base frame and the camera frame. This is the eye-to-hand case.

The mathematics is nearly the same and the practical consequences are not. An eye-in-hand calibration survives you picking the whole robot up and putting it down somewhere else, because both frames moved together. An eye-to-hand calibration does not survive anyone nudging the tripod. Most desk setups on an SO-100 have both cameras at once — one at the wrist, one overhead — which means two separate calibrations, not one. Recording setups on this platform routinely run several cameras; see data collection on the SO-100 for how the camera set is normally arranged.

Three different things are all called calibration

Before going further it is worth separating three procedures that share a name and solve unrelated problems. People arrive at hand-eye calibration having already done one of the other two and reasonably assume they are finished.

The joint one is the one everybody meets first. In LeRobot it is a command that asks you to centre every joint and then sweep each through its full range of motion; the platform's own version is described step by step in getting started with the SO-100, and the term itself is defined in the glossary. Both of them teach the software about the arm. Neither of them has ever seen a camera.

The classic route: a printed board, intrinsics, and AX = XB

The textbook procedure has two stages, and the first one is not hand-eye calibration at all. You first calibrate the camera itself: show a planar pattern of known geometry — the black-and-white chessboard everyone recognises — at several different orientations, and solve for the camera's internal parameters. That is Zhang's method, and it became the standard because it needs only a printed plane rather than a machined 3D calibration rig.

With intrinsics in hand you can work out where the board sits relative to the camera in each shot. Now the second stage. Move the arm to a set of poses. At every pose you record two things: where the board is relative to the camera, which vision gives you, and where the end effector is relative to the base, which forward kinematics gives you. Take the relative motion between any two poses and the unknown transform appears in the classic form AX = XB, where A is the motion the arm made, B is the motion the camera observed, and X is the fixed camera-to-arm transform you want. Horaud and Dornaika's treatment of the problem is a readable statement of that formulation and of what it costs.

You do not have to implement the solver. OpenCV ships calibrateHandEye with five classical methods behind it — Tsai and Lenz, Park and Martin, Horaud and Dornaika, Andreff, and Daniilidis's dual-quaternion formulation. You hand it the arm poses and the observed target poses and it returns the transform. The library is not the hard part.

The hard part is everything you must supply before calling it:

1. A rigid, accurately printed target. A board that came out of a home printer at 98 percent scale introduces a scale error into everything downstream, and a board taped to a warped surface is not planar.
2. Camera intrinsics good enough that the board pose it produces is trustworthy — which means a second calibration session before the first one starts.
3. A set of arm poses chosen so the rotations are genuinely varied. Poses that differ mainly by translation leave the rotation part of the problem badly conditioned, and the solver converges on something confidently wrong.
4. Arm poses that are actually where the arm says they are. Every one of these solvers treats forward kinematics as ground truth.
5. A detection of the board that is reliable across all of those poses, including the oblique ones where the squares are heavily foreshortened.

Why that fits an industrial cell better than a printed arm

Look at the fourth item again, because it is the one that decides whether this route is sensible on your desk. AX = XB assumes the arm's reported pose is the arm's true pose. On an industrial manipulator with harmonic drives and encoders on the output side, that is a reasonable assumption and the residual error of the calibration really is dominated by the camera. An SO-100 is five joints plus a gripper built from Feetech STS3215 bus servos at 7.4 V in a 3D-printed structure, for roughly 110 to 150 euro in parts. The servos measure the motor side of a gear train, not the output side. Whatever backlash and flex sit between them goes straight into your calibration as if it were camera error, and the solver has no way to tell the difference.

The research community has been chipping away at the other items. EasyHeC, for example, is explicitly motivated by how much careful design of joint poses and how much specialised marker hardware the traditional approaches demand, and replaces both with differentiable rendering and space exploration. A separate line of work attacks the correspondence requirement rather than the marker, treating calibration as a distribution-matching problem solved with a GAN when data correspondence is absent or only loosely established. Both are research code aimed at people who already have a working vision stack, not a fifteen-minute procedure for someone whose arm arrived last week.

The route from the MHS run: the arm is its own ruler

On 27 August 2026 Anthropic previewed the Model Hardware Standard, a shared specification for AI agents to operate physical devices, currently in a research preview that organisations apply to join. The announcement names early users in drug discovery and imaging — Genentech, HHMI Janelia, and others — and lists Hugging Face among the early adopters, "adding MHS support in LeRobot, their robotics library", giving no date and no arm model for it. ay-robots took part in the research preview with its own run. That run and Hugging Face's LeRobot support are two separate things, and ay-robots is a participant in the preview rather than a partner of Anthropic. If you want the standard itself explained, that is the subject of a separate article on what MHS changes for your robot arm, and the numbers that run reported — together with what they do and do not establish — are written up in our report on that calibration run.

What matters here is the calibration method that run used, because it does not look like the textbook procedure at all. Nothing was trained for it: no policy, no teleoperation, no demonstrations. The agent measured the workspace itself and wrote the motion. And the way it established the camera-to-arm relationship used neither a chessboard nor camera intrinsics.

1. 1
   Torque falls off

   The servos are released so the arm can be moved by hand without fighting back. This is the same physical state the joint calibration wizard puts the arm in, and it has the same consequence: an arm that was holding itself up will drop when you release it, so take its weight before you start.

2. 2
   The software draws points in the camera view

   Sixteen points, rendered into the camera's own perspective. This is the step that removes the printed target from the procedure: the reference pattern is not an object in the world that the camera has to find, it is a set of positions in the image that a human is asked to go to.

3. 3
   A human places the closed gripper on each point

   The gripper is closed so that it is a single well-defined physical point rather than two jaws with a gap between them. The person moves the limp arm until the gripper sits at the place that appears at the drawn point, and confirms.

4. 4
   The robot reads back where it is

   At each of the sixteen confirmations the arm reports its joint positions. That yields sixteen pairs: an image position, and an arm configuration known to put the gripper there. Those pairs are the calibration data, and they were produced without measuring anything with a ruler.

The phrase worth keeping is that the arm is its own ruler. In the classic route the printed board carries the known geometry and the arm is a device for moving the camera around it. Here there is no known geometry anywhere: the arm's own joint readings supply the scale, and the human eye supplies the correspondence between an image position and a physical one. Camera intrinsics never enter, because at no point does anything have to convert a pixel into a metric direction — the mapping is fitted directly from image positions to arm configurations.

How finely the arm can report a position at all

If the arm is the ruler, the ruler's smallest division is worth knowing before you quote anything in millimetres. On this platform a joint angle is not what the servo stores. The servo stores a raw tick, and the backend converts it: take the difference from the calibrated home the short way round the circle, scale it, round it to one decimal place.

# raw servo tick -> joint degrees, calibrated home = 0 deg
home  = cal_home[motor]
delta = ((raw - home + 2048) % 4096) - 2048
deg   = round(delta / 2047.0 * 150.0, 1)

2047 ticks are 150 degrees, so one tick is about 0.073 degrees — and the value the API hands you is rounded to 0.1 degrees regardless, which is the coarser of the two. What that is worth in millimetres depends on the lever. At 0.45 m, the outer workspace bound the browser solver uses for the SO-100, 0.1 degrees at the base subtends about 0.8 mm. That is a floor under any arm-as-ruler figure on this hardware. It sits well below the 4.1 mm the MHS run reported, which is the useful conclusion here: the resolution of the readout is not what limits that number. What does limit it is not established anywhere in this article.

The same conversion has one quirk that matters if you store the gripper value alongside the arm pose. The circular arithmetic folds anything above +100 degrees back by 300, because the physical gripper travel crosses the boundary of that arithmetic and a firmly closed gripper would otherwise read about +149 instead of -151. The number that comes out is the right one; the scale simply is not continuous across that boundary. If your fit treats joint values as plain real numbers, keep the gripper out of it.

Resolution is one half of what a reading is worth; freshness is the other, and it is the half that quietly ruins collected data. The state endpoint does not talk to the servos when you call it. A background poller reads the bus at 20 Hz and the endpoint hands back whatever that poller last accepted, and accepted is doing real work in that sentence. A joint is written back only once it has moved more than a tenth of a degree. A reading that lands more than 15 degrees outside that joint's limit table is discarded on the reasoning that the arm cannot physically be there, and the previous value is kept. And a jump faster than 500 degrees per second is held back until the next read confirms it, capped so that one long gap between polls cannot open that gate by more than 75 degrees at once.

None of those three trips while a human hand guides an arm — nobody moves a joint at 500 degrees per second — but all three fail in the same direction. They hand you a stale value rather than an obviously wrong one. In a collection loop that is the worst available failure: the pair you store looks clean and carries the previous point's angles. Let the arm settle, read twice a fraction of a second apart, and keep the point only when the two reads agree.

poller      20 Hz over the serial bus; the state endpoint serves its last accepted value
written     only once a joint has moved more than 0.1 deg
discarded   readings more than 15 deg outside that joint's limit table
deferred    jumps faster than 500 deg/s, until the next read confirms them
            (capped at 75 deg, so a long poll gap cannot open the gate wide)

What the closed gripper buys you

The probe in this procedure is the gripper itself, closed. That choice does more work than it looks like. A closed gripper is rigid, it is the part of the arm whose position you actually care about, and it is visible to the person doing the placement, so the human can judge contact directly rather than through the camera. It also means the calibration is expressed in terms of the thing that will later do the grasping, with no separate tool-offset measurement in between.

There is a limit built into the same choice, and it should be stated plainly rather than discovered later. Placing a gripper on points means placing it on a surface — the table. Sixteen points reached on one plane constrain the mapping on that plane well and say considerably less about positions well above it. The run as described does not establish how the mapping behaves off that surface, and nothing in it should be read as a full three-dimensional camera calibration.

What the platform ships today, and what it does not

This is the section where a guide article usually tells you which button to press. Here it cannot, and saying so is more useful than inventing a menu path. Of the three calibrations in the table above, the platform ships one properly, one halfway, and one not at all.

Joint calibration: shipped, and it lives in the web cockpit

The desktop client has no calibration screen at all. Its navigation is Dashboard, Record, Inference, Replay, Settings and Logs, plus a setup page for the first run, and the word calibration does not occur anywhere in it. The wizard lives in the web cockpit instead, behind a button labelled Calibrate Arm that sits next to Configure Motors and stays disabled until the motor ids have been set. Opening it releases torque on all six servos immediately, before you have agreed to anything, which is why its first screen is a safety notice rather than a start button.

The wizard runs four steps — Safety, Position 1, Position 2, Done. Position 1 asks for the arm extended straight forward with the gripper fully closed. Position 2 asks for the base rotated left. Each capture is a single read of the six servo positions; nothing moves during it, and what comes back is raw ticks rather than degrees, together with each joint's offset from the electrical centre at tick 2048. One thing the wizard does not do is turn the motors back on. Neither the cockpit wizard nor the vendored copy of it calls the torque-enable route when it closes, so an arm that has just been through joint calibration is still limp — convenient if the next thing you do is place a gripper on sixteen points, and worth knowing if it is not.

POST /motors/calibration/capture
{
  "success": true,
  "positions": {
    "shoulder_pan": { "id": 1, "position": <raw tick 0-4095>, "offset": <position - 2048> },
    ...
  },
  "timestamp": "..."
}

What happens to those two poses afterwards is worth knowing before you build anything on top of them, because they are not treated equally. The wizard sends both, the second one under its own key. The backend never reads that key: the save route takes the first pose and writes a record of a timestamp plus those motor positions, and the second pose is discarded on arrival. The range limits that Position 2 appears to establish are in fact a fixed table compiled into the backend — the same numbers on every arm, not measured on yours. The command-line path draws the same conclusion and does not ask for a second pose at all.

So the stored joint calibration is the centre pose, and only the centre pose. For a hand-eye mapping that is the thing to write down: every joint value in your sixteen pairs is expressed relative to that one captured zero, and nothing else about the wizard survives into the file.

Two responses from the save route are worth recognising when they appear. An empty table is refused rather than written, because writing it would discard the working home offsets and leave a commanded zero degrees pointing somewhere else entirely. And a captured zero that sits more than 1200 ticks from the electrical centre still saves, but comes back with a warning and the offending joints listed — that is the endstop case, and the fix is to recapture from a genuinely neutral pose rather than to ignore it.

# refused: an empty table would wipe the stored home offsets
POST /motors/calibration/save  ->  400
"calibration save needs a non-empty 'positions' map - refusing to
 overwrite the stored home offsets with an empty table"

# saved, but flagged: a zero far from tick 2048 saturates the joint against an endstop
POST /motors/calibration/save  ->  200
"Calibration saved + reloaded - WARNING: homes far from centre {...}
 (recalibrate from a neutral pose to avoid endstop saturation)"

The session reports its progress in sentences rather than numbers, and they are the most useful part of it if you are trying to decide whether the lens is your problem. Three endpoints carry the whole feature. Nothing in the cockpit or the desktop client calls the first one. The correction itself is still reachable even though its button is not: the camera stream takes a defish parameter, and asking for defish=-1 applies that camera's stored chessboard profile to the feed. The cockpit's own camera component hard-codes it to 0.

POST   /camera/{device_id}/calibration/start     # 409 if a session is already running
GET    /camera/{device_id}/calibration/status    # {"session": {...}, "calibrated": bool, "rms": float}
DELETE /camera/{device_id}/calibration           # drop the stored profile for this camera

# what a running session says back, verbatim (German, as shipped):
"Kein Muster erkannt - Schachbrett voll ins Bild halten"
"Muster erkannt - jetzt bewegen/kippen für die nächste Ansicht"
"Ansicht gespeichert - weiter bewegen/kippen"

# the two ways it gives up:
"Zu wenige Ansichten (N) - Muster näher/ruhiger zeigen und neu starten"
"Ansichten zu einseitig (RMS X.Xpx) - Muster beim nächsten Mal stärker KIPPEN, nicht nur verschieben"

Hand-eye calibration: not shipped, and not hiding somewhere

There is no camera-to-arm registration anywhere in the platform. No endpoint stores such a transform, no screen collects one, and no part of the recording or inference path consumes one. The only inverse-kinematics code in the website is the browser solver behind the 3D teleoperation view, which maps a dragged handle to joint angles and has nothing to do with cameras. Nor does the standard LeRobot workflow for the SO-101 fill the gap: its calibration step aligns leader and follower joint values and stops there.

This is worth being blunt about because the platform is otherwise camera-heavy. Recording handles several cameras including a wrist camera, the device scan reports each camera's stable hardware id and whether a lens calibration exists for it, and policies are trained on those images. All of that works without any camera-to-arm transform, because an imitation policy learns the pixels-to-joints relationship implicitly from demonstrations instead of being told it. Hand-eye calibration is what you need when you want to skip the demonstrations — which is exactly the situation an agent driving the arm directly is in.

Building the sixteen-point routine on your own arm

What follows is a recipe, not a feature. Every endpoint named below exists in the robotapp backend and does what is described; the sequence that strings them together is yours to write. Treat it as the shortest honest path from what is shipped to what the MHS run did.

1. 1
   Calibrate the joints first, and do not touch them again

   Everything downstream is expressed relative to the joint zero, and the platform already demonstrates what that costs when it changes. A saved home pose records which calibration it was captured against, and the endpoint that reads it back compares that record against the current one and returns calibration_stale as true when they differ — the degrees in the stored pose then point at a different physical position. Your hand-eye mapping has exactly the same dependency and no such guard, so record which joint calibration it was fitted against and treat a recalibration as invalidating it.

2. 2
   Decide the camera, and fix it

   Pick one camera and settle its position before you collect a single point. Note its stable hardware id from the device listing rather than its device index: the index moves between reboots and re-plugs, the hardware id does not, and it is the key everything camera-related is stored under. The same listing tells you whether a lens calibration already exists for that camera. For an eye-to-hand setup, clamp the camera to something that will not be nudged. One caveat before you clamp anything down: on Linux that id is assembled from the USB vendor and product ids, the serial number when the device has one, and the USB port path. A camera without a serial number that moves to a different port therefore arrives as a new hardware id, is handed a new camera_N name, and no longer matches the lens calibration filed under the old one. Leave the cable in the port it is in.

   bash

   # human-readable: SERIAL PORTS and CAMERAS, ids and names only
   ay-robots devices
   
   # machine-readable: this is where hardware_id and calibrated actually appear
   ay-robots --json devices
   
   # or straight from the backend, no token needed for this one
   GET /devices
   #   cameras: id, name, hardware_id, isGripper, calibrated
   #   robots:  id, port, device_path, isLeader, temps, maxTemp
   # the result is cached for 2.5 s (5 s while a recording runs), so a camera you
   # just moved or re-plugged can take one more call to show up correctly

3. 3
   Release torque, then guide the arm by hand

   Two endpoints do this. One releases every arm on the host, which is what the calibration wizard uses the moment it opens. The other releases only the arm you name, which matters on a two-arm bench where you do not want the second arm to drop. The per-arm one answers with the timeout it just armed — 180 seconds by default — and pulls the motors back in when it expires, rather than leaving the arm limp until it falls over. That re-engagement is done position-synced: each servo's current position is read first and written back as its goal, so the arm freezes where your hand left it instead of snapping to whatever goal it held before. Copy that if you re-enable torque yourself between points; a naive torque-on moves the arm and destroys the measurement you just took. That timeout is also the constraint that shapes the session, because sixteen hand placements do not fit into three minutes. Three ways out, in order of how much you have to accept: re-issue the release between points, raise ROBOTAPP_HOMEPOSE_RELEASE_TIMEOUT_S before the backend starts, or fall back on the all-arms release, which arms no watchdog at all — that is not a free option but a different risk, since nothing will then catch the arm for you.

   bash

   POST /motors/torque/disable     # every arm on this host goes limp, no watchdog
   POST /motors/homepose/release   # only the named arm -> {"timeout_s": 180.0, ...}
   
   # refusals you will meet, verbatim (the backend answers in German):
   409  "Teleoperation läuft - Arm nicht kraftfrei geschaltet"
   409  "Inferenz läuft - Arm nicht kraftfrei geschaltet
         (die Policy würde gegen kraftfreie Motoren kämpfen)"
   404  "Robot <id> not connected"      # named an arm that is not attached
   503  "Kein Arm verbunden"            # no arm at all

4. 4
   Read the joint positions back at each point

   This is the whole measurement. With the gripper resting on the drawn point, read the arm state and store it alongside the image coordinates of that point. Sixteen of these pairs is what the run used. Read the camera health out of the same response while you are there: it reports per camera whether the capture is open, how old the newest frame is, and how often the camera has had to be reopened or USB-reset. A camera that has stalled keeps showing its last frame, and a point placed against a frozen image records perfectly cleanly as a pair that is simply wrong. An agent can do the reading through the local MCP server, which exposes the same state as a tool.

   bash

   GET /state
   #   motors.<joint>              -> {position, velocity, torque, target_position}, calibrated degrees
   #   arms.<robot_id>.positions   -> the same per arm, on multi-arm hosts
   #   camera_health.<camera_id>   -> {open, state, last_frame_age_s, reopen_count, usb_resets}
   #     state: idle | no_frames | ok | stalled   ("stalled" = no fresh frame for 10 s)
   
   # the same read, exposed to an agent as an MCP tool:
   get_joint_positions

5. 5
   Fit the mapping, and keep points back

   Fit whatever your setup justifies — for a fixed camera looking at a flat table, a planar homography between image coordinates and the arm's own coordinates is usually enough, and it is far easier to sanity-check than a full pose. Whatever you fit, do not fit it on all sixteen points. Hold several back, and keep the raw ticks as well as the degrees: degrees depend on the calibrated zero, ticks do not, so a pair recorded in ticks survives a recalibration that a pair recorded in degrees does not.

If the arm is being driven by an agent rather than by your own script, the movement side of this is already exposed: the local MCP server carries tools for reading state and for moving a single joint or homing the arm, and it refuses every movement tool that arrives without an explicit confirmation flag, before any request reaches the arm. The setup for that is documented on the MCP page, and the command-line side on the CLI reference. Neither of them will fit a hand-eye mapping for you.

# the movement tools refuse until an explicit flag arrives, before the request reaches the arm:
{
  "error": "confirmation required",
  "reason": "move_joint moves a physical robot arm. Ask the user to confirm,
              then call again with confirm: true."
}

The tool set is deliberately narrow in a way that matters here. get_joint_positions and list_devices are the two reads this procedure needs; move_joint and move_home are the two writes, and both are gated on the confirmation flag. There is no tool that overwrites a calibration, assigns motor IDs or breaks the pairing — an agent is meant to use the arm, not reconfigure it. That is also why no agent can produce a hand-eye mapping for you as a side effect of anything: the tool that would store one does not exist. For this procedure the decisive absence is torque. No tool releases or re-engages the servos, so an agent cannot put the arm into the limp state the sixteen points are collected in. The collection half is a human act by construction: a person releases the torque, and a person guides the gripper onto each drawn point. What an agent can contribute is the reading, and afterwards the driving to your held-back points.

One thing will stop your very first write request if you do not know about it. The control routes are token-gated, but conditionally: while the machine is unpaired the local backend lets everything through, which is why the cockpit works out of the box. As soon as it is paired with an account, every control route wants an Authorization: Bearer header matching the stored pairing token, and answers 401 for a missing header or 403 for a wrong one. The two reads this recipe leans on — the state and the device listing — are not gated at all, so a read-only probe works before the question ever comes up.

One safety net you get without asking for it. The backend runs a flight recorder at one hertz that appends the joint angles of every connected arm, the recording and inference status and the age of each camera frame to a JSON-lines file named for the day. Files older than seven days are removed, and the writer pauses once a day's file passes 80 MB. If your collection script dies at point eleven, the angles for the first ten are still on disk and can be matched to your image timestamps by the second.

blackbox/bb-20260829.jsonl          one JSON object per second
{"ts": ..., "arms": {"robot_1": {"shoulder_pan": 12.4, ...}}, "cams": {...}}

ROBOTAPP_BLACKBOX=0                 turns it off

Driving to a fitted point is not a single command

A mapping is only worth having if you can send the arm to what it predicts, and the control route is stricter than it looks. A motor command — one POST to /motor — takes one joint at a time, in absolute degrees, and three filters sit between it and the servo. The target is clamped to that joint's limit table. It is then clamped again to at most 18 degrees away from where the joint currently is, 12 for the gripper. And a second command for the same joint within 12 milliseconds is dropped outright rather than queued.

None of that is hidden. The response carries both what you asked for and what actually reached the servo, so a clamped or dropped command is visible instead of quietly doing nothing. But it does mean a single command cannot reach a point more than 18 degrees away, which is the shape most first attempts at this take.

The backend's own home-pose drive shows the pattern to copy. It re-issues the same absolute target for every joint in rounds of 0.12 seconds, up to eighty rounds, and drops a joint from the working set once the applied value is within half a degree of the target. Two details in it are worth copying as well: it interpolates all joints in parallel rather than one after another, so the total time is the longest path instead of the sum of all of them, and it reports per joint whether the target was met, so a partial arrival is legible rather than silent. That last field needs reading carefully before you build anything on it.

Where it goes wrong

Most of the failure modes below are mechanical rather than mathematical, which is the opposite of where people look first. The table separates what you would notice from what is usually behind it.

Two of those deserve a caveat about how firmly they are established. Backlash and sag are the standard explanations for direction-dependent and extension-dependent error in a geared arm, and the SO-100's construction — motor-side measurement through a gear train, printed links — makes both mechanisms available. What is not established anywhere in this article's evidence is how large either effect is on a particular SO-100, or how much of the 4.1 mm figure from the MHS run either accounts for. Measuring that is a worthwhile experiment and an open question, not something to assert. If a camera has dropped off the listing entirely, start at camera not detected instead, because none of this applies until the device is visible.

There is one constraint that is not an error at all and is regularly mistaken for one. An SO-100 has five joints plus a gripper. Six independent joints are what a general arm needs to reach an arbitrary position and orientation, so a five-joint arm cannot reach every pose: for many positions, the orientation is not yours to choose. If your calibration seems fine at some points and impossible at others, check whether the arm can physically make the requested pose before concluding the mapping is wrong.

There is a quieter version of the same trap. The platform carries two joint-limit tables that do not agree joint for joint: the backend clamps in degrees, the browser solver in radians, and converting one to the other shows the gap. The browser allows the base to swing 110 degrees either way where the backend allows 150; for the shoulder lift it runs the other way round, about 100 degrees in the browser against 90 in the backend. A target one of them accepts is therefore not automatically one the other will pass through unclamped, and if you generate targets from a mapping using one table and send them through the other, some of them come back moved.

Checking the result instead of believing it

A hand-eye calibration will always report that it fits the data it was fitted on. That number tells you nothing you want to know. The only figure worth quoting is the error at points the fit never saw, and producing one takes about ten extra minutes.

1. Hold points back. Collect more points than you fit on — twenty collected, sixteen fitted, four held back — and report the error on the held-back four. If that number is much worse than the fit error, the mapping has memorised your point layout.
2. Test by placing, not by looking. Ask the arm to drive to a held-back point and measure where the gripper actually lands with a ruler. This catches the whole chain, including the parts a numerical residual cannot see.
3. Approach from two directions. Drive to the same target from the left and from the right and measure both. The gap between them is your backlash, and it is a floor under any accuracy you can claim.
4. Send the arm home and come back. Home the arm, then return to a test point. A mapping that only holds while the arm stays in one region will show itself here.
5. Repeat at the end of a session. Re-measure two or three points after an hour of work. Drift over a session usually means something is moving that you assumed was fixed, and it is much cheaper to find now than inside a failed run.
6. Write down what the mapping was fitted against. Which camera hardware id, which joint calibration, which camera position. A mapping without that record cannot be invalidated when one of them changes, which means it will quietly be used after it stopped being true.

Set the bar with the task rather than with a round number. Placing a 25 mm block into a 40 mm opening tolerates far more error than seating it against a fixture. The MHS run reports 3.0 mm placement accuracy, and separately a best run of twelve bricks placed, and it is the task that makes that number mean something. Decide what your task needs before you decide whether your calibration is good enough, and if you want to watch an arm run against a live setup before building any of this, there is one at the live arm.

The next step depends on which side of the fork you are on. If you are recording demonstrations and training a policy, none of this is needed and data collection and the teleoperation documentation are the path. If you are pointing an agent at the arm, the calibration described here is the piece you will have to build, and the client plus the local MCP server give you the readings to build it from. And before you quote a figure of your own, it is worth reading how little a single session establishes: that is the subject of our SO-ARM101 calibration report.