Can a language model control a robot arm?Yes as the operator. No as the controller.
A language model is good at deciding what should happen, calling a tool and explaining what went wrong. It is not the thing that sends joint targets to a servo bus many times a second. That is what a vision-language-action policy is for. This page draws the line between them.
- 9
- Tools a local agent can call on a real arm
- 2
- Of them refuse to run without an explicit confirmation
- 20 ms
- One action step of the quickest supported policy, ACT
- SO-100 + SO-101
- Arm families with full support on this platform
A language model does not produce joint angles in the control loop.
That one sentence is the whole page. Everything below is what follows from it, and what you can build once you accept it.
What it cannot do
Continuous control is a closed loop: look, predict the next positions, send them, look again, all before the object has moved. A model that answers in sentences, over a network, cannot sit inside that loop, and no prompt fixes it. It is a timing problem, not a cleverness problem.
What it does instead
It sits one level above the loop. It reads the state of the arm, decides a demonstration is worth recording, opens the buffer, closes it, lists what landed on disk, and says why the last attempt looked wrong. Separate decisions, with time to think in between.
Why the two get confused
Both are called AI and both take a sentence as input. A VLA policy is language conditioned too. The difference is not the input. It is what comes out, and how often.
Six jobs, and each one has exactly one owner.
Nothing in this table is shared. Most disappointing robot demos come from handing a line to the wrong side.
| What has to happen | Language model | VLA policy or ACT | Why it sits there |
|---|---|---|---|
| Turning a vague request into a task | Somebody says tidy the table. Turning that into a series of runs under one task wording is language work. | ||
| Calling the tools | Read the status, start a recording, stop it, list the datasets. Discrete calls, each with a result to check. | ||
| Explaining and debugging | Reading a failed run, naming the likely cause, proposing the one thing to change next. | ||
| Seeing the scene | Camera frames in, an internal idea of where the object is now. Never put into words, never leaves the model. | ||
| Producing the motion | The next chunk of joint targets, conditioned on the current image. This is the part learned from your demonstrations. | ||
| Holding the rate | One action step takes roughly 20 to 485 ms depending on the family, and it repeats for as long as the task runs. |
The agent decides and orchestrates, the policy sees and moves, and the two meet only when a run is started or stopped.
Want the practical version instead: an agent, your arm, and the first safe commands?
Claude and your robot arm9 tools on a real arm, 2 of them behind a lock.
A local MCP server runs on your own machine, next to the robot app, and exposes exactly this list to an agent that speaks MCP. Nothing beyond it is reachable.
Read the state
Free to call and free of consequences: backend and hardware status, the ports and cameras found, the current joint positions, whether a recording runs, and the datasets on disk.
- get_status
- list_devices
- get_joint_positions
- get_recording_status
- list_local_datasets
Move the arm
One drives to the calibrated home pose, the other sets a single named joint to an absolute angle. The backend clamps the target to the calibrated range and reports what reached the servo.
- move_home
- move_joint
Record episodes
One opens the recording buffer while you teleoperate, the other saves the episode or throws it away. Throw the failures away: a saved failure teaches the policy to fail.
- start_recording
- stop_recording
The interlock, word for word
The two moving tools are refused unless the call carries an explicit confirmation. An agent that tries without it gets this back, and nothing moves:
move_joint moves a physical robot arm. Ask the user to confirm, then call again with confirm: true.
So an agent cannot move your arm by accident, or by misreading a sentence. It has to ask you first, every time.
How to wire this to your own arm
A separate page, because it is a separate job: which server entry to add, what the agent sees, and the first safe commands. If you would rather type them yourself, the command-line client talks to the same backend.
Do not confuse it with the hosted server
A second MCP server on this site answers questions about the model families, the arms, the guides and the known fixes, and with an API key it reads your account and starts a cloud run. What it has no path to at all is a robot: it cannot see one and cannot move one.
The hosted server, and what it reachesThe families that actually do the moving.
These are the families you can fine-tune on your own episodes, with the figures the platform trains and serves with. No success rates: one measured without your task, your gripper and your light is worth nothing.
GR00T N1.7
A large pretrained vision-language-action model. The one to reach for when the dataset is solid and the object does not start in the same place twice.
- Card
- A100 80 GB or H100 80 GB
- Data
- From about 50 episodes
- Action step
- About 152 ms
Pi0.5
Flow matching instead of discrete steps, so the trajectories come out smooth. Aimed at contact-rich work like inserting and stacking, and it pays for that in latency.
- Card
- A100 80 GB or H100 80 GB
- Data
- From about 50 episodes
- Action step
- About 485 ms
SmolVLA
Small, language conditioned, and it trains on a consumer card. The cheap first run that tells you whether your data is worth scaling up.
- Card
- RTX 4090 or any card with 24 GB
- Data
- From about 30 episodes
- Action step
- About 245 ms
ACT
No pretraining and no language. It learns your one task from your demonstrations, and it is by far the quickest of the four at inference.
- Card
- RTX 4090 or any card with 24 GB
- Data
- From about 50 episodes
- Action step
- About 20 ms
And one older entry
The previous GR00T generation is still selectable, purely so that runs started before the current one can be reproduced. New projects have no reason to pick it.
Which one fits your task is not something this page can decide. The arena puts them side by side with the published benchmarks, and the training page shows what a run costs.
No arm and no graphics card yet? Some of this runs in a browser with no account at all.
Try it without hardwareA talking agent is one link. Here is the rest of it.
Nobody gets useful work out of a robot by prompting harder. The work is a loop of recording, training, serving and looking at what broke.
An arm, cameras, and the local app
The app runs on your own machine and talks to the arm over the serial bus. The agent talks to the app, never to the servos.
Episodes in LeRobot format
You teleoperate the task, the app records it as episodes. Same task wording every time, failures discarded rather than saved.
Training on a rented GPU
Pick a family, point at the dataset, start the run. The card is rented for the job and released when it finishes.
A checkpoint served warm
The trained policy sits on a GPU pod, and the arm asks it over HTTP for the next actions. On your own card the same thing runs locally.
Evaluation, then around again
Every policy gets its own evaluation folder. What failed there decides what gets recorded next.
The whole chain, end to end
Each link has its own moving parts, failure modes and cost. They are laid out together on the harness page instead of crammed into this one.
See the full harnessOr get the client and start recordingFour questions people ask before they build this.
Can I just hand an agent the serial port and let it drive?
You can expose the joint tool, and it will move one joint at a time after you confirm. What you will not get is control: every step is a round trip, and the arm is blind in between. For anything that reacts to the camera, train a policy and let the agent start it.
What is the actual difference between an LLM and a VLA model?
Both take language in. A language model emits tokens that mean something to a person or a tool. A vision-language-action model emits actions: joint targets, conditioned on the current camera image, at the rate the robot needs them. One can call the other, but not replace it.
Do I need a graphics card for any of this?
Not for the agent side, and not for training: training and cloud inference both run on rented GPUs, billed for the time they run. Your own card only changes where inference happens, and only for the two smaller models: SmolVLA and ACT run locally, GR00T and Pi0.5 are cloud-only here. Recording needs a real arm regardless.
Which robot arms does this work with?
The SO-100 and the SO-101 from the LeRobot family are fully supported, and everything here is tested on them first. A further 2 community arms are compatible: a different bus means a different driver, but the pipeline behind it is the same.
Related on AY-Robots
Desktop client
Record your own datasets with your arm and cameras, straight into LeRobot format.
Read moreCloud training
Fine-tune GR00T, Pi0.5, SmolVLA or ACT on a rented GPU, billed by the minute.
Read moreCloud inference
Serve a checkpoint in the cloud and watch a real arm run it in your browser.
Read moreGive the agent the tools. Give the policy the motion.
The split is not a limitation to work around. It is the reason the thing works: one side decides and explains, the other sees and moves, and neither pretends to be the other.