Mobile Manipulation: What Changes When the Robot Base Moves

A table-top arm rests on one quiet assumption: the world holds still unless the gripper moves it. The camera sees the same background in frame 1 and frame 900, and a shoulder angle of 42 degrees means the same point in space every time. Much of the imitation learning stack leans on that without saying so.

Put the arm on wheels and it is gone. The background becomes a function of where you drove, joint angles no longer pin down a point in the room, and a new channel appears in the action vector that behaves nothing like the other six. What changes, using the LeKiwi implementation in LeRobot and the published results, and why the policy you trained on a fixed table will not carry over.

The action vector gets longer, and that is the smallest problem

In LeRobot 0.6 the LeKiwi class defines one state feature dictionary and reuses it for both the action space and the non-camera half of the observation. Since LeRobot dataset columns come straight from these keys, it is the fastest way to see what a mobile episode contains.

# src/lerobot/robots/lekiwi/lekiwi.py  (lerobot main, read 2026-08-24)
@property
def _state_ft(self) -> dict[str, type]:
    return dict.fromkeys(
        (
            "arm_shoulder_pan.pos",
            "arm_shoulder_lift.pos",
            "arm_elbow_flex.pos",
            "arm_wrist_flex.pos",
            "arm_wrist_roll.pos",
            "arm_gripper.pos",
            "x.vel",
            "y.vel",
            "theta.vel",
        ),
        float,
    )

@cached_property
def action_features(self) -> dict[str, type]:
    return self._state_ft          # action space == state space

Six of those behave like a fixed SO-100: write a goal, the servo goes there, read back where it is. The last three do not. x.vel is a rate. Writing 0.25 does not mean go to 0.25, it means keep moving until told otherwise. Where the robot ends up is the integral of everything you wrote, plus whatever the floor did to the wheels.

Velocity, not position: why drift is structural

LeKiwi's base observation is not a pose estimate. get_observation sync-reads Present_Velocity from wheel IDs 7, 8 and 9 and pushes the speeds through the inverse of a 3x3 matrix built from the mounting angles (240, 0, 120 degrees offset by -90), wheel radius 0.05 m, base radius 0.125 m. Nothing integrates the result into a pose, and the schema has no place to put one.

So the network never learns where the robot is. It learns what the wheels were doing and infers position from the image alone. Every source of base error is invisible to the state vector and gets corrected visually, or not at all.

The paper blames "the stochasticity of the ground contact and low-level controller", and states the general case up front: a small deviation in base pose leads to large drifts in the arm's end effector pose. The trigonometry is unforgiving. A 2 degree heading error is about 3.5 cm of lateral offset per metre of standoff, and centimetres are the difference between grasping a handle and knocking it over.

• Wheel slip. Omni wheels are rollers on a hub, built to slide sideways, which is what makes dead reckoning from wheel speed optimistic.
• Command latency. Mobile ALOHA measured a delay between target and actual base velocity far larger than on the arms. The two halves of your action vector do not land together.
• Loop jitter. The host runs at max_loop_freq_hz = 30, with a comment telling you to lower it if the robot jitters. Every overrun is distance nobody accounted for.

Three groups, three budgets, one design decision. Mobile ALOHA concatenates 14 joint positions with the base's linear and angular velocity into a 16-dimensional action, reusing existing imitation code almost unchanged. Pi0.5 does the same at 18 or 19 dims, adding a torso lift. The flow-matching action expert that consumes those dimensions is taken apart in the pi-zero write-up. None of the three feeds the policy an absolute pose.

Can the five policies even eat a 9-dimensional action?

Answer this before recording anything. Two pad the vector, one sizes itself from your data, two need a config edited by hand. Widths below come from the LeRobot and Isaac-GR00T sources read on 2026-08-24; episode counts and GPU tiers from the policy pages.

That padding is why SmolVLA and Pi0.5 are the least painful start: nine channels plus 23 zeros is a 32-vector, and the same checkpoint shape covers both. ACT is the opposite: the docs say it "will automatically adapt to the number of motor states, motor actions and cameras of your robot ... which have been saved in your dataset", which is convenient until you run a LeKiwi checkpoint on a plain SO-100 and the head has three columns too many.

# examples/SO100/so100_config.py in NVIDIA/Isaac-GR00T (read 2026-08-24)
so100_config = {
    "video": ModalityConfig(delta_indices=[0], modality_keys=["front", "wrist"]),
    "state": ModalityConfig(delta_indices=[0],
                            modality_keys=["single_arm", "gripper"]),
    "action": ModalityConfig(
        delta_indices=list(range(0, 16)),        # 16-step horizon
        modality_keys=["single_arm", "gripper"],
        action_configs=[
            ActionConfig(rep=ActionRepresentation.RELATIVE,  # delta from current state
                         type=ActionType.NON_EEF, format=ActionFormat.DEFAULT),
            ActionConfig(rep=ActionRepresentation.ABSOLUTE,  # gripper target
                         type=ActionType.NON_EEF, format=ActionFormat.DEFAULT),
        ],
    ),
    "language": ModalityConfig(delta_indices=[0],
                               modality_keys=["annotation.human.task_description"]),
}
register_modality_config(so100_config, embodiment_tag=EmbodimentTag.NEW_EMBODIMENT)

Note the two ActionConfig entries. The arm is RELATIVE, a delta from the current state, and NON_EEF, which the file itself comments as joint-space rather than end-effector; the gripper is ABSOLUTE. N1.7's headline relative end-effector action space is the other setting of that type field, not what the SO100 example uses. A base group forces the same call, and RELATIVE is wrong there: x.vel is already a rate, so a delta from it is an acceleration. Get that backwards and the model trains cleanly, then drives inexplicably.

What breaks when you move a table-top policy onto a base

Row four deserves a moment. The LeRobot advice for a fixed arm, "Keep the cameras fixed and maintain consistent grasping behavior throughout the recordings", is right for a table and also produces a dataset where the background pixel at (10, 10) predicts the joint angle. On wheels that shortcut is harmful: the pixel now encodes where you drove, not where the object is. The fixed-arm version is at policy only works in one setup.

The manual path: recording a whole-body LeKiwi dataset

This describes LeRobot main at 0.6.2; the latest tagged release when this was written was v0.6.1, 3 August 2026. LeKiwi splits across two machines: a Raspberry Pi 5 owning the servo bus and cameras, and your laptop owning the leader arm and the policy, over ZeroMQ on ports 5555 and 5556. The arm half of it is the ordinary SO-100 procedure, walked through end to end in the SO-100 setup guide.

1. 1
   Install the LeKiwi extra on both machines

   It pulls in the Feetech SDK and ZeroMQ. Both ends of the link are LeRobot code.

   bash

   pip install -e ".[lekiwi]"

2. 2
   Set motor IDs, arm first then wheels

   One control board carries everything. The command walks the arm from 6 down to 1, then the wheels as 9, 8, 7. The order is not optional.

   bash

   lerobot-setup-motors \
       --robot.type=lekiwi \
       --robot.port=/dev/ttyACM0

3. 3
   Calibrate the arms, skip the wheels

   Over SSH on the Pi for the follower, then on the laptop with --teleop.type=so100_leader. Middle of range, Enter, then sweep every joint.

   bash

   lerobot-calibrate \
       --robot.type=lekiwi \
       --robot.id=my_awesome_kiwi

4. 4
   Start the host on the robot

   This process owns the bus and both cameras. Its config carries a 500 ms watchdog that stops the base when no command arrives.

   bash

   python -m lerobot.robots.lekiwi.lekiwi_host \
       --robot.id=my_awesome_kiwi

5. 5
   Teleoperate both channels at once

   Set remote_ip and port in the example first. The leader arm drives six joints, w/a/s/d the base, z/x rotate, r/f step through three speed modes. The docs table lists them as 0.4, 0.25 and 0.1 m/s; lekiwi_client.py defines 0.1, 0.2 and 0.3 and starts slow. Believe the code.

   bash

   python examples/lekiwi/teleoperate.py

6. 6
   Record with both teleoperators in one loop

   The example instantiates an SO100Leader and a KeyboardTeleop and feeds both into record_loop. Defaults are small: NUM_EPISODES 2, FPS 30, EPISODE_TIME_SEC 30.

   bash

   python examples/lekiwi/record.py

7. 7
   Train, then roll out

   lerobot-train reads the shape of your dataset, so the nine columns propagate on their own. lerobot-rollout needs the robot as well or it has nothing to drive: lekiwi_client takes the same remote_ip the teleop example uses. For slow VLAs, --inference.type=rtc keeps chunked execution smooth on the rollout side.

   bash

   lerobot-train \
     --dataset.repo_id=${HF_USER}/lekiwi_pick \
     --policy.type=act \
     --output_dir=outputs/train/act_lekiwi \
     --policy.device=cuda
   
   lerobot-rollout \
     --strategy.type=base \
     --policy.path=${HF_USER}/act_lekiwi \
     --robot.type=lekiwi_client \
     --robot.remote_ip=<your_pi_ip> \
     --task="Fetch the cube from the side table" \
     --duration=60

Co-training: the result that made mobile imitation work

Your old fixed-table data is worthless as a checkpoint and valuable as a co-training ingredient. The Mobile ALOHA recipe is almost insultingly simple: sample 0.5 from the static dataset and 0.5 from the mobile one, zero-pad the base columns of the static episodes, normalise on the mobile statistics alone, batch size 16. No image tricks, no domain adaptation, even though the static data was shot on a black tabletop with the arms facing each other.

Two things matter more than the headline. The gains sit in the precision sub-tasks: a faucet knob 0.7 cm in diameter, a 2 cm by 2 cm elevator button, the steps where the arm must visually servo to compensate for base error. And the last row is flat, 85 percent either way. Co-training fixes a specific weakness rather than multiplying performance, and reporting the row where it did nothing is why it is credible.

Pi0.5 publishes the scaling curve: about 400 hours of mobile manipulator data from about 100 homes, yet 97.6 percent of first-stage training examples come from elsewhere, including non-mobile robots and web images. Their ablation trained on 3, 12, 22, 53, 82 and 104 environments, and average performance improves with more training locations, with the 104-location model matching a control trained directly on the test homes. Diversity of place bought the generalisation, which is why how you collect the data outranks which model you pick.

Two routes to a trained mobile policy

Latency stops being an abstraction when the robot is moving

On a fixed arm, inference latency costs smoothness. On a moving base it costs position. Check which speed you are actually getting before you do this arithmetic: the LeRobot LeKiwi page prints a table of 0.4, 0.25 and 0.1 m/s, while lekiwi_client.py defines speed_levels of 0.1, 0.2 and 0.3 m/s and starts at the slow one. The wheels follow the code. The table below uses the code's fast mode of 0.3 m/s and its slow mode of 0.1 m/s, multiplied by the per-step inference time from the policy specs.

Those are multiplications, not measurements, and action chunking cuts both ways. SmolVLA and Pi0.5 predict 50 steps per call and ACT predicts 100, so the model is not consulted every step and the figure is amortised. What chunking does not fix is staleness: the last action in a 50-step chunk came from an observation 50 frames old, taken somewhere else in the room. That is why Mobile ALOHA staggers a chunk, executing the first k minus d arm actions alongside the last k minus d base actions.

Before you record 50 episodes

1. Confirm all nine keys appear in a recorded frame. Constant zero in x.vel means the keyboard teleoperator never connected, and you are recording a fixed-arm dataset with three dead columns.
2. Record in more than one location even when the task is identical.
3. Keep the wrist camera in the observation set. It is what lets the arm correct base error visually.
4. Keep a fixed-table dataset for the same task. You are going to co-train with it.

Questions people actually ask

Sources