Cross-Embodiment Learning: What Transfers Between Robots

What cross-embodiment learning actually is

A policy trained on one robot is usually useless on another. Change the link lengths, swap the gripper, move the wrist camera two centimetres, and the mapping from pixels to joint targets no longer holds. Cross-embodiment learning trains one policy on data from many robots at once, so whatever is shared is learned once and only the robot-specific part is relearned.

The evidence is the Open X-Embodiment collaboration, which pooled 60 robot datasets from 34 labs into one format. The paper reports 22 robots across 21 institutions, 527 skills and 160266 tasks; the project page puts the pool above one million real trajectories.

The three mismatches every system has to solve

Strip away the branding and every cross-embodiment architecture answers the same three questions. Keep them separate: a design can be clever about one and naive about the others.

The fourth row gets skipped in papers and eats a weekend in practice. Two SO-100 arms from the same kit, given the same calibration, still disagree by a few degrees per joint. A model that generalises across a Franka and a WidowX can still fail because your shoulder sags under load. See policy only works in one setup.

Four architectures, four places to put the seam

RT-X: standardise the action space up front

RT-X is blunt. Each dataset's actions are converted into the same 7-DoF end effector command (x, y, z, roll, pitch, yaw, gripper opening, or their rates), normalised per dataset, then tokenised into 256 bins along each of eight dimensions: those seven plus one for episode termination. What the paper does not do is align coordinate frames across datasets, and it lets values mean either absolute or relative positions or velocities, whichever the original robot used, so the same action vector can induce very different motions on two robots. The cost is coverage: the mixture used across all RT-X experiments is 9 manipulators from 12 datasets, not the full 22-embodiment pool.

Octo: readout tokens and a swappable head

Octo is a transformer trained on 800k Open X-Embodiment trajectories. Task, observation and learned readout tokens enter a block-wise masked transformer where a readout token attends to earlier observations and tasks but is never attended to in return, and a diffusion head turns readout embeddings into action chunks. Because nothing depends on the head, you can replace it: the paper states that pretrained transformer weights are wholly retained.

Octo ships as Octo-Small at 27M parameters and Octo-Base at 93M. Each reported fine-tuning setup uses around 100 in-domain demonstrations and finishes in under 5 hours on one NVIDIA A5000, including cases with a new observation input and a new action space. Both sit in the model arena as octo-base and octo-small.

CrossFormer: one head per embodiment class

CrossFormer drops the requirement that action spaces be aligned at all. Trained on 900K trajectories across 20 embodiments at 130M parameters, it controls single and dual arm manipulators, wheeled navigation robots, quadcopters and quadrupeds with the same weights. Images go through ResNet-26 encoders, proprioception is projected directly, and missing observation types are masked so every batch element keeps all token groups at fixed positions.

It is tempting to read the chunk lengths as a control-rate adapter, and the bimanual row invites it: 100 steps at 20 Hz is five seconds of committed motion. The quadruped row kills that story, since it also runs at 20 Hz and predicts one action. The paper says chunk sizes are taken from prior work in each setting, not derived. What is true is that action chunking is what lets one trunk serve different rates. CrossFormer's entry is at crossformer.

HPT: stems, trunk, heads

Heterogeneous Pre-trained Transformers name the parts. Embodiment-specific stems tokenise proprioception and vision into a fixed token count, a shared trunk processes them, and heads emit actions; exactly one stem and head pair is active per embodiment. The scaling study runs to 52 datasets and reports over 20 percent improvement on unseen tasks. Checkpoints run from hpt-small to hpt-xlarge.

Embodiment tags in practice: how GR00T N1.7 does it

An embodiment tag is what the idea looks like as a CLI flag on a production vision-language-action model. In GR00T N1.7, every command takes --embodiment-tag, which picks the modality config: which state and action keys exist, how they are normalised, which projector runs. The model card describes proprioception as an MLP indexed by embodiment ID with inputs padded to a configurable maximum, and actions decoded by an MLP, one per unique embodiment.

# gr00t/data/embodiment_tags.py (Isaac-GR00T main, N1.7)
class EmbodimentTag(Enum):
    # Pretrain tags: baked into nvidia/GR00T-N1.7-3B, ready for zero-shot
    OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT = "oxe_droid_relative_eef_relative_joint"
    XDOF                                  = "xdof_relative_eef_relative_joint"
    XDOF_SUBTASK                          = "xdof_relative_eef_relative_joint_subtask"
    REAL_G1                               = "real_g1_relative_eef_relative_joints"
    REAL_R1_PRO_SHARPA                    = "real_r1_pro_sharpa_relative_eef"
    # ... plus _HUMAN / _MAXINSIGHTS / _MECKA variants of R1 Pro Sharpa

    # Pre-registered posttrain tags: require a finetuned checkpoint
    UNITREE_G1         = "unitree_g1_full_body_with_waist_height_nav_cmd"
    UNITREE_G1_SONIC   = "unitree_g1_sonic"
    SIMPLER_ENV_GOOGLE = "simpler_env_google"
    SIMPLER_ENV_WIDOWX = "simpler_env_widowx"
    LIBERO_PANDA       = "libero_sim"

    # Finetune-only tags, in no shipped checkpoint. The first is the one you will use.
    NEW_EMBODIMENT         = "new_embodiment"
    ROBOCASA_GR1_TABLETOP  = "robocasa_gr1_tabletop"
    ROBOCASA_PANDA_OMRON   = "robocasa_panda_omron"

The pretrain tags are DROID, a generic X-DOF description (X being a variable number of degrees of freedom), a Unitree G1 humanoid and R1 Pro variants. There is no SO-100 tag. Your SO-100, Koch v1.1 and LeKiwi all land in the same bucket: NEW_EMBODIMENT.

Telling the model what your action vector means

Two files do the work. The dataset carries a meta/modality.json slicing the flat state and action arrays into named fields, and a Python modality config registers those fields under EmbodimentTag.NEW_EMBODIMENT. For the SO-100 example that ships with the repo, the whole robot is six numbers.

{
  "state":  { "single_arm": {"start": 0, "end": 5},
              "gripper":    {"start": 5, "end": 6} },
  "action": { "single_arm": {"start": 0, "end": 5},
              "gripper":    {"start": 5, "end": 6} },
  "video":  { "front": {"original_key": "observation.images.front"},
              "wrist": {"original_key": "observation.images.wrist"} },
  "annotation": { "human.task_description": {"original_key": "task_index"} }
}

# examples/SO100/so100_config.py (abridged, comments are upstream)
so100_config = {
    "video": ModalityConfig(delta_indices=[0],
                            modality_keys=["front", "wrist"]),
    "state": ModalityConfig(delta_indices=[0],
                            modality_keys=["single_arm", "gripper"]),
    "action": ModalityConfig(
        delta_indices=list(range(0, 16)),   # predict 16 future steps
        modality_keys=["single_arm", "gripper"],
        action_configs=[
            # single_arm: RELATIVE = delta from current state (better generalization)
            ActionConfig(rep=ActionRepresentation.RELATIVE,
                         type=ActionType.NON_EEF,   # joint-space, not end-effector
                         format=ActionFormat.DEFAULT),
            # gripper: ABSOLUTE = target position (binary open/close works better absolute)
            ActionConfig(rep=ActionRepresentation.ABSOLUTE,
                         type=ActionType.NON_EEF,
                         format=ActionFormat.DEFAULT),
        ]),
    "language": ModalityConfig(delta_indices=[0],
                               modality_keys=["annotation.human.task_description"]),
}

register_modality_config(so100_config, embodiment_tag=EmbodimentTag.NEW_EMBODIMENT)

Shared action spaces: standardise, pad, or both

The padding number bites. In openpi, Pi0Config.action_dim defaults to 32 and action_horizon to 50, for Pi0 and Pi0.5 alike. A six-channel SO-100 vector, five arm joints plus the gripper, fills 6 of 32 slots; the other 26 are zeros the model still predicts. Normalisation statistics must therefore be computed with the padding you serve with, which is how you get loss falls, policy does nothing.

What the dataset format does and does not carry

Cross-embodiment training needs a container that holds data from different robots without lying about it. The LeRobot dataset format is the de facto standard. In v3.0: meta/info.json for schema, FPS and path templates, meta/stats.json for normalisation statistics, meta/tasks.jsonl for language annotations, meta/episodes/ for per-episode records, plus shards holding many episodes each.

Here is the honest bit. robot_type does exist in the LeRobot info schema, but in the source it sits under a comment reading # Optional metadata and is typed str | None = None. No trainer keys off it. To learn what your action vector means, GR00T reads modality.json and the modality config you passed.

The manual path: fine-tuning GR00T N1.7 on a new embodiment

This is the sequence NVIDIA documents for a custom arm, against the main branch of Isaac-GR00T. The repo ships demo_data/cube_to_bowl_5, five real SO-100 episodes that already carry a modality.json, so you can run the whole thing before you own an arm.

1. 1
   Convert your dataset to the v2 flavour GR00T expects

   GR00T reads a GR00T-flavoured LeRobot v2 dataset. Anything from a current lerobot is v3.0 and must be converted down. The converter lives in its own environment because it still needs a lerobot that can read v3.0. Skip this if you use the shipped demo data.

   bash

   cd scripts/lerobot_conversion
   uv venv
   source .venv/bin/activate
   uv pip install -e . --verbose
   python convert_v3_to_v2.py --repo-id <YOUR_DATASET_REPO_ID>

2. 2
   Put a modality.json in the dataset

   This file tells the model which slice of the flat action vector is the arm and which is the gripper. Without it the projector does not know what it is predicting. For a standard SO-100 or SO-101 recording the shipped example is already correct.

   bash

   cp examples/SO100/modality.json <YOUR_DATASET_ROOT>/meta/modality.json

3. 3
   Register the modality config under NEW_EMBODIMENT

   Only one NEW_EMBODIMENT modality config may be registered per Python process, so do not import two into the same script. In CLI use the file passed to --modality-config-path is the only one imported.

   python

   register_modality_config(so100_config,
                            embodiment_tag=EmbodimentTag.NEW_EMBODIMENT)

4. 4
   Launch the fine-tune

   A tyro CLI; run --help first, because the flag set changes between releases. Note that --global-batch-size is summed across GPUs before gradient accumulation, not after. --save-steps is lowered to 500 so you get intermediate checkpoints.

   bash

   CUDA_VISIBLE_DEVICES=0 uv run python \
       gr00t/experiment/launch_finetune.py \
       --base-model-path nvidia/GR00T-N1.7-3B \
       --dataset-path ./demo_data/cube_to_bowl_5 \
       --embodiment-tag NEW_EMBODIMENT \
       --modality-config-path examples/SO100/so100_config.py \
       --num-gpus 1 \
       --output-dir /tmp/so100 \
       --save-total-limit 5 \
       --save-steps 500 \
       --max-steps 2000 \
       --global-batch-size 32 \
       --dataloader-num-workers 4

5. 5
   Check that the fine-tune learned something

   Open-loop evaluation replays a training trajectory and overlays predicted against ground-truth actions. NVIDIA publishes no target MSE, because a fixed number would not transfer between datasets. Look for the trend across checkpoints and a curve that tracks the ground-truth shape rather than sitting flat.

   bash

   uv run python gr00t/eval/open_loop_eval.py \
       --dataset-path ./demo_data/cube_to_bowl_5 \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path /tmp/so100/checkpoint-2000 \
       --traj-ids 0 \
       --execution-horizon 16 \
       --steps 400 \
       --modality-keys single_arm gripper

6. 6
   Serve it and close the loop

   Policy server and robot client are separate processes talking over ZMQ. With no trained model yet, start the server with --dataset-path and no --model-path: it replays recorded actions so you can verify wiring first. That replay path also requires --execution-horizon, and the server raises a ValueError without it.

   bash

   uv run python gr00t/eval/run_gr00t_server.py \
       --embodiment-tag NEW_EMBODIMENT \
       --model-path /tmp/so100/checkpoint-2000 \
       --device cuda:0 --host 0.0.0.0 --port 5555

NVIDIA's own run of that 2000-step command on one H100, evaluated on training trajectory 0, gave an average MSE of 87.5 at checkpoint 500, then 25.4, 13.2 and 10.0 at 1000, 1500 and 2000, with MAE falling from 5.63 to 1.76. Those absolute values are not a target. The shape is the signal.

What actually gets updated during a fine-tune

The default tuning fields in the Isaac-GR00T fine-tune config are the clearest statement of where the seam sits in this model.

Frozen perception, trained action path. The part that understands scenes is shared and untouched; the part that talks to your servos is what your fine-tuning run pays for. The AY-Robots catalog puts the trainable slice of GR00T N1.7 at roughly 40 M parameters out of about 3 B, which is the same story from the other end.

Two ways to get there

What actually transfers, and what does not

Read it as a gradient. Transfer is strongest between close relatives, which is why the SmolVLA line is the most reliable one here, its pretraining pool being SO-100 community data and its reported real-world tasks covering SO100 and SO101 alike, and why SO-100 versus SO-101 is a smaller gap than SO-100 versus Koch. It is weakest, and can go negative, when a shared model is capacity-limited against a specialist trained on a large single-robot dataset.

What this means if you own exactly one arm

Almost nobody reading this will train across 20 embodiments. The value for a single-arm owner is indirect and still large: somebody else paid for the pretraining, and your job is to teach the action path what your robot is.

• Pick the smallest model that clears your task. Run ACT as a baseline first: if an 80 M model at 20 ms solves it, a 3 B model at 152 ms is a downgrade on hardware.
• Keep the camera setup fixed across every episode. The transfer you get is largely visual and it evaporates when the wrist camera moves between recording and inference.
• Record more distinct starting positions rather than repetitions of one. Diversity is what the shared trunk can use; repetition mostly teaches the head.
• Write real language annotations. task_index mapped to one constant sentence wastes the language pathway. Recording your first dataset covers the mechanics.
• Compare against a from-scratch baseline before concluding that pretraining helped.

For background, the VLA overview covers the architecture family and the RT-2 article covers the web-knowledge half this page leaves out. For the route from nothing to a running policy, train your first policy is the short version and the training docs the long one.