How to Train GR00T N1.7 on Your Own SO-100 Dataset

NVIDIA ships a fine-tuning example for exactly the arm you probably own. Inside the Isaac-GR00T repository there is a folder called demo_data/cube_to_bowl_5: five episodes, 4,148 frames at 30 fps, already written as LeRobot v2.1, with a matching modality config under examples/SO100/. Its meta/info.json reports robot_type: so101_follower, which in LeRobot is the same config class as so100_follower. That is genuinely useful, because it means the reference path for GR00T N1.7 on a six-degree-of-freedom hobby arm is maintained by the people who wrote the model. It is not a humanoid demo scaled down, it is the same arm.

The bad news is the distance between I recorded 60 episodes and the arm does the task. There are about six places where this pipeline fails quietly rather than loudly, and four of them live in files most people never open: meta/modality.json, the Python data config, meta/relative_stats.json, and the dataset's own version string. This guide walks the manual route end to end with the real commands, then shows the same job as a form on AY-Robots. Everything below was checked against the Isaac-GR00T main branch as of 20 August 2026 (the n1.7-release line) and lerobot 0.6.1, published to PyPI on 3 August 2026. Upstream moves fast, and where a flag was renamed this article says so.

What GR00T N1.7 actually is

GR00T N1.7 is a vision-language-action model with the dual-system layout described in the original GR00T N1 paper: a vision-language module that reads the cameras and the instruction, and a diffusion transformer that turns that into a chunk of continuous motor commands. N1.7 replaced the first half. The Eagle backbone from N1.6 is gone, swapped for nvidia/Cosmos-Reason2-2B on a Qwen3-VL architecture, and the model was pretrained on roughly 20,000 hours of egocentric human video on top of the robot data. NVIDIA's own write-up puts the figure at 20,854 hours and reports that going from 1k to 20k hours more than doubles average task completion.

The second half changed too, in ways that matter for your run. The action head dropped from 32 diffusion layers to 16, the predicted action chunk grew from 16 steps to 40, and the maximum state and action width went from 29 to 132. Those three numbers come from the changelog in the repository README; NVIDIA's own launch post still describes System 1 as a 32-layer DiT, so where the two disagree, trust the repo you are about to clone. Actions are expressed by default in a relative end-effector space, deltas from the current pose rather than absolute targets, which is what lets manipulation priors learned from human video transfer into robot control at all. The head itself is a flow-matching diffusion transformer, the same family as Pi0.5 but with a different backbone in front of it. If you are still on the previous generation, N1.7 against N1.5 covers whether the upgrade justifies redoing your pipeline.

Those last three rows explain most of the disappointment people report. The headline 27.9 ms is a TensorRT engine on an H100 with one camera and four denoising steps. Plain PyTorch on the same card is 85.8 ms, and the model card puts the gap at 3.08x. Neither number includes a serving layer, a second camera or a network hop. The 152 ms per action step AY-Robots quotes for its served GR00T N1.7 is the figure with serving in the loop, and a public-internet round trip sits on top of that. More on this at the end. For the numbers next to other models, GR00T N1.7 against Pi0.5 and GR00T N1.7 against SmolVLA lay them out side by side.

What the run needs before you type anything

Step 0: the episodes themselves

Everything below assumes you already have recorded episodes. If you do not, that is the real first step and it is the one that decides how good the result can be, because imitation learning cannot recover information that is not in the data. Calibrate both arms first, then drive the follower with a leader arm while lerobot-record writes the parquet files and the camera streams. If calibration is off, the joint values in your dataset describe a slightly different robot than the one that will later execute the policy, and no amount of training fixes that.

lerobot-record \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_follower_arm \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
    --teleop.type=so100_leader \
    --teleop.port=/dev/ttyACM1 \
    --teleop.id=my_leader_arm \
    --dataset.repo_id=${HF_USER}/cube-into-bowl \
    --dataset.num_episodes=60 \
    --dataset.single_task="put the cube in the yellow bowl" \
    --display_data=true

LeRobot's own advice is to record at least 50 episodes with about 10 per object location, keep the cameras fixed, and keep the grasping behaviour consistent. Add variation later, not at the start. The rule of thumb worth remembering: if you could not do the task yourself from the camera images alone, the policy cannot either. For the arm-specific setup, SO-100 getting started and the SO-100 LeRobot page cover ports, calibration and camera indices. On AY-Robots you can also do this over the internet from the browser using teleoperation and record straight from the session.

Step 1: the dataset has to be LeRobot v2.1

This is the single most common wall. LeRobot's current CODEBASE_VERSION on main is v3.0, so anything you record today with a current toolchain comes out as v3.0. GR00T's loader expects v2. The repository is explicit about why: many upstream datasets such as DROID, LIBERO and Bridge are published in v2, and native support for both is planned but not shipped. So the conversion is on you, and it runs in its own virtualenv for a concrete reason: scripts/lerobot_conversion carries its own pyproject that asks for Python 3.10 or 3.11 and pins lerobot to one git commit, while Isaac-GR00T itself requires Python 3.12. Install the converter from the repository root and you get the gr00t package instead, which is the mistake its README warns about. If you are new to the format, the LeRobot dataset glossary entry explains what is actually inside one.

# from the Isaac-GR00T repo root
cd scripts/lerobot_conversion
uv venv
source .venv/bin/activate
uv pip install -e . --verbose

# pulls the dataset from the Hub and writes a v2.1 copy into the default cache
python convert_v3_to_v2.py --repo-id <your-hf-user>/<your-dataset>

# or, back in the repo root, keep it next to the SO-100 example
uv run --project scripts/lerobot_conversion \
  python scripts/lerobot_conversion/convert_v3_to_v2.py \
  --repo-id <your-hf-user>/<your-dataset> \
  --root examples/SO100/my_dataset_lerobot

The structure GR00T wants after conversion is the classic v2 layout: meta/info.json, meta/episodes.jsonl, meta/tasks.jsonl, parquet files under data/chunk-000/, MP4 files under videos/chunk-000/observation.images./, and one extra file that standard LeRobot does not have. That extra file is where most of the remaining failures live.

Step 2: modality.json, the six numbers that decide everything

In a LeRobot dataset the robot state and the action are stored as flat float32 arrays. For an SO-100 both have shape [6]: five arm joints and a gripper. The demo dataset names them shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos, but those names live in info.json and nothing in the parquet file says which index is which. meta/modality.json supplies that mapping, and GR00T will not train without it. Here is the one the repository ships for the SO-100, verbatim.

{
  "state": {
    "single_arm": {
      "start": 0,
      "end": 5
    },
    "gripper": {
      "start": 5,
      "end": 6
    }
  },
  "action": {
    "single_arm": {
      "start": 0,
      "end": 5
    },
    "gripper": {
      "start": 5,
      "end": 6
    }
  },
  "video": {
    "front": {
      "original_key": "observation.images.front"
    },
    "wrist": {
      "original_key": "observation.images.wrist"
    }
  },
  "annotation": {
    "human.task_description": {
      "original_key": "task_index"
    }
  }
}

Copy it into your converted dataset at meta/modality.json and rename the video keys to whatever your cameras are actually called. If you recorded with a single overhead camera named top, then original_key is observation.images.top and the friendly name is whatever your data config will reference. The two have to agree, and neither of them checks the other for you. The language annotation is worse, because the same key has to appear in three places.

Step 3: the data config, relative arm and absolute gripper

The modality config is a Python file rather than JSON, because it also decides how each action group is represented. This is the part of the N1.7 workflow that did not exist in the same form in N1.5, and the part worth reading twice. The shipped SO-100 config predicts the five arm joints as RELATIVE deltas from the current state and the gripper as an ABSOLUTE target position, because a binary open-or-closed signal behaves better as a target than as a delta.

from gr00t.configs.data.embodiment_configs import register_modality_config
from gr00t.data.embodiment_tags import EmbodimentTag
from gr00t.data.types import (
    ActionConfig, ActionFormat, ActionRepresentation, ActionType, ModalityConfig,
)

so100_config = {
    "video": ModalityConfig(
        delta_indices=[0],                       # current frame only
        modality_keys=["front", "wrist"],        # must match modality.json
    ),
    "state": ModalityConfig(
        delta_indices=[0],
        modality_keys=["single_arm", "gripper"],
    ),
    "action": ModalityConfig(
        delta_indices=list(range(0, 16)),        # predict 16 future steps
        modality_keys=["single_arm", "gripper"],
        action_configs=[
            ActionConfig(rep=ActionRepresentation.RELATIVE,   # arm joints
                         type=ActionType.NON_EEF,
                         format=ActionFormat.DEFAULT),
            ActionConfig(rep=ActionRepresentation.ABSOLUTE,   # gripper
                         type=ActionType.NON_EEF,
                         format=ActionFormat.DEFAULT),
        ],
    ),
    "language": ModalityConfig(
        delta_indices=[0],
        modality_keys=["annotation.human.task_description"],
    ),
}

register_modality_config(so100_config, embodiment_tag=EmbodimentTag.NEW_EMBODIMENT)

Two details here will cost you a day if you do not know them. First, action_configs is positional: the docs require the same length and the same order as modality_keys, and they are blunt about the consequence of getting it wrong, which is that the wrong representation is applied silently. Your gripper gets trained as a delta and your arm as an absolute target, and there is no error message. Second, register_modality_config asserts that the tag is not already registered, so a second NEW_EMBODIMENT config in the same Python process dies with Embodiment tag ... already registered. You cannot import two of these into one script. A third rule is enforced later, at deployment: the action delta_indices must be the contiguous range starting at zero. A sparse window such as [0, 4, 8] is rejected, because everything downstream indexes the predicted chunk linearly and would otherwise execute the wrong rows.

Step 4: the environment

N1.7 moved the repository to uv and Python 3.12. The old conda plus pip install -e . path still exists in a collapsed section of the README, but it warns that GPU dependencies including flash-attn and TensorRT may need manual installation. Use uv unless you have a specific reason not to. On the flash-attn front, one detail saves confusion: you will see Installing flash-attn printed on every uv run. It is not rebuilding. uv is re-validating a URL-pinned wheel that is already cached, and it takes two or three seconds.

1. 1
   Install git-lfs, then clone with submodules

   git-lfs is required, not optional. Without it the parquet files in demo_data/ come down as pointer stubs, and the demo run fails on a dataset that looks present in the file listing.

   bash

   sudo apt install git-lfs && git lfs install
   git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T
   cd Isaac-GR00T

2. 2
   Install uv and sync the environment

   The default install pulls the GPU dependencies including flash-attn and TensorRT. On a fresh A100 or H100 image this is the longest single step, so do it before you start paying attention to anything else.

   bash

   curl -LsSf https://astral.sh/uv/install.sh | sh
   sudo apt-get update && sudo apt-get install -y ffmpeg
   uv sync --python 3.12
   uv run python -c "import gr00t; print('GR00T installed successfully')"

3. 3
   Authenticate against Hugging Face

   Do this before the first training launch, not after it fails eight minutes in.

   bash

   uv run huggingface-cli login   # or: export HF_TOKEN=<your_token>

4. 4
   Sanity-check on the shipped SO-100 demo data

   Before touching your own recording, run 2000 steps on demo_data/cube_to_bowl_5. It is five episodes, it finishes fast, and it proves the environment rather than your data. If this run fails, nothing you do to your dataset will help.

   bash

   CUDA_VISIBLE_DEVICES=0 uv run python \
       gr00t/experiment/launch_finetune.py \
       --base-model-path nvidia/GR00T-N1.7-3B \
       --dataset-path demo_data/cube_to_bowl_5 \
       --embodiment-tag NEW_EMBODIMENT \
       --modality-config-path examples/SO100/so100_config.py \
       --num-gpus 1 \
       --output-dir /tmp/test_finetune \
       --max-steps 2000 \
       --global-batch-size 32 \
       --dataloader-num-workers 4

Step 5: the fine-tune command and what its flags really default to

Swap the demo dataset for yours and add the knobs you actually want. Below is the full form the repository uses in its own new-embodiment tutorial, including the augmentation and checkpointing flags that the short README example leaves out. This is fine-tuning in the narrow sense: the language backbone and the visual encoder stay frozen, and what trains is the projector and the diffusion action head.

export NUM_GPUS=1
CUDA_VISIBLE_DEVICES=0 uv run python \
    gr00t/experiment/launch_finetune.py \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path ./my_dataset_lerobot \
    --embodiment-tag NEW_EMBODIMENT \
    --modality-config-path examples/SO100/so100_config.py \
    --num-gpus $NUM_GPUS \
    --output-dir /tmp/so100 \
    --save-total-limit 5 \
    --save-steps 2000 \
    --max-steps 20000 \
    --use-wandb \
    --global-batch-size 32 \
    --color-jitter-params brightness 0.3 contrast 0.4 saturation 0.5 hue 0.08 \
    --dataloader-num-workers 4

That last row is not a typo. launch_finetune.py is a tyro CLI generated from a dataclass, and that dataclass has no seed field. The README separately notes 5 to 6 percent variance between runs caused by non-deterministic image augmentation. Two runs with identical flags will not produce identical checkpoints, which matters a lot when you are trying to decide whether a hyperparameter change helped or whether you got lucky. For comparison, lerobot's own trainer defaults to seed 1000, and the LeRobot GR00T recipe passes --seed=42 explicitly.

What a 20000-step run costs

GR00T N1.7 needs an 80 GB card, so the cost question has a narrow answer. On AY-Robots the groot1.7 trainer runs on the A100 80 GB or H100 80 GB tier, where a run takes 3 to 6 hours at 1.20 to 2.00 USD per hour on the spot market. That is roughly 4 to 12 USD for the default 20000-step job. The same task on SmolVLA or ACT lands on a 24 GB card at 0.30 to 0.60 USD per hour and 1 to 3 USD per run. That is the real trade-off: GR00T costs about four times as much per attempt, and you cannot run it on the 4090 under your desk.

The 50-episode minimum is a floor, not a target. NVIDIA's own FAQ is more demanding: roughly 100 trajectories for a simple fixed-location pick and place, 500 or more for complex or multi-step scenes, and 100 to 500 for fine manipulation. If you are sitting at 20 episodes, spend the afternoon recording rather than the evening tuning. The data collection guide covers what separates a useful episode from a wasted one, record your first dataset is the short version, and SO-100 data collection is the arm-specific one.

Step 6: open-loop evaluation before you touch the arm

Do not put a fresh checkpoint on a physical arm to find out whether training worked. Run the open-loop evaluation first. It replays a recorded episode, asks the model for actions at each step, and plots prediction against ground truth with MSE and MAE. It costs nothing and it catches the mapping mistakes from steps 2 and 3.

uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path ./my_dataset_lerobot \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path /tmp/so100/checkpoint-20000 \
    --traj-ids 0 \
    --execution-horizon 16 \
    --steps 400 \
    --modality-keys single_arm gripper

The repository deliberately refuses to publish a target MSE for custom data, and that is the right call: the number depends on your action units, your task and your dataset size, so a threshold copied from someone else's arm means nothing. What is meaningful is the trend. Here is the reference run the repo documents on a single H100 with the five-episode demo dataset and 2000 steps.

The shape is the signal, not the absolute values. Error should fall steadily as training steps accumulate. Averaged over all five training episodes rather than trajectory 0 alone, the repo's final checkpoint scored roughly 7.5 MSE and 1.5 MAE, so even the reference run reads differently depending on which episodes you average. Record your own baseline on the unmodified demo command before you change anything about your own data: if you cannot reproduce a known-good run, you cannot tell a setup mistake from a data problem. The repository also maps the common symptoms to causes, and every one of them is operational rather than a model bug.

The other route: lerobot-train instead of Isaac-GR00T

The current LeRobot release, 0.6.1 on PyPI since 3 August 2026, offers a second and quite different way to fine-tune the same base weights. LeRobot exposes GR00T N1.7 as a policy type and trains it through its own lerobot-train entry point. Two things matter here. The LeRobot CLI is a set of console scripts, so anything you read that says python lerobot/scripts/train.py is stale and will not run. And LeRobot removed GR00T N1.5 support entirely, rejecting N1.5 checkpoints and configs with a migration note, so if you need N1.5 through LeRobot you have to pin lerobot==0.5.1, the last release that supports it, published 7 April 2026.

pip install "lerobot[groot]" "lerobot[training]"
hf auth login

lerobot-train \
  --dataset.repo_id=$HF_USER/$DATASET_NAME \
  --dataset.image_transforms.enable=true \
  --policy.type=groot \
  --policy.device=cuda \
  --policy.base_model_path=nvidia/GR00T-N1.7-3B \
  --policy.embodiment_tag=new_embodiment \
  --policy.chunk_size=16 \
  --policy.n_action_steps=16 \
  --policy.use_relative_actions=true \
  --policy.relative_exclude_joints='["gripper"]' \
  --policy.use_bf16=true \
  --seed=42 \
  --batch_size=64 \
  --steps=20000 \
  --save_freq=5000 \
  --output_dir=$OUTPUT_DIR

Two ways to get the same checkpoint

Getting the checkpoint back onto the arm

Isaac-GR00T uses a server-client split over ZMQ. The policy runs on the GPU, and a thin client on the robot machine sends observations and receives action chunks. The SO-100 example is complete enough to copy: start run_gr00t_server.py with your checkpoint and --embodiment-tag NEW_EMBODIMENT, then run eval_so100.py on the robot side with the serial port, the robot id, the camera indices and the language instruction. The camera names in that command must match the friendly names from your modality.json, not the OS device numbers.

# GPU side
uv run python gr00t/eval/run_gr00t_server.py \
  --model-path /tmp/so100/checkpoint-20000 \
  --embodiment-tag NEW_EMBODIMENT \
  --device cuda:0 \
  --host 0.0.0.0 --port 5555

# robot side, from gr00t/eval/real_robot/SO100
uv run --no-sync python eval_so100.py \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM2 \
  --robot.id=orange_follower \
  --robot.cameras="{ front: {type: opencv, index_or_path: 6, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
  --policy_host=localhost --policy_port=5555 \
  --lang_instruction="put the cube in the yellow bowl"

Now the honest part about where the policy runs, because training and serving have different hardware stories. Fine-tuning wants 40 GB or more. Inference does not: the README puts it at 16 GB or more and names the RTX 4090 explicitly, so a card you already own can serve a checkpoint it could never have produced. What decides whether the policy feels responsive is not VRAM, it is where the server sits. On AY-Robots GR00T N1.7 is cloud-only, so the control loop pays a public-internet round trip on top of the 152 ms per action step, and only SmolVLA and ACT also run locally. For slow pick and place a remote pod is survivable. For anything reactive it is not: the policy becomes hesitant in a way that looks exactly like a training failure and is not one. ACT at 20 ms per action step is the model that tolerates the tightest loop, SmolVLA sits at 245 ms, and no amount of latency tuning buys back a round trip that has already been spent. Run your first policy walks through the serving side end to end.

What actually goes wrong

• GatedRepoError on the first run. You have not been granted access to nvidia/Cosmos-Reason2-2B, or you did not authenticate. This happens after the GPU clock has already started.
• Dataset rejected on load. Almost always a v3.0 dataset. Convert it down. See dataset rejected as v3.
• IndexError about mismatched boolean dimensions. You changed delta_indices and did not regenerate the stats.
• Out of memory at batch 32. Reduce --global-batch-size and raise --gradient-accumulation-steps, or reduce --num-shards-per-epoch, which the config explicitly suggests when VRAM is limited. See out of memory during training.
• Loss falls, policy does nothing. There is no validation split by default, so a clean training curve proves very little. This page covers the diagnosis.
• Works in your setup and nowhere else. Expected with a small dataset filmed under one lighting condition. NVIDIA recommends colour jitter augmentation plus 20 to 50 episodes across different lighting. More here.
• Gripper never closes properly. Check that the gripper action is ABSOLUTE and the arm joints RELATIVE, in that order in action_configs. Gripper does not close lists the other causes.
• A camera silently drops out mid-recording. The episode still saves and the video key still exists, which is why this one is nasty. Camera not detected covers it.

The whole index of failure modes lives at the fix pages. If you are choosing between models rather than debugging one, the policy comparison and the arena entry for GR00T N1.7 have benchmark numbers with sources attached, and ACT against GR00T N1.7 is the comparison most people actually need, because it is the choice between a model you can train on the card under your desk and one you have to rent an 80 GB node to fine-tune. For background on why these models behave the way they do, the VLA overview and the SO-100 complete guide are worth reading first. And if you do not own an arm yet, the live arm streams a physical SO-100 with no signup.