How to Train Pi0.5 on Your Own Robot Data

Pi0.5 is the model you reach for when a policy has to work in a room it has never seen. It is also the most awkward of the five trainable policies here to fine-tune by hand: the upstream repository is JAX first, the LeRobot port moved deployment out of lerobot-record in 0.6.0 and made the base install too small to train with, and a full fine-tune does not fit on a 4090. Below: what flow matching costs at inference, the two command routes, and the 485 ms number that decides whether the result is usable.

What Pi0.5 actually is

Physical Intelligence published pi0 in October 2024: a flow matching vision-language-action model on a pretrained VLM: PaliGemma at 3 billion parameters plus a 300M action expert initialised from scratch, 3.3 billion in total. The paper reports pre-training on over 10,000 hours of robot data across 7 robot configurations and 68 tasks. More in the pi0 article.

Pi0.5 (arXiv 2504.16054, 22 April 2025) keeps that skeleton and changes the training diet: web data, object detection, verbal instructions from humans coaching the robot, subtask labels, cross-embodiment data from robots in many homes. The result is a robot cleaning kitchens in houses that were not in the training set. The openpi README adds a detail the title does not: the released pi0.5 was trained with knowledge insulation (arXiv 2505.23705).

What flow matching changes about training and inference

A policy you can train on an SO-100 either regresses actions directly (ACT) or tokenizes them and decodes autoregressively (pi0-FAST). Flow matching does neither: it learns a vector field that transports noise to a clean action chunk, then integrates it. The pi0 paper uses 10 integration steps at step size 0.1; LeRobot's pi05 config carries the same num_inference_steps: int = 10.

• Training: the loss regresses a noised action chunk. No tokenizer to tune, no discretisation of your joint angles.
• Inference: one chunk costs ten forward passes through the action expert. That is structural, not a tuning problem.
• Output: continuous actions of dimension 32, padded internally, loss taken on the dimensions your robot has. A 6-DoF arm plus gripper uses 7.

# openpi/src/openpi/models/pi0_config.py - the defaults every pi05 config inherits
@dataclasses.dataclass(frozen=True)
class Pi0Config(_model.BaseModelConfig):
    dtype: str = "bfloat16"
    paligemma_variant: _gemma.Variant = "gemma_2b"
    action_expert_variant: _gemma.Variant = "gemma_300m"

    action_dim: int = 32
    action_horizon: int = 50
    max_token_len: int = None      # 200 if pi05 else 48

    pi05: bool = False             # flips discrete_state_input to True

The PaliGemma backbone, and the gate in front of it

PaliGemma (arXiv 2407.07726) is a SigLIP-So400m vision encoder on a Gemma-2B language model, about 3B parameters. Pi0.5 inherits its tokenizer, and that tokenizer sits in a gated repository. This is the most common way a first run dies, before the first training step.

Before you start: dataset format, episodes, hardware

Pi0.5 in LeRobot reads a LeRobot dataset in v3.0 layout, which landed with lerobot 0.4.0 in October 2025: many episodes per Parquet and MP4 file instead of one file per episode, with boundaries in meta/episodes/ rather than filenames. Record with the desktop client or follow record your first dataset and you have it.

# v2.1 -> v3.0, run against a dataset already on the Hub
python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=<HF_USER/DATASET_ID>

# aggregates episode-0000.parquet, episode-0001.parquet, ... -> file-0000.parquet
# aggregates episode-0000.mp4, episode-0001.mp4, ...         -> file-0000.mp4
# rewrites meta/episodes/* with per-episode lengths, tasks and offsets

The platform wants at least 50 episodes for Pi0.5, the same floor as GR00T and ACT. Pre-training does the heavy lifting, so 50 clean episodes beat 200 sloppy ones. New to this? Start with the data collection guide.

Route A: fine-tune with the openpi repository

The reference implementation: JAX first, tested on Ubuntu 22.04 only, driven by config objects rather than flags. A PyTorch path arrived in September 2025, validated on LIBERO. Three steps: convert your data, define a config, train and serve.

1. 1
   Clone and install

   openpi uses uv. GIT_LFS_SKIP_SMUDGE is what lets LeRobot come in as a dependency without LFS blobs.

   bash

   git clone --recurse-submodules git@github.com:Physical-Intelligence/openpi.git
   cd openpi
   
   GIT_LFS_SKIP_SMUDGE=1 uv sync
   GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .

2. 2
   Convert your data to a LeRobot dataset

   Upstream ships converters for LIBERO and DROID and expects you to adapt one. The work is the key mapping.

   bash

   uv run examples/libero/convert_libero_data_to_lerobot.py --data_dir /path/to/your/data

3. 3
   Add a training config

   Configs are TrainConfig entries in src/openpi/training/config.py. This one adapts pi05_droid_finetune, with the weight loader pointed at pi05_base.

   python

   TrainConfig(
       name="pi05_so100",
       model=pi0_config.Pi0Config(
           pi05=True,
           action_dim=32,        # pi05 is trained with 32-dim actions
           action_horizon=16,
       ),
       # Copy LeRobotLiberoDataConfig in the same file and rewrite the key
       # mapping for your camera names, then reference your copy here.
       data=LeRobotLiberoDataConfig(
           repo_id="your_hf_username/my_so100_dataset",
           base_config=DataConfig(prompt_from_task=True),
       ),
       weight_loader=weight_loaders.CheckpointWeightLoader(
           "gs://openpi-assets/checkpoints/pi05_base/params"
       ),
       batch_size=32,
       num_train_steps=20_000,
   )

4. 4
   Compute norm stats

   Runs against the config name, not the dataset. Skipping it is the classic first failure here.

   bash

   uv run scripts/compute_norm_stats.py --config-name pi05_so100

5. 5
   Train

   The variable lets JAX use 90 percent of GPU memory instead of the default 75. On an 80 GB card that decides whether the run survives.

   bash

   XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py pi05_so100 \
       --exp-name=my_experiment \
       --overwrite

6. 6
   Serve it

   The server listens on port 8000 and answers observation queries; your robot runtime is the client.

   bash

   uv run scripts/serve_policy.py policy:checkpoint \
       --policy.config=pi05_so100 \
       --policy.dir=checkpoints/pi05_so100/my_experiment/20000

Route B: fine-tune with lerobot pi05

The LeRobot port wraps the same weights in the CLI you already use for ACT and SmolVLA. Everything below was checked against lerobot 0.6.1, the release since 3 August 2026, and the pi05 docs on 23 August 2026. Versions matter here: v3.0 datasets arrived with 0.4.0 in October 2025, the 0.5.0 blog of 9 March 2026 presents pi0-FAST and Real-Time Chunking, and 0.6.0 on 6 July 2026 made the base package lightweight and moved deployment to lerobot-rollout.

One thing did not move: lerobot-train and lerobot-record were already entry points in 0.3.2, August 2025. A tutorial that still calls python -m lerobot.scripts.train predates a year of changes and is worth distrusting on the rest too.

1. 1
   Install with the right extras

   Since 0.6.0 the base install carries core ML dependencies only, and the pi extra adds no dataset or training code, so a fine-tune needs the training extra too. Older one-liners fail here at import time.

   bash

   # policy dependencies plus the training stack
   pip install 'lerobot[pi,training]'
   
   # from a source checkout
   pip install -e ".[pi,training]"
   
   # driving an SO-100 afterwards needs the robot extras too
   pip install 'lerobot[core_scripts,feetech]'

2. 2
   Authenticate

   Accept the paligemma-3b-pt-224 license in a browser, then log in on the machine that trains.

   bash

   hf auth login

3. 3
   Add quantile statistics

   Pi0.5 normalises STATE and ACTION with quantiles, so meta/stats.json needs q01 and q99. Older datasets carry only min, max, mean, std.

   bash

   lerobot-edit-dataset \
       --repo_id your_dataset \
       --new_repo_id your_dataset \
       --operation.type recompute_stats \
       --operation.overwrite true

4. 4
   Fine-tune from lerobot/pi05_base

   Set batch size to what your card survives; the docs use 64 on LIBERO at 80 GB. Pass bfloat16 explicitly, the config default is float32.

   bash

   lerobot-train \
       --dataset.repo_id=${HF_USER}/my_so100_dataset \
       --policy.type=pi05 \
       --policy.pretrained_path=lerobot/pi05_base \
       --policy.gradient_checkpointing=true \
       --policy.dtype=bfloat16 \
       --policy.device=cuda \
       --policy.push_to_hub=false \
       --output_dir=./outputs/pi05_so100 \
       --job_name=pi05_so100 \
       --batch_size=4 \
       --num_workers=8 \
       --steps=30000 \
       --save_freq=5000 \
       --seed=1000

5. 5
   Run it on the arm

   In 0.6.x this is lerobot-rollout: lerobot-record refuses a policy and rejects eval_ dataset names. Base drives the arm, sentry records the rollout.

   bash

   lerobot-rollout \
       --strategy.type=base \
       --policy.path=${HF_USER}/my_policy \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM1 \
       --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}}" \
       --task="Put lego brick into the transparent box" \
       --duration=60
   
   # same run, recorded into an eval_ dataset
   lerobot-rollout \
       --strategy.type=sentry \
       --policy.path=${HF_USER}/my_policy \
       --robot.type=so100_follower \
       --robot.port=/dev/ttyACM1 \
       --dataset.repo_id=${HF_USER}/eval_so100 \
       --dataset.single_task="Put lego brick into the transparent box" \
       --duration=600

Three sets of defaults, and which ones reach the trainer

openpi's TrainConfig is the reference, LeRobot's PI05Config is what lerobot-train uses when you say nothing, and the form here sends a third set. The surprise is the learning rate: both upstream defaults peak at 2.5e-5, while openpi's own pi05_libero config and this platform raise it to 5e-5.

Is the port faithful? The LeRobot team fine-tuned the LIBERO base model for a further 6k steps and compared with openpi's reference numbers on four suites: 97.5 percent average against 96.85, ahead on Libero 10, behind on Spatial. The route is a tooling choice, not a quality one.

The 485 ms reality, and where it stops being usable

This decides your project, so it comes before the cost table. The inference latency per action step measured here for Pi0.5 is 485 ms. Against the other four:

Upstream has an answer, and half of it is already in the release you can install. Real-Time Chunking generates the next chunk while the arm is still executing the current one, then guides the overlapping steps of the new chunk to stay close to the part already executed, so the arm does not stall or jerk at the seam. The inference-time form shipped in the 0.4.2 changelog for Pi0, Pi0.5 and SmolVLA and is a rollout flag, not a retrain:

lerobot-rollout \
    --strategy.type=base \
    --policy.path=${HF_USER}/my_policy \
    --inference.type=rtc \
    --inference.rtc.execution_horizon=10 \
    --inference.rtc.max_guidance_weight=10.0 \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM1 \
    --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}}" \
    --task="Put lego brick into the transparent box" \
    --duration=60

It does not make the model faster. It hides the gap: the 485 ms is still spent, but off the critical path, so the queue does not run dry between chunks. The training-time variant from arXiv 2512.05964 goes further: the model learns to condition on a clean action prefix, so at inference the overlap is hard-inpainted with per-action flow timesteps instead of guided, and the guidance backward pass disappears. During training it samples a prefix length per example and takes the flow loss on the remaining postfix. That flag lives on main only, not in 0.6.1, and the delay you train for has to stay below chunk_size.

Two ways to get a Pi0.5 checkpoint

lerobot-train \
    --dataset.repo_id=${HF_USER}/my_so100_dataset \
    --policy.type=pi05 \
    --policy.pretrained_path=lerobot/pi05_base \
    --policy.rtc_training_max_delay=10 \
    --policy.gradient_checkpointing=true \
    --policy.dtype=bfloat16 \
    --batch_size=4 --steps=30000 --seed=1000

When it does not work

• Dataset rejected: v3 required
• Out of memory during training
• Loss falls but the policy does nothing
• Policy freezes mid-motion
• Policy only works in one setup
• Training job stuck in queued

What one run costs

Pi0.5 sits in the expensive tier because it needs an 80 GB card, and spot prices move, so the figure is a range rather than a price. For many SO-100 tasks, train SmolVLA or ACT on the 24 GB tier first, confirm the task is learnable at all, then spend A100 hours on it. Train your first policy walks that cheaper loop, and pricing carries the current tiers.

Before committing an evening: GR00T N1.7 against Pi0.5 and Pi0.5 against SmolVLA lay the same numbers out head to head. The Arena holds 85 VLA models with 332 benchmark results, each linked to its source, and background on the family is in our VLA overview.