Synthetic Data for Robot Policies: Where Simulation Helps

Every few months somebody works out that recording fifty episodes by hand is slow, and asks whether a simulator could produce them instead. It is a fair question. The honest answer has three parts: generated data does help, it does not replace the real recordings, and the ratio between those two facts depends entirely on which kind of synthetic data you mean.

This page goes through what the published work actually measured, what you can run today on a low-cost arm such as the SO-100, and where the sim-to-real gap eats the gains. The short version, before the detail: the systems that report the biggest multipliers multiply a small set of real human demonstrations. They do not remove the need for them.

Four different things get called synthetic data

Before comparing numbers it is worth separating the families, because a paper reporting a 100x multiplier and a paper reporting a 5 point success gain are often describing the same pipeline from different ends. The common thread is that something in the LeRobot dataset was produced by a machine rather than recorded off a physical arm. What differs is which part.

Only the first three are synthetic data in the sense this article means. The fourth is worth mentioning because it gets bundled into the same conversation and is by far the cheapest thing on the list. If you have not already turned on the augmentations your trainer ships with, do that before you install a simulator.

What the published numbers actually say

Here is the evidence base, with the numbers as the sources state them rather than as the press releases summarise them. Every row below comes from a paper or project page read on 23 August 2026.

The pattern across every honest ablation is the same. Synthetic data widens coverage cheaply. It does not create information about your gripper, your servo sag, your lighting or your table height that was not in the real demonstrations somewhere. If your policy fails because the end effector closes half a second too late, no amount of simulated variation fixes it. That is a gripper timing problem in the real recordings.

One MimicGen finding is worth carrying into your own recording sessions, because it cuts against the usual advice. The project generated two datasets on Square D2, one seeded with 10 demos from a better-quality human operator and one with 10 demos from a worse-quality operator, both taken from the robomimic multi-human Square dataset. Policies trained on each achieved comparable results, which the authors read as a sign that in the large-scale data regime data quality might not matter as much. Read carefully, that is a statement about the ten seed demonstrations, not about your fifty real episodes. It means a slightly scrappy seed set is not the thing standing between you and a usable generated dataset. It does not mean the real episodes you co-train on can be scrappy, because those are the ones carrying the information the simulator does not have.

The sim-to-real gap, concretely

The gap is usually discussed as a single quantity, which is unhelpful. It is at least five separate mismatches, and they have different sizes on a 110 to 150 EUR hobby arm than they do on a Franka.

• Contact and friction. Isaac Lab states plainly that given the same hardware and the same Isaac Sim and PhysX version the simulation is reproducible, but that results vary across different hardware configurations due to floating point precision and rounding errors, and that PhysX does not guarantee determinism for any scene with non-rigid bodies such as cloth or soft bodies.
• Actuation. A Feetech STS3215 bus servo running at 7.4 V sags under load, has backlash, and changes behaviour as it heats. The MJCF model for the SO-101 borrows its motor parameters from an unrelated project rather than identifying them on your arm.
• Rendering. Camera noise, rolling shutter, auto-exposure, and the exact hue of your table are not in the render. This is the half of the gap that Cosmos-Transfer1 was built to close: its robotics augmentation workflow maps one robotics synthetic example to multiple realistic examples from segmentation, depth or edge conditioning.
• Timing. A simulator steps at a fixed rate. A real control loop does not, and the model itself costs 20 to 485 ms per action step depending on which one you picked. See inference latency.
• Object statistics. Simulated scenes are sampled from a distribution somebody wrote down. Your kitchen table is not.

What a simulated SO-100 actually knows about your arm

This is the part that decides whether any of the above is worth your weekend, and the first surprise is that the SO-100 and the SO-101 are not equally served. TheRobotStudio's SO-ARM100 repository keeps its simulation assets under Simulation/. The SO100 folder contains a single URDF file and nothing else. The SO101 folder contains both URDF and MuJoCo files: scene.xml, so101_new_calib.xml, so101_old_calib.xml, the matching URDFs and a joints_properties.xml. If you want a physics model rather than a kinematic chain, you want the SO-101 files.

They were generated with the onshape-to-robot plugin from a CAD model designed in Onshape, which means the kinematics and the visual meshes are as good as the CAD. The dynamics are a different story, and the repository's own README is candid about three things. Base collision meshes were removed due to problematic collision behaviour during simulation and planning. The STS3215 motor properties are adapted from the Open Duck Mini project rather than measured on an SO-101. And the LeRobot gripper convention, where 0 is fully closed and 100 is fully open, is explicitly not yet reflected in the URDF and MuJoCo files. Every one of those is a place where a policy trained purely in that model will behave differently on your desk.

Domain randomization, and what it does not fix

The standard answer to the gap is to stop trying to match reality and instead train across a distribution wide enough that reality falls inside it. Tobin and colleagues showed the strong version of this in 2017: an object detector trained only on simulated images with non-realistic random textures, with no pre-training on real images at all, localised real objects to an accuracy of 1.5 cm and stayed robust to distractors and partial occlusions.

Isaac Lab exposes the same idea as event terms you attach to an environment config. These are the knobs, by their actual function names in isaaclab.envs.mdp, so you can go and read them rather than guess.

Here is the limit, and it is the one people trip over. Randomization widens the distribution the policy has seen inside the model you built. It cannot introduce a physical effect the simulator does not represent. If PhysX is not modelling the backlash and thermal droop of your STS3215 servos, randomizing their stiffness teaches the policy nothing about backlash. That is why an arm that twitches then sags under a sim-trained policy is not a randomization-budget problem. It is a modelling problem.

The manual path: generating data in Isaac Lab

Isaac Gym is legacy software. NVIDIA's own page is headed "Isaac Gym - Now Deprecated" and says developers may download and continue to use it but that it is no longer supported, pointing at Isaac Lab instead. If you want the history, we covered both: Isaac Gym and GPU-native physics and Isaac Lab. For new work in 2026, start at Isaac Lab.

1. 1
   Install Isaac Sim and Isaac Lab

   The pip installation page states that the instructions are for Isaac Sim 5.X, which requires Python 3.11. The source clone gives you the scripts the next steps need.

   bash

   pip install "isaacsim[all,extscache]==5.1.0" \
       --extra-index-url https://pypi.nvidia.com
   
   git clone https://github.com/isaac-sim/IsaacLab.git --branch main
   cd IsaacLab
   sudo apt install cmake build-essential
   ./isaaclab.sh --install
   
   # smoke test
   ./isaaclab.sh -p scripts/tutorials/00_sim/create_empty.py

2. 2
   Record about ten human demonstrations

   The Isaac Lab documentation is specific: about 10 successful demonstrations are required for the following steps to succeed. Its tips are equally specific. Keep demonstrations short, take a direct path rather than moving along arbitrary axes, and do not pause, because it is not obvious to a policy why and when to pause.

   bash

   ./isaaclab.sh -p scripts/tools/record_demos.py \
       --task Isaac-Stack-Cube-Franka-IK-Rel-v0 \
       --device cpu \
       --teleop_device spacemouse \
       --dataset_file ./datasets/dataset.hdf5 \
       --num_demos 10

3. 3
   Annotate the subtask boundaries

   Mimic splits the input demonstrations into subtasks so it can re-time and re-target the segments. The --auto flag does this without a human in the loop for tasks that define automatic annotation; without it you pause with B, continue with N and mark a boundary with S. Note the task id gains a -Mimic suffix.

   bash

   ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
       --device cpu \
       --task Isaac-Stack-Cube-Franka-IK-Rel-Mimic-v0 \
       --auto \
       --input_file ./datasets/dataset.hdf5 \
       --output_file ./datasets/annotated_dataset.hdf5

4. 4
   Generate the multiplied dataset

   This is the step that turns 10 into 1000. Mimic applies a boolean success criterion to each candidate and only keeps the ones that completed the task, so the output count is lower than the trial count. The docs put that candidate success rate as high as 70 percent in simple cases and below 1 percent for hard tasks and complex robots: about 50 percent for the Franka cube stack, and 65 to 80 percent for the GR1T2 pick and place, where 1000 demos take 18 to 40 minutes (19 minutes on an RTX ADA 6000 at 80 percent).

   bash

   ./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
       --device cpu \
       --num_envs 10 \
       --generation_num_trials 1000 \
       --headless \
       --input_file ./datasets/annotated_dataset.hdf5 \
       --output_file ./datasets/generated_dataset.hdf5

5. 5
   Convert HDF5 to a LeRobot dataset

   Everything above produces robomimic-flavoured HDF5, and Isaac Lab core ships no LeRobot converter of its own: its docs only say you may convert the generated dataset to LeRobot format. Two projects supply the actual converter. IsaacLab-Arena ships a GR00T-targeted one driven entirely by a YAML config, and LeIsaac ships its own pair for the SO-101 route (see below).

   bash

   # IsaacLab-Arena, GR00T LeRobot format
   python isaaclab_arena_gr00t/lerobot/convert_hdf5_to_lerobot.py \
       --yaml_file isaaclab_arena_gr00t/lerobot/config/gr1_manip_config.yaml

Why generation attempts fail, and what to change

A candidate success rate that swings between 70 percent and under 1 percent is not a mystery, and Isaac Lab documents the common pitfalls rather than leaving you to guess. Every one of them is something you control at recording time, which is why it pays to read this list before you record the ten seed demonstrations rather than after the first disappointing generation run.

• Demonstrations are too long. A longer time horizon is harder for a policy to learn. Start close to the first object and minimise motion.
• Demonstrations are not smooth. Irregular motion is hard for a policy to decipher, and better teleoperation hardware gives better data: the docs say plainly that a SpaceMouse beats a keyboard.
• Pauses. Pauses are difficult to learn, because it is not obvious to a policy why and when to pause. Keep the motion fluid.
• Too many subtasks. More subtasks mean more stitching between trajectory segments, which yields less smooth motion and a lower generation success rate. Annotate boundaries where the arm is unlikely to collide with anything.
• No action noise. Action noise makes the resulting policies more robust.
• Recording cropped too tight. If recording stops on the exact frame the success term triggers, it may not re-trigger during replay. Leave a buffer at the end.
• Non-deterministic replay. Physics in Isaac Lab are not deterministically reproducible across env.reset, so some human demos fail on replay. Collect more than you need and keep the ones that survive annotation. Everything that lands in a Mimic-generated HDF5 file is a successful demo and can be used for training even if replay later fails.

The interpolation step between stitched subtask segments has its own tuning knob, and the number of interpolation steps you need scales with how fast the robot moves and how wide the object reset distribution is. A complex task with a large reset distribution leaves larger gaps between segments, which needs more interpolation steps to come out as continuous motion. If your generated videos show the arm lurching between phases, that is the parameter to look at before you blame the seed demos.

The same pipeline on an SO-101, with the real leader arm

This is the interesting one for anyone reading this page, because it is the only open pipeline that puts an SO-101 inside Isaac Lab and lets you drive it with the physical leader arm you already own. LeIsaac, version 0.4.0 at time of writing, is the official imitation-learning simulation playground integrated into LeRobot's EnvHub. Its compatibility table lists three working combinations; the newest pairs Isaac Sim 5.1 with Isaac Lab v2.3.0, CUDA 12.8, PyTorch 2.7.0 and Python 3.11, and the docs recommend Isaac Sim 5.0 or newer for 50-series cards.

git clone https://github.com/LightwheelAI/leisaac.git --recursive
conda create -n leisaac python=3.11 && conda activate leisaac
conda install -c "nvidia/label/cuda-12.8.1" cuda-toolkit
pip install -U torch==2.7.0 torchvision==0.22.0 \
  --index-url https://download.pytorch.org/whl/cu128
pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com
sudo apt install cmake build-essential
cd leisaac/dependencies/IsaacLab && ./isaaclab.sh --install && cd ../..
pip install -e source/leisaac
pip install -e "source/leisaac[lerobot]"
pip install numpy==1.26.0

With that in place, the leader arm on /dev/ttyACM0 drives the simulated follower and records straight to HDF5. The leader-follower loop is the same one you already know from real recording, only the follower is a rigid body in PhysX.

python scripts/environments/teleoperation/teleop_se3_agent.py \
    --task=LeIsaac-SO101-PickOrange-v0 \
    --teleop_device=so101leader \
    --port=/dev/ttyACM0 \
    --num_envs=1 \
    --device=cuda \
    --enable_cameras \
    --record \
    --dataset_file=./datasets/dataset.hdf5

Most of those ids also exist as a -Direct-v0 variant, and python scripts/environments/list_envs.py prints the current list. You can also skip the HDF5 detour entirely and write LeRobot format during teleoperation by adding three flags. Two caveats come from the documentation itself: the recorder automatically skips the first 5 frames of each episode to avoid instability from initial states, and it may cause slight delays in teleoperation, which is exactly the sort of thing that quietly changes the character of your demonstrations. It also only flushes episodes the task marked successful.

python scripts/environments/teleoperation/teleop_se3_agent.py \
    --task=LeIsaac-SO101-PickOrange-v0 \
    --teleop_device=so101leader \
    --port=/dev/ttyACM0 \
    --num_envs=1 --device=cuda --enable_cameras --record \
    --use_lerobot_recorder \
    --lerobot_dataset_repo_id=<your-user>/<dataset-name> \
    --lerobot_dataset_fps=30

The multiplication step then runs on those recordings. LeIsaac wraps Isaac Lab Mimic in four commands, because Mimic generalises trajectories from end-effector and object poses: convert the joint-space actions to IK-based actions, annotate, generate, then convert back to joint space.

python scripts/mimic/eef_action_process.py \
  --input_file ./datasets/mimic-lift-cube-example.hdf5 \
  --output_file ./datasets/processed_mimic-lift-cube-example.hdf5 \
  --to_ik --headless

python scripts/mimic/annotate_demos.py --device cuda \
  --task LeIsaac-SO101-LiftCube-Mimic-v0 \
  --input_file ./datasets/processed_mimic-lift-cube-example.hdf5 \
  --output_file ./datasets/annotated_mimic-lift-cube-example.hdf5 \
  --enable_cameras

python scripts/mimic/generate_dataset.py --device cuda \
  --num_envs 1 --generation_num_trials 10 \
  --input_file ./datasets/annotated_mimic-lift-cube-example.hdf5 \
  --output_file ./datasets/generated_mimic-lift-cube-example.hdf5 \
  --enable_cameras

python scripts/mimic/eef_action_process.py \
  --input_file ./datasets/generated_mimic-lift-cube-example.hdf5 \
  --output_file ./datasets/final_generated_mimic-lift-cube-example.hdf5 \
  --to_joint --headless

Then convert to LeRobot. This is the step where the platform's format rule bites, and LeIsaac happens to ship exactly the two converters you need: isaaclab2lerobot.py writes LeRobot v2, which is what the GR00T loaders take, and isaaclab2lerobotv3.py writes v3 for Pi0.5, SmolVLA and ACT. The two scripts take identical arguments but pin different lerobot versions, and only successful episodes are converted.

pip install lerobot==0.3.3
pip install numpy==1.26.0

python scripts/convert/isaaclab2lerobot.py \
    --task_name=LeIsaac-SO101-PickOrange-v0 \
    --repo_id=<your-user>/so101_pick_orange_sim \
    --hdf5_root=./datasets \
    --hdf5_files=dataset.hdf5

Two routes to a trained policy

Video world models: the newest layer, and the least measured

The idea behind DreamGen is that a video generative model, adapted to the target robot embodiment, can imagine plausible episodes in scenes you never visited. The pipeline has four stages: fine-tune the video world model, generate photorealistic synthetic robot videos, recover pseudo-action sequences with a latent action model or an inverse dynamics model, then train the robot policy on the result. NVIDIA's GR00T-dreams repository implements exactly that.

The headline result is real and worth taking seriously: teleoperation data from only a single pick-and-place task in one environment produced 22 new behaviours on a humanoid, in both seen and unseen environments. The caveat is equally real and sits in the third stage.

Co-training is the recipe that actually shows gains

If you take one operational lesson from the literature, take this one. The sim-and-real co-training study (Maddukuri and colleagues, 2025) set out to find a simple recipe for using simulation data to solve vision-based robotic manipulation tasks, across two domains, a robot arm and a humanoid, and its conclusion is that you train on a mixture. Simulation data enhanced real-world task performance by an average of 38 percent, and the paper is explicit that this held even with notable differences between the simulation and the real-world data.

That last clause matters more than the 38 percent. It means the simulation does not have to be a perfect digital twin to be useful, provided the real data is in the mixture to anchor it. Sim-only transfer is the expensive path: the same paper states that training a policy solely in simulation and transferring it to the real world often demands substantial human effort to bridge the reality gap. Co-training skips most of that effort by never asking the policy to close the gap on its own.

Practically, on this platform, co-training means one thing: put both sets of episodes in the same LeRobot dataset with consistent camera keys, consistent joint order and consistent units, then run a normal fine-tune. There is no mixture-weight knob in the training form. If you want a 3:1 sim-to-real ratio, you express that by how many episodes of each you put in the dataset.

The cheapest win from simulation is not training data

It is evaluation. Running dozens of real trials per task to compare two checkpoints is a day of arm time, and the arm drifts between trials. SIMPLER (Li and colleagues, 2024) built simulated environments whose purpose is scoring real policies rather than training them, and then measured how well the sim ranking predicts the real one. A single SIMPLER environment renders at 3,500 simulation steps per second on a consumer RTX 4090 at 640 by 512 resolution, which under a 500 Hz simulation frequency is a 7x speedup over real evaluation.

Those are averages over three Google Robot task groups for six common open-source checkpoints: three RT-1 checkpoints at different training stages, RT-1-X, RT-2-X and Octo-Base. The real side is not a uniform trial count, which is worth knowing before you quote it: 75 trials for pick coke can, 60 for move near, 54 for the open and close drawer tasks and 27 for the longer drawer-and-apple task. The comparison against validation MSE is the useful part. Model selection by validation loss ranks these checkpoints badly, and a Pearson r of 0.924 under visual matching means that if a checkpoint scores better in SIMPLER it very probably scores better on the bench. That is a repeatable overnight scoreboard, and it does not require you to believe anything about sim-to-real training transfer.

If you want the wider context on what these benchmark numbers do and do not tell you about a vision-language-action model, we wrote that up separately in the VLA overview.

Where this platform does not help you

Being clear about the boundary saves everybody time. AY-Robots is a recording, training and serving platform. It has no simulator in it.

• No Isaac Lab, no MimicGen, no world model, no scene authoring. If you want generated data, you generate it elsewhere and bring the result.
• The trainers consume LeRobot datasets and nothing else. A simulator export has to be converted before it is an input, and it has to be the right version: v2.0 or v2.1 for GR00T N1.5 and N1.7, v3.0 for Pi0.5, SmolVLA and ACT.
• GR00T's fine-tuning entry point is a tyro CLI that exposes no seed, so GR00T runs are not bit-for-bit reproducible. If you are running a careful sim-versus-real ablation, that is a real limitation. lerobot's own default seed is 1000, and the ACT, SmolVLA and Pi0.5 forms do expose a seed field.
• Gradient accumulation is only actually applied for the two GR00T trainers. For Pi0.5 and SmolVLA the field exists in the form but lerobot 0.5.1 has no such flag, so it does nothing.
• Inference has to sit next to the servos for fast tasks. The control loop is 20 to 485 ms per action step depending on the model, and adding public-internet round trips turns a working policy into a hesitant one. Remote inference is viable for slow pick-and-place, not for fast reactive motion.

A budget you can defend

Put the two paths side by side with the numbers each one actually publishes, and the decision usually makes itself for a single-task project on a low-cost arm.

For a first policy on an SO-100, record. The recording walkthrough and first training run get you to a served checkpoint for the price of a coffee, and you will have the real half of any future co-training mixture. Reach for the simulator when you have a working baseline and a specific generalisation failure you can name, such as a policy that only works in one setup.

A recipe that respects the evidence

1. 1
   Record the real baseline first

   30 episodes for SmolVLA, 50 for ACT, GR00T N1.7 and Pi0.5. Train once. Whatever that policy fails at is your specification for the synthetic data.

2. 2
   Name the generalisation failure

   Object pose? Lighting? Table height? Distractors? A different phrasing of the instruction? Synthetic data is good at exactly one of these at a time, and useless if you cannot say which.

3. 3
   Pick the cheapest family that covers it

   Pose and layout variation: trajectory multiplication. Lighting and texture: image augmentation first, world model second. Whole new scenes: physics rollouts, and accept the modelling cost.

4. 4
   Generate, then throw away aggressively

   Generation attempts fail, and Isaac Lab's own candidate success rate ranges from 70 percent down to under 1 percent depending on the task. Keep only successful, task-completing trajectories, and eyeball a sample as video before you trust the batch.

   bash

   python scripts/mimic/generate_dataset.py --device cuda \
       --num_envs 8 --generation_num_trials 500 \
       --input_file ./datasets/annotated.hdf5 \
       --output_file ./datasets/generated.hdf5 --enable_cameras

5. 5
   Co-train, do not replace

   Merge the generated episodes with the real ones into a single LeRobot dataset with identical camera keys and joint ordering. The 38 percent figure is a co-training figure.

6. 6
   Evaluate on the real arm, and only there

   Simulated evaluation is a good ranking signal (Pearson r 0.924 in SIMPLER's visual-matching setup) but it is not the acceptance test. Run the checkpoint on the bench before you believe it.

If you would rather start from a checklist for the real recordings themselves, the data collection guide covers camera placement, action chunking implications and the failure modes that make an imitation learning dataset unusable. Details on the format itself live in the dataset docs, and the hyperparameter side in the training docs.