Sim-to-Real for a Low-Cost Arm: What Simulation Actually Gives You

An SO-100 costs about 110 to 150 EUR in parts, and every hour you spend driving it by hand is an hour spent on nothing else. A simulated arm costs nothing per attempt, never strips a gear, and runs a thousand copies of itself on one GPU. The question is what you get back when you carry a simulated policy over to a real arm.

The honest answer, from published numbers rather than launch posts: a great deal for some task shapes, close to nothing for others. Below, which simulators contain an SO-100, what randomization costs, how large the gap measures, and where it will not help an imitation learning project.

How big is the reality gap, in numbers

Several groups have published paired evaluations: the same trained policy scored in simulation and then on the physical robot under the same task definition. Those pairs are the only honest measurement of the gap.

Reaching transfers essentially perfectly, stacking does not. The gap grows with contact precision, and fastest where a low-cost arm is weakest: holding a pose while the gripper closes. The Dactyl rows give the scale of the problem on far better hardware.

The four gaps, and which one bites a hobby arm

NVIDIA's SO-101 course splits the gap into sensing, actuation, physics and modeling. On an SO-100 the four are not equally dangerous. Three you can attack with randomization. The fourth is hardware.

NVIDIA is blunt about the actuation term: these are hobby servos, they introduce backlash, and it accumulates along the kinematic chain. The course's answer is SAGE, a Tongji University, Peking University and NVIDIA project that replays the same commanded motion in both domains and compares position, velocity and torque per joint. Bridging what it measures needs a separate learned actuator model, GapONet, and the course lists compensation inside the deployment loop as future work. Measurement comes first because the motor model is what randomization cannot fix.

Which simulator actually contains an SO-100

This is where people lose a weekend, because the answer differs between the SO-100 and the SO-101, and between an arm description and a usable task. Checked on 23 August 2026.

Domain randomization, with the real default values

Domain randomization comes from Tobin et al. in 2017: rather than make one simulator accurate, make many varied enough that reality looks like another sample. That paper trained a real-world object detector accurate to 1.5 cm from simulated images with non-realistic random textures. Peng et al. extended it to dynamics the same year.

The abstract version is not useful. Here is what the ManiSkill3 SO-100 grasping task randomizes, from the environment source.

# mani_skill/envs/tasks/digital_twins/so100_arm/grasp_cube.py
@dataclass
class SO100GraspCubeDomainRandomizationConfig:
    # task agnostic
    initial_qpos_noise_scale: float = 0.02
    robot_color: Optional[Union[str, Sequence[float]]] = None   # set "random" to vary
    randomize_lighting: bool = True
    max_camera_offset: Sequence[float] = (0.025, 0.025, 0.025)  # metres, per axis
    camera_target_noise: float = 1e-3
    camera_view_rot_noise: float = 5e-3
    camera_fov_noise: float = np.deg2rad(2)                     # 2 degrees

    # task specific, applied while the scene loads
    cube_half_size_range: Sequence[float] = (0.022 / 2, 0.028 / 2)
    cube_friction_mean: float = 0.3
    cube_friction_std: float = 0.05
    cube_friction_bounds: Sequence[float] = (0.1, 0.5)
    randomize_cube_color: bool = True

@register_env("SO100GraspCube-v1", max_episode_steps=64)
class SO100GraspCubeEnv(BaseDigitalTwinEnv):
    def _default_sim_config(self):
        return SimConfig(sim_freq=100, control_freq=20)

Three numbers there are worth stealing. The camera may move 2.5 cm on any axis and its field of view may wander 2 degrees, which is the tolerance you are granted for aligning a real camera by hand. The cube is not one cube: half size from 1.1 to 1.4 cm, friction centred on 0.3 and clipped to 0.1 to 0.5. Physics runs at 100 Hz, control at 20 Hz.

NVIDIA's course randomizes wider because it generates demonstrations rather than running RL: exposure -4.0 to 3.0, colour temperature 2500 to 9500 K, HDRI backgrounds, camera offsets of 2 cm in x and y and 1 cm in z, camera rotation of 0.05 radians per axis, plus object mass, friction and restitution, joint damping and limits, and actuator delays.

The manual path: reinforcement learning in ManiSkill, deployed zero-shot

The shortest route ending with an SO-100 picking up a cube from a policy trained without a single real demonstration. It comes from the lerobot-sim2real tutorial by Xander Hinrichsen and Stone Tao, built on ManiSkill3. You need an NVIDIA GPU with at least 8 GB, working Vulkan, one camera and an assembled arm.

1. 1
   Install the simulator, then pin LeRobot

   ManiSkill and SAPIEN depend on NVIDIA drivers and Vulkan, and the demo command tells you whether yours work. The tutorial pins LeRobot to commit a989c795587d122299275c65a38ffdd0a804b8dc because import paths moved upstream.

   bash

   conda create -n ms3-lerobot "python==3.11"
   git clone https://github.com/StoneT2000/lerobot-sim2real.git
   cd lerobot-sim2real
   pip install -e .
   pip install torch
   python -m mani_skill.examples.demo_random_action

2. 2
   Place the camera in simulation first, not on the desk

   The robot spawns at the origin with the table top at z = 0, so camera position in env_config.json is metres from the arm base. Record the reset distribution to see where cubes actually spawn.

   bash

   python lerobot_sim2real/scripts/record_reset_distribution.py \
     --env-id="SO100GraspCube-v1" \
     --env-kwargs-json-path=env_config.json

3. 3
   Nudge the real camera until the overlay lines up

   This overlays the simulated render on the live feed and tunes field of view with the arrow keys. The tutorial calls it the hardest step and reports that worse alignment transfers worse. Randomization covers a margin of error, not a mistake.

   bash

   python lerobot_sim2real/scripts/camera_alignment.py \
     --env-id="SO100GraspCube-v1" \
     --env-kwargs-json-path=env_config.json

4. 4
   Photograph the empty table for greenscreening

   Unmount the arm, shoot the background, point greenscreen_overlay_path at the result. The simulated foreground is composited onto your real background, removing most of the visual gap.

   bash

   python lerobot_sim2real/scripts/capture_background_image.py \
     --env-id="SO100GraspCube-v1" \
     --env-kwargs-json-path=env_config.json \
     --out=greenscreen.png

5. 5
   Train PPO on rendered images

   The tuned script uses 8 to 10 GB of GPU memory at 1024 parallel environments. The flag says 100 million timesteps, but the authors report 25 to 40 million is enough, about an hour on a 4090, and that training longer sometimes produces worse policies. For your own task, see the custom task tutorial.

   bash

   seed=3
   python lerobot_sim2real/scripts/train_ppo_rgb.py \
     --env-id="SO100GraspCube-v1" --env-kwargs-json-path=env_config.json \
     --ppo.seed=${seed} \
     --ppo.num_envs=1024 --ppo.num-steps=16 \
     --ppo.update_epochs=8 --ppo.num_minibatches=32 \
     --ppo.total_timesteps=100_000_000 --ppo.gamma=0.9 \
     --ppo.num_eval_envs=16 --ppo.num-eval-steps=64 \
     --ppo.no-partial-reset \
     --ppo.exp-name="ppo-SO100GraspCube-v1-rgb-${seed}"

6. 6
   Run the checkpoint on the arm

   Use a cube around 2.5 cm, the middle of the trained size range, inside the spawn region from step 2. The tutorial recommends control frequency 15 on SO-100 hardware.

   bash

   python lerobot_sim2real/scripts/eval_ppo_rgb.py \
     --env_id="SO100GraspCube-v1" \
     --env-kwargs-json-path=env_config.json \
     --checkpoint=path/to/ckpt.pt \
     --no-continuous-eval --control-freq=15

Two routes to a policy that works on your desk

What simulation will not do for this project

The gap between what simulation is good at and what an SO-100 project needs is wider than the sim-to-real gap itself. Reinforcement learning wants a reward function and millions of resets. Fine-tuning a VLA wants demonstrations of a task you can name in a sentence. Only the first is what a physics engine is built for.

Feeding simulated data to a real policy

The bridge between the two worlds is the dataset, not the policy. If your simulator writes episodes in LeRobot format, a trainer cannot tell them from recorded ones. LeIsaac does this for Isaac Lab: teleoperate in Isaac Sim, record HDF5, convert, fine-tune. NVIDIA's course then co-trains on a mixture.

The recommended mixture is deliberately lopsided: about 5 real episodes against 70 to 100 simulated ones. The real handful is not there for volume, it anchors the sensing and actuation terms the renderer got wrong. Their sim evaluation of a vial pick-and-place policy is quoted at 50 to 70 percent.

Does a simulated score predict a real one?

This is the underrated use of simulation, and it pays off even if you never train in it. The SIMPLER study built simulated replicas of two real robot setups and ran paired evaluations of six checkpoints in both. The question was not whether success rates match in absolute terms, but whether they rank policies in the same order, which is all model selection needs.

Averaged over three task families on the Google Robot setup. Sit with the top row: picking a checkpoint by validation loss, the habit carried over from supervised learning, correlates with real performance at r = 0.308. A matched simulated evaluation reaches 0.924. That is the quantitative version of the loss curve looking fine while the arm does nothing, which has its own failure page.

Two caveats the authors state themselves: the environments cover rigid objects only, and the greenscreening assumes a fixed camera and does not reproduce object shadows. Both apply to a desk with one webcam and a printed gripper.

That matters when you read a leaderboard. Most results on the arena come from simulated benchmarks. They signal relative capability, not what a checkpoint does on your desk under your lighting.

A decision rule that holds up

• Contact-light reaching or a coarse pick from a known spawn area, and you want reinforcement learning: simulate. This is what the tooling was built for.
• Precise contact, insertion, deformables, cable routing, or anything where the gripper must feel something: record real episodes.
• Comparing two checkpoints before spending arm time: a matched simulated evaluation ranks them better than validation loss.
• A VLA that follows a language instruction: record real demonstrations, then consider adding simulated ones. Start at train your first policy.
• One free evening, no NVIDIA GPU, an assembled arm: record 50 episodes. That is the minimum for four of the five policies and beats fighting Vulkan.
• Fast reactive task: none of this saves you from the control loop, 20 ms per action step for ACT and 485 ms for Pi0.5. See inference latency.

Simulation moves work, it does not delete it. On the RL route you trade demonstration hours for camera alignment, GPU time and reward shaping. If the task is geometric and repetitive, that trade is good. If it is the sort of thing you would show a person once, imitation learning from real episodes gets there faster, and how to collect high-quality VLA training data will do more than any randomization. On the simulator side, the Isaac Lab write-up covers the framework.