LIBERO, SimplerEnv and Robot Benchmarks Explained

Open almost any vision-language-action paper from the last two years and you meet the same table: four columns labelled Spatial, Object, Goal and Long, one row per model, a number between 70 and 99 in every cell. That is LIBERO. In the NVIDIA Isaac-GR00T examples a second table sits next to it with rows named google_robot_pick_coke_can and widowx_spoon_on_towel. That is SimplerEnv. The two tables look like they belong on the same axis, and neither of them says anything directly about what an SO-100 will do on your desk.

This page walks through what LIBERO, SimplerEnv and Meta-World actually measure, which knobs move a published score without touching the policy at all, and how to check a benchmark claim instead of taking it on faith. Every number below was read out of the paper, the repository or the evaluation script it comes from, and dated, because these benchmarks change while keeping the same name. If you want the shorter version of the model landscape first, the VLA overview covers the models themselves.

What a benchmark number is made of

A manipulation success rate is not a measurement the way a mass or an inference latency is. It is the output of a pipeline: a task suite, a demonstration dataset, a training recipe, an evaluation harness, an episode budget, a seed, and a rule for choosing which checkpoint gets evaluated at all. Change any stage and the number moves. Most stages are documented somewhere in a repository. Almost none of them appear in the results table.

None of that is dishonest. Two groups can each report LIBERO-Spatial correctly and still be measuring different things. The useful reflex is not distrust, it is reading one layer deeper before you put two numbers in the same row.

LIBERO: the number in almost every VLA paper

LIBERO was published as Benchmarking Knowledge Transfer for Lifelong Robot Learning by Liu, Zhu, Gao, Feng, Liu, Zhu and Stone, first posted 5 June 2023 with a v2 on 14 October 2023. It is built on robosuite and MuJoCo, uses a Panda arm (the repository registers MountedPanda and OnTheGroundPanda), and specifies each task goal as a conjunction of PDDL predicates such as On(A, B). The simulation terminates when every predicate is true, and the reward is sparse: plus one on completion, nothing otherwise. That last detail is why LIBERO is used almost entirely for imitation learning rather than reinforcement learning.

LIBERO-90 and LIBERO-Long together form LIBERO-100, which is why the paper says four suites and 130 tasks. The original protocol is not what a modern VLA paper runs: the paper trains a small policy for 50 epochs on 50 demonstrations per task, evaluates every 5 epochs over 20 rollouts capped at 600 steps, uses Adam with batch size 32 and a cosine schedule from 1e-4 to 1e-5, repeats everything over seeds 100, 200 and 300, and reports forward transfer (FWT), negative backward transfer (NBT) and area under the success curve (AUC). Modern papers throw away the lifelong-learning framing entirely and report a single per-suite success rate after fine-tuning one model per suite.

import os
from libero.libero import benchmark, get_libero_path
from libero.libero.envs import OffScreenRenderEnv

benchmark_dict = benchmark.get_benchmark_dict()
task_suite = benchmark_dict["libero_10"]()   # or libero_spatial / libero_object / libero_goal
task = task_suite.get_task(0)
print(task.language)   # the instruction the policy is conditioned on

bddl = os.path.join(get_libero_path("bddl_files"), task.problem_folder, task.bddl_file)
env = OffScreenRenderEnv(
    bddl_file_name=bddl,
    camera_heights=128,   # the default in the LIBERO README example
    camera_widths=128,
)
env.seed(0)
env.reset()

# fixed initial states, so every method starts from the same scene
init_states = task_suite.get_task_init_states(0)
env.set_init_state(init_states[0])

obs, reward, done, info = env.step([0.0] * 7)   # 7-dim action, last element is the gripper

The word LIBERO covers at least two different datasets

The OpenVLA paper (arXiv 2406.09246, LIBERO added in the v2 revision of September 2024) did not train on the demonstrations that ship with LIBERO. It regenerated them by replaying every demonstration in the environment. The header comment of regenerate_libero_dataset.py lists exactly what changed.

• Images are saved at 256x256 instead of 128x128.
• Transitions with no-op actions are removed. A no-op is defined as all non-gripper dimensions near zero and the gripper command unchanged from the previous step.
• Unsuccessful demonstrations are dropped entirely.
• In the HDF5 to RLDS conversion the images are rotated 180 degrees, because the environments returned them upside down on that platform.

# regenerate one suite into the "no_noops" form used by most published numbers
python experiments/robot/libero/regenerate_libero_dataset.py \
    --libero_task_suite libero_spatial \
    --libero_raw_data_dir ./LIBERO/libero/datasets/libero_spatial \
    --libero_target_dir ./LIBERO/libero/datasets/libero_spatial_no_noops

# or take the pre-converted RLDS version (about 10 GB for all four suites)
git clone git@hf.co:datasets/openvla/modified_libero_rlds

That regenerated form is now the de facto standard. The openpi LIBERO example points at it, and the Isaac-GR00T LIBERO example downloads IPEC-COMMUNITY/libero_spatial_no_noops_1.0.0_lerobot and friends, which are the same data converted to LeRobot dataset format. So a 2023 number and a 2025 number under the same suite name were produced from different pixels, at different resolution, with a different number of episodes and a different image orientation. The 2025 number is on the easier data, and the OpenVLA README says so plainly: the modification was made to improve the performance of all methods.

The harness moves the score as much as the policy does

Three widely used LIBERO harnesses disagree on almost every evaluation setting. None of them is wrong. They simply produce numbers that should not be compared to the second decimal place.

The episode budget row is the one that matters most. A VLA that hesitates, retries a grasp and eventually succeeds scores as a failure at 520 steps and a success at 720. The openpi script carries a comment worth quoting for a different reason: the seed appears to affect object positions even when a fixed initial state is set. That is the kind of detail that decides whether a reproduction attempt matches to within a point or misses by five.

Two things stand out. First, the field went from 76.5 in the OpenVLA v2 table of September 2024 to roughly 97 by the OpenVLA-OFT release of February 2025, which means LIBERO is close to saturated: the remaining headroom on three of the four suites is one or two percentage points, well inside the noise band you get from 200 to 500 rollouts. Second, the GR00T table reports Spatial as 195/200 (97.65%) in the same cell, and 195 out of 200 is 97.5 percent, not 97.65. It is a small inconsistency in a vendor README, and it is exactly why you check a number at its source before you build a comparison on it. If you want the per-model view rather than the per-benchmark view, the policy comparison page and the head-to-head GR00T N1.7 against Pi0.5 page use the same figures from the same vendor sources.

SimplerEnv: the benchmark that measures the benchmark

Evaluating Real-World Robot Manipulation Policies in Simulation (Li, Hsu, Gu and colleagues, 9 May 2024) starts from a different premise. Its goal is not to be a hard task set. Its goal is to be a simulated evaluation whose ranking of policies agrees with the ranking a real robot would produce. The paper reports a comparison across roughly 1500 evaluation episodes on two embodiments, a Google Robot and a WidowX from the BridgeData setup.

The authors name two gaps and attack them separately. The control gap is closed by offline system identification against real trajectories. The visual gap is closed by Visual Matching: green-screening real backgrounds behind the simulated scene and baking real textures onto the assets. An alternative, Variant Aggregation, instead randomises backgrounds, lighting, distractors and table textures and averages across the variants. The runtime is SAPIEN with the CPU-based ManiSkill2; the Bridge environments were also ported to ManiSkill3, which the SimplerEnv repository reports as 10 to 15 times faster. If you are choosing a simulator to build in rather than reading someone else's score, Isaac Lab is the other end of that decision.

Those are the averages from Table I of the paper, over three tasks (Pick Coke Can, Move Near, Drawer) and six open-source checkpoints: three RT-1 checkpoints, RT-1-X, RT-2-X and Octo-Base. MMRV is Mean Maximum Rank Violation, a metric the paper introduces because Pearson correlation alone rewards a linear fit and can be dragged around by noise when two policies perform almost identically in the real world. MMRV asks a narrower question: when the simulated ranking disagrees with the real ranking, how large was the real-world gap it got backwards.

SimplerEnv also shows why averages are dangerous. The Isaac-GR00T SimplerEnv example publishes per-task results for GR00T N1.6 and N1.7 on the Bridge suite. The average rose from 56.6 to 62.3 percent. Inside that average, widowx_stack_cube went from 5 to 48 percent and widowx_open_drawer from 95 to 100, while widowx_put_eggplant_in_basket fell from 89 to 53 and widowx_put_eggplant_in_sink fell from 33 to 2. If your task looks like the eggplant, the headline improvement is a regression.

Meta-World, and what six years does to a benchmark

Meta-World (Yu, Quillen, He and colleagues, first posted 24 October 2019, v2 on 14 June 2021) is older and aimed at multi-task and meta reinforcement learning rather than imitation. It is 50 simulated manipulation tasks on a Sawyer arm, with a deliberately small action space: Box(-1.0, 1.0, (4,), float32), three Cartesian end-effector displacements and one gripper command. The benchmark splits are MT1, MT10 and MT50 for multi-task learning and ML1, ML10 and ML45 for meta learning, where ML10 gives 10 training tasks and 5 held-out test tasks and ML45 gives 45 and 5.

# pip install metaworld   (Farama build, Python 3.10 to 3.13, Gymnasium API)
import gymnasium as gym
import metaworld

env = gym.make("Meta-World/MT1", env_name="reach-v3")
observation, info = env.reset()
for _ in range(500):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
env.close()

That version suffix is the whole story. In May 2025 a group including several of the original authors published Meta-World+: An Improved, Standardized, RL Benchmark, whose abstract states that since Meta-World's introduction there have been numerous undocumented changes which inhibit a fair comparison of algorithms. A benchmark that existed for six years drifted far enough that the community needed a paper to work out which published numbers were comparable to which.

How many trials before a number means anything

A success rate over N rollouts is a binomial proportion, and binomial proportions from small N are much less precise than they look. An NVIDIA developer article published on 11 July 2026 works the arithmetic with the exact Clopper-Pearson interval and states plainly that most published benchmarks do not run enough rollouts to reach statistical significance when comparing two policies.

Narrowing that interval from 15.4 points wide to about 4 points wide costs roughly 15 times more rollouts. Put that next to the LIBERO harnesses: 500 rollouts per suite is respectable, 200 is thinner, and a real-robot evaluation of 20 trials, which is what most hardware papers can afford, carries an interval wide enough to swallow the difference between two models entirely. Snyder and colleagues make the same point from the other direction in Is Your Imitation Learning Policy Better than Mine? (14 March 2025): policy comparison is fundamentally constrained by feasible sample sizes on the order of 10 or 50, and their sequential stopping rule cuts the required trials by up to 32 percent rather than pretending the problem away.

What none of it tells you about your own arm

The gap is not subtle. LIBERO's Panda has a wrist that can reach a pose from many directions; an SO-100 has five joints, so whole classes of approach angle simply do not exist for it. Read the degrees of freedom entry once and the difference stops being abstract. A model that scores 98 percent on LIBERO-Object has demonstrated that a 3 B parameter network can fit a fixed simulated pick-and-place; it has not demonstrated that it will find your particular cube under your particular lamp with your particular gripper.

For deployment the deciding numbers are different ones. On this platform ACT runs an action step in 20 ms and GR00T N1.7 in 152 ms, while Pi0.5 needs 485 ms. Minimum useful dataset size is 30 recorded episodes for SmolVLA and 50 for the others. Neither of those appears in any LIBERO table, and both will decide whether your task works long before model quality does.

Two ways to get an evaluation number you trust

Checking a benchmark claim in ten minutes

1. 1
   Pin the suite and the trial count

   Find the exact suite name and how many rollouts per task produced the figure. If the source does not say, treat the number as an anecdote rather than a measurement.

2. 2
   Find out which dataset version it was trained on

   Search the repository for the string that gives it away. A hit on no_noops means the regenerated 256 pixel data, not the original release.

   bash

   git clone --depth 1 https://github.com/NVIDIA/Isaac-GR00T
   grep -rn "no_noops\|modified_libero\|libero_spatial" Isaac-GR00T/examples | head

3. 3
   Find the episode budget

   The step cap is usually a hard-coded ladder in the evaluation script rather than a documented flag.

   bash

   grep -n "max_steps\|max-episode-steps\|num_trials_per_task\|replan_steps" \
     experiments/robot/libero/run_libero_eval.py examples/libero/main.py

4. 4
   Put a confidence interval on it

   Clopper-Pearson takes four lines. Do this before you claim one model beats another by two points.

   python

   from scipy.stats import beta
   
   def clopper_pearson(k, n, alpha=0.05):
       lo = beta.ppf(alpha / 2, k, n - k + 1) if k > 0 else 0.0
       hi = beta.ppf(1 - alpha / 2, k + 1, n - k) if k < n else 1.0
       return lo, hi
   
   for k, n in [(63, 70), (927, 1030), (195, 200), (424, 500)]:
       lo, hi = clopper_pearson(k, n)
       print(f"{k}/{n} = {k/n:6.1%}   95% CI [{lo:.1%}, {hi:.1%}]")

5. 5
   Ask whether the embodiment resembles yours

   A Panda score transfers to your SO-100 as a rough prior on model family, nothing more. If the model has no matching embodiment tag for your arm at all, the benchmark number is not the constraint you will hit first.

Where robot evaluation is heading

The field knows this. RoboArena (Atreya, Pertsch and colleagues, 22 June 2025, revised 29 November 2025) skips standardised task sets entirely: evaluators at seven academic institutions run double-blind pairwise comparisons of policies on their own DROID setups and on tasks of their own choosing, and the preferences are aggregated into a ranking. The reported scale is over 600 pairwise real-robot evaluation episodes across seven generalist policies. It is a different bet from LIBERO: give up on controlling the environment, and buy back validity by evaluating in many uncontrolled ones.

Until something like that becomes standard, treat LIBERO and SimplerEnv as what they are. They are useful screens for excluding a model family, they are reproducible in a way real hardware is not, and they are close to saturated at the top. They are not a forecast of your success rate. That number you have to measure yourself, on your table, with the episodes you recorded, and with enough rollouts that the interval is narrower than the difference you care about.