Can a Simulated Robot Score Predict Real Performance? X Square Releases X2Real

X Square Robot has introduced X2Real, a benchmark aimed at answering a practical question for robot developers: how much does a good simulation score tell you about performance on physical hardware?

In its October 19 announcement, the company described tests spanning visual and language reasoning, precise manipulation, coordinated arm movements and tool use. X2Real builds on NVIDIA’s Isaac Lab-Arena simulation infrastructure.

Key Takeaways

Show all

• X2Real covers 44 tasks across 10 capability dimensions, with a public evaluation runtime available on GitHub.
• The paper reports a 0.84 correlation for task progress and 0.74 for success rates in an eight-task simulation-versus-hardware comparison.
• The current public release provides in-distribution evaluation; out-of-distribution task definitions and training tools are excluded.

Checking simulation against hardware

The technical report compares WALL-OSS-0.5, trained without simulation data, on eight real and simulated tasks. Across those tasks, progress scores correlated at 0.84, while success rates correlated at 0.74. Neither figure means 84% of tasks transferred successfully.

Precision remained a weakness: placing glasses on a shelf succeeded in 70% of physical trials but none of the simulated trials. That matters because a simulator can underestimate a capable policy as well as overestimate one.

The authors also identify limits in task coverage: rigid objects dominate, while deformable materials, fluids, dexterous hands and tactile sensing remain outside the current scope. The reported comparison supports a bounded evaluation tool, rather than a replacement for testing physical robots.

What developers can use

The GitHub release lists 73 runnable configurations across ArtiXon Arm-6A, Quanta X1, ARX R5 and Franka platforms. These are configurations of the benchmark’s tasks, rather than 73 distinct tasks. Simulation assets and demonstration data are distributed separately.

Its Mana environment handles task execution and scoring, while a separate interface connects the robot-control model being tested. This separation lets developers compare different models without embedding each model’s software dependencies into the evaluator.

There is an important release boundary: the published scope excludes out-of-distribution definitions, training and several automatic data-generation tools. The broader research system and the downloadable package therefore should not be treated as identical.

For developers, the immediate opportunity is a reproducible test environment with an explicit attempt to check its scores against hardware. Whether that relationship holds across more models, robots and tasks will determine how useful it becomes beyond the reported experiments.