RoboColosseum Launches Real-World Robot AI Benchmark With Unitree Support

RoboColosseum is a new open initiative for testing robot AI on physical hardware, announcing access to more than 100 robots, over 140 real-world scenes and 10,000 free evaluation hours for academic researchers. Unitree has publicly backed the project, whose humanoid category uses its G1 platform.

Key Takeaways

Show all

• RoboColosseum is offering shared robot infrastructure and 10,000 free evaluation hours, with evaluation recordings to be released for inspection.
• Unitree supports the initiative and identifies G1 as its primary humanoid platform.
• Initial research results show a substantial gap between partial progress and completed humanoid tasks.

The launch announcement identifies Hugging Face, BitRobot, the Singapore Embodied AI Centre and researcher Jiafei Duan as participants. Alongside G1 humanoids, the advertised fleet includes SO-101 arms, YAM systems and Franka robots.

For developers, the offer addresses a practical obstacle: testing robot software requires access to machines, environments and people to run repeated attempts. In the launch video, BitRobot's Michael Cho explains the service: “even if you don't have a real robot, just send us your policy and we'll deploy it on real hardware.”

The video describes Singapore as the starting location. Its central argument is that comparisons across laboratories are difficult when each uses different evaluation methods. RoboColosseum aims to test competing models under common protocols in varied real-world settings.

In its supporting post, Unitree welcomed the use of G1s in the humanoid benchmark. The statement does not specify a financial contribution or hardware donation.

What the first tests found

An accompanying preprint reports 5,699 blinded, pairwise comparisons at 124 sites. That completed study is distinct from the launch's advertised access to more than 140 scenes.

The Open Track tests models without task-specific adaptation. A separate Fine-tune Track gives each policy the same 100 demonstrations per task, then tests it over 25 attempts. The study covers five vision-language-action policies and three frontier AI agents, with participation varying by platform and track.

The humanoid results are sobering. None of the five policies completed any of the five G1 tasks after fine-tuning. Those tasks included carrying a box, folding a towel and loading clothes into a washing machine. Average partial-progress scores ranged from 13.3% to 23.2%.

The live G1 fine-tuning leaderboard lists 625 trials across the five policies, with no fully completed tasks. Its nonzero progress scores measure partial completion, rather than successful attempts.

Speaking to Humanoids Daily, Jiafei Duan said the open-source models were not yet suitable for zero-shot operation in the humanoid Open Track, and that most struggled with the humanoid tasks even after training on 100 demonstrations per task. His response did not identify a specific failure mode shared by the models.

Standardized testing and public results

Asked how the initiative separates model limitations from differences in robot setup, Duan said RoboColosseum uses standardized hardware configurations and a trained evaluation crew. He said its protocols are open and that the team will release all evaluation rollouts for inspection, allowing others to examine individual attempts. The launch video identifies Hugging Face as the publication platform and explicitly includes failed attempts. Duan's response describes the release as forthcoming; the announcement alone does not establish that every recording is already available.

For now, the initiative is seeking academic teams willing to make their models' performance public on its leaderboard. Selection will be handled by the founding members, Duan told Humanoids Daily. He did not specify how the free evaluation hours would be allocated.

Teams can apply for evaluation access, indicating their preferred track, robot platform and project. The requirement Duan described concerns public performance results; it does not establish that applicants must release their model weights or code.

For humanoid researchers and builders, the useful question is whether a demonstrated skill survives repeated attempts under different conditions. RoboColosseum offers a way to investigate that question—and to make unsuccessful attempts part of the public evidence alongside successful demonstrations.