WorldOracle Tests Whether World Models Can Reliably Predict Real-World Executions

Key Takeaways

Show all

• WorldOracle is a benchmark for action-conditioned world models (ACWMs) that scores each model's predicted rollout against the paired real-world execution, with both starting from the same initial observations and following the same action sequence.
• Seven representative models were evaluated; arm motion was predicted relatively well, but object positions and movements often deviated from reality, and performance dropped over complete operation sequences.
• The platform opened to developers on October 16 and accepts both open-weight and closed-source models.

In one case showcased by WorldOracle, a robot's grasp missed, yet the world model predicted that the robot had successfully picked up the fruit.

Given the initial frame and action sequence of a real manipulation episode, the model generated a rollout that looked plausible on screen but differed from the actual execution. The success it depicted never happened in reality.

Researchers from Princeton University, the University of Texas at Austin, the University of Toronto, NVIDIA and AGIBOT have jointly introduced WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) by comparing their predictions with real-world executions that start from the same initial observations and follow the same action sequences.

The team evaluated seven representative models — among them DreamDojo, Cosmos 3, IRASim and GE-Sim 2.0 — and identified two prominent problems. First, while the motion paths of robot arms are predicted relatively well, the positions and movements of manipulated objects can still deviate from reality. Second, models that perform well over short horizons can degrade significantly over complete operation sequences.

On October 16, WorldOracle, an evaluation platform for action-conditioned world models, officially opened to developers. The research team has collected real manipulation episodes in advance, recording both video and actions. Developers use the initial frames and action sequences provided in the evaluation set to generate predicted videos and submit them to the platform, which scores each submission against the corresponding real execution record. The results show whether a model has accurately predicted the consequences of the actions, helping developers pinpoint weaknesses and verify improvements.

Project page

Paper

How Do Seven World Models Perform?

Arm motion is right, but objects don't always follow

The predicted motion of a robot arm reaching toward a piece of fruit and closing its gripper may closely match its actual motion, yet the model can still incorrectly predict whether the fruit is lifted and where it moves next. To assess the two separately, the research team compared the position and motion of the robot arm and of the manipulated object in each predicted video against the real-world recording, and scored them individually.

The results show that all five models evaluated on the human-operated AGIBOT G2 data scored higher on the arm than on the object. In other words, even when the arm in the generated video follows the given actions well, the position and movement of the object may still depart from reality. IRASim scored 78.8 on the robot arm but only 29.1 on the manipulated object, while Cosmos 3 scored 89.4 and 45.4, respectively. The remaining models showed gaps of varying size.

Accuracy declines as the rollout horizon grows

Time presents a second challenge. Completing a continuous task often requires multiple grasps, placements and adjustments, and the object state left by one interaction becomes the starting point for the next. Whether a model stays close to reality must therefore be tested over the full process.

WorldOracle evaluates both the first 30 seconds of each trajectory and the complete long sequence.

On the same set of AGIBOT G2 teleoperation data, the WorldOracle-SV scores of all five models fell. DreamDojo dropped from 50.56 to 27.08, and Cosmos 3 from 53.99 to 26.60, nearly halving in both cases. GE-Sim 2.0 fell from 73.86 to 63.33; it posted the highest score of the five models at both horizons and the smallest decline over the full sequence.

The relative ranking also shifted. IRASim scored below Ctrl-World, DreamDojo and Cosmos 3 over the first 30 seconds, but above all three over the full sequence.

What Can WorldOracle Measure?

Accurately predicting a failure earns a high score

WorldOracle starts from pre-collected records of real manipulation. A model is given the initial frame and the action sequence and generates a predicted video, which is compared with the real execution of the same actions.

What models must predict includes missed grasps, drops and spills. If an operation fails in reality, accurately predicting that failure counts as a correct prediction.

Failed executions therefore serve as an important class of test samples: they reveal whether a model follows the given actions and their consequences, or defaults to a successful version of the task.

The benchmark covers two robot embodiments, AGIBOT G2 and YAM, as well as first-person human-hand manipulation. Four tasks (fruit picking, towel hanging, water pouring and drawer pulling) involve interaction with rigid, deformable, fluid and articulated objects, respectively. Task coverage varies by embodiment, as defined in the evaluation set.

The human-hand setting is intended for world models that can accept hand-motion input, testing whether a model can predict subsequent object changes and interaction outcomes from a recorded sequence of hand poses. Models with egocentric world-modeling capabilities are evaluated zero-shot in this setting, without any additional hand-adaptation data provided by the platform.

Watch the task recordings

The data come from two sources. The first is human-operated data collection, including robot teleoperation trajectories and human hand-manipulation trajectories. The second is policy rollouts, in which an autonomous policy controls a real robot.

When a robot executes autonomously, it can produce pauses, jitter and motion corrections that differ from human operation. Including such data helps test whether a world model remains accurate when faced with actions from different sources.

The platform offers three tracks:

• WorldOracle-Core: evaluates predictions over the first 30 seconds of human-operated trajectories.

• WorldOracle-Policy: evaluates predictions over complete episodes of autonomous policy rollouts on AGIBOT G2.

• WorldOracle-Long: evaluates sustained prediction over complete, human-operated long sequences.

The WorldOracle study compares several embodied world-model benchmarks. Of these, WorldOracle is the only one that simultaneously covers action-sequence input, real-world execution data, failure cases, long-horizon interaction, multiple embodiments, multi-view evaluation and autonomous policy rollouts.

Breaking down prediction error: outcomes and process

The predicted and real robot arm positions may closely match even when the model gets the fruit’s grasp state wrong; two final states may look alike while the paths that led to them differ. A single score can hardly capture such differences.

WorldOracle checks predictions with four complementary metrics:

• Physical State Correctness: checks whether object positions, grasp states and liquid transfer at the end of each interaction match the real outcome.

• Interaction Spatial Alignment: checks whether the positions and movement paths of objects, robot arms or human hands are consistent with the real process.

• Key Interaction Area Fidelity: focuses on regions such as the object, gripper and hand, combining pixel-level, perceptual and semantic similarity to reduce the influence of large background areas on the score.

• Multi-View Action Coherence: for models that support multi-view output, checks whether robot motion across synchronized camera views is aligned with the real execution.

The first three metrics are averaged with equal weights to form the WorldOracle-SV (single-view) score; for multi-view evaluation, the fourth metric is added and the four are again equally weighted. Sub-scores are also retained, allowing developers to distinguish problems in state prediction, motion trajectories and local detail.

To validate the metrics, the researchers asked 119 participants to compare 128 pairs of predicted videos. The four metrics agreed with human choices on which prediction was better 88.1% to 93.2% of the time.

The researchers also note the benchmark's limits. It measures how closely a prediction matches a single recorded real execution. For fluids, deformable objects and stochastic contact, the same action can lead to several plausible outcomes, and a single reference cannot fully describe that range.

Training support also differs across embodiments: AGIBOT G2 models use a benchmark-specific training set, YAM models use public data, and the human-hand setting is evaluated zero-shot. Scores across the three settings should therefore not be used directly to judge embodiment difficulty or cross-embodiment generalization.

How Can Developers Participate?

Open submissions give model iteration concrete feedback

WorldOracle supports both open-weight and closed-source models. After registration and approval, developers choose a track according to the embodiment or manipulation setting their model supports, generate and upload predicted videos following the submission guide, and view composite scores, sub-scores and comparisons.

For model iteration, this feedback can be tied to specific questions: Did a change make object motion more accurate? Did it reduce incorrect predictions of failed operations? Does the improvement still hold as the prediction horizon grows?

Whether the futures generated by a world model can be relied upon must be tested against real execution. WorldOracle turns that test into a unified set of tasks, data and scoring methods, giving developers a basis for comparing models, identifying errors and verifying improvements.

Project page

Submission guide