Apple's HALO Teaches a Unitree G1 to Navigate Unseen Rooms With Simulation-Generated Data

Apple researchers have trained a Unitree G1 humanoid to walk to objects, sit on chairs, crawl under tables and push boxes in rooms it has never seen, training mostly on motion data generated in simulation. The system, called HALO, completed 91% of 360 real-world trials across six unmapped environments without any teleoperation data.

Key Takeaways

Show all

• HALO is an Apple research system that runs a Unitree G1 from camera input and natural-language requests, with all models running on the robot.
• Its motion generator was trained on about 300 hours of synthetic data: generated human-like motions placed in 1,000 procedurally generated rooms and rendered in Isaac Sim.
• The robot succeeded in 91% of 360 single-task trials in six real environments. Chained tasks were harder, with a 70% average across three sequences.
• Code for the data pipeline, training and deployment is on GitHub; the paper reports no comparisons against other published systems.

The paper, Perceptive Humanoid Motion from Simulation, was posted to arXiv on November 4. All eight authors are listed under Apple, led by Siyuan Hu, with Vladlen Koltun, who ran Intel's Intelligent Systems Lab before joining Apple, as the final author. HALO stands for "Humanoid Autonomy, Learned in simulation, running Onboard."

Last month we covered ETH Zürich's contact-planning research, which also ran on a Unitree G1 and credited Apple alongside the ETH AI Center and NVIDIA. HALO is an in-house Apple project, with code published on the company's machine learning research GitHub.

The data problem it targets

The authors' starting point is that teleoperation and wearable motion capture "require at least an hour of human time to produce an hour of data." HALO replaces that with a pipeline that generates training data without anyone operating the robot.

The pipeline uses Kimodo, a motion diffusion model, to produce whole-body movements from text descriptions, waypoints and a handful of keyframes. Those motions are placed in 1,000 rooms generated with Infinigen, a procedural 3D scene generator: 45% living rooms, 30% offices and 25% bedrooms. Each scene is rendered in NVIDIA's Isaac Sim with ray tracing from the viewpoint of a fisheye camera matched to the robot's own, with randomised blur, noise, exposure and colour to narrow the sim-to-real gap.

The result is 162,698 episodes, or 296 hours at 30 frames per second, covering six skills. Some human data remains: the tracker's first training stage uses clips from LAFAN1, a public human motion-capture dataset, retargeted to the G1; the crawling and crouching keyframes came from phone videos of a person going under a table, lifted to 3D and retargeted; and the box-pushing data includes a motion-capture recording.

Three layers on the robot

HALO splits control into three models that run at different speeds:

• Orchestrator: an off-the-shelf vision-language model, by default Qwen3.6-35B-A3B in 4-bit precision, runs about every two seconds. It breaks a request such as "go sit on the chair" into subtasks and decides when to move on or retry.
• Motion generator: about three times a second, a model built on a frozen Qwen3.5-2B vision encoder and a 49-million-parameter diffusion transformer takes the camera image and current subtask and plans five seconds of whole-body motion.
• Motion tracker: at 50 Hz, a policy trained with reinforcement learning in Isaac Lab follows that plan using depth images, proprioception and the robot's IMU, and outputs targets for the G1's 29 joints.

The hardware is a stock G1 with two additions: an OAK-1W wide-angle camera in a custom head mount and a Mac mini with an M4 Pro chip and 64 GB of memory in an enclosure on the robot's back. The Mac mini runs the orchestrator and generator, while the tracker runs on the G1's built-in NVIDIA Jetson Orin NX. The robot is untethered.

The results

Each of the six skills was tested 10 times in each of six real environments and in 100 simulated rooms:

Chaining skills was less reliable. Going through a door and then to an object succeeded in all 10 real trials, but crouching under a table and then sitting worked 60% of the time, and crouching, walking to a box and pushing it worked 50% of the time. Those chained tests used a single real environment. The authors note that chained success falls below what the single-task rates would predict, and suggest adding recovery behaviours such as searching and turning in place.

The ablations show where the gains come from. Training the generator on one room gave a 43% success rate on the sitting task in held-out simulated rooms, against 84% with all 1,000 rooms. Swapping the diffusion decoder for plain regression dropped it to 19%, and removing depth from the tracker cut it to 74%.

The paper does not compare HALO against other published systems on the same tasks. A table lists the number of real-world trials reported by related work such as ZeroWBC, GRAIL and TANGO, but not their success rates under matching conditions.

What's missing

The authors list several limitations. Each new skill still needs a person to write prompts and author keyframes, so "the repertoire grows only as fast as a person can specify additional tasks." The robot has no sense of touch, and the tracker "follows the kinematic plan without knowing what the plan is for." None of the six skills involves grasping or manipulating objects with the hands.

The GitHub repository includes the data-generation pipeline, the generator and tracker training code, and deployment code for MuJoCo and the real G1. The README does not list trained model weights or the 300-hour dataset for download.