AdaPT Bridges Sim-to-Real to Teach Humanoids the Tennis Styles of Federer, Nadal, and Djokovic

Key Takeaways

Hide

• Researchers from the Shanghai AI Laboratory, Dobot Robotics, Shanghai Jiao Tong University, and Noitom Robotics have introduced AdaPT, a framework that teaches humanoid robots to mimic the distinctive playing styles of tennis legends Roger Federer, Rafael Nadal, and Novak Djokovic.
• Unlike prior athletic benchmarks focused solely on raw hit success, AdaPT extracts kinematics directly from broadcast match video to preserve nuanced whole-body biomechanics and technical signatures, including topspin follow-throughs and wrist pronation.
• To overcome the sim-to-real gap inherent in decoupled planner-tracker architectures, the system integrates explicit speed adaptation and residual tracking to compensate for real-world actuator latency and perception noise.
• Evaluated physically on the 1.35-meter Unitree G1 and the full-sized, 1.7-meter Dobot Atom, the team also demonstrated markerless, "in-the-wild" serving using onboard stereo cameras and VIVE trackers.

In recent months, dynamic racket sports have emerged as a premier proving ground for high-speed humanoid agility. While prior efforts like the LATENT framework demonstrated that robots could sprint and maintain multi-shot rallies, they largely focused on raw task completion—hitting the ball over the net by any kinematic means necessary. In human sports biomechanics, however, elite form is not merely aesthetic; coordinated kinetic chains are essential for efficient force transmission, energy management, and rapid post-strike recovery.

Now, a research collaboration between the Shanghai AI Laboratory, Dobot Robotics, Shanghai Jiao Tong University, and Noitom Robotics has unveiled AdaPT (Adaptive Motion Planning and Tracking). The framework enables humanoid robots to learn both rally strokes and full serving sequences directly from broadcast television footage, faithfully replicating the distinctive physical signatures of tennis greats Roger Federer, Rafael Nadal, and Novak Djokovic.

Mining Broadcast Video for Athletic Priors

Capturing clean motion-capture (MoCap) data across a full-sized tennis court is prohibitively complex and expensive. To bypass this limitation, the researchers built a data extraction pipeline that reconstructed 3D body meshes from broadcast match clips using SMPL models, followed by retargeting to humanoid joint limits.

The resulting dataset encompasses roughly 10 hours of rally footage annotated by stroke type (forehand, backhand, serve) and ball spin (flat, topspin, slice), alongside supplementary high-fidelity MoCap sessions. Because video pose estimators struggle with rapid wrist occlusions during high-speed swings, the pipeline applies domain-specific wrist adjustments and physically corrects the motions in simulation to ensure dynamic plausibility before training.

Decoupled Architecture with Speed Adaptation

To preserve the authentic stylistic flavor of each player while maintaining reactive task performance, AdaPT employs a decoupled planning-and-tracking hierarchy:

1. Rally Generation: For open rallies, where incoming balls require diverse responses, an MVAE (Motion Variational Autoencoder) motion generator produces kinematic trajectory candidates conditioned on predicted ball flight.
2. Serve Execution: Serving is a structured, self-initiated motion. AdaPT tracks reference serve trajectories directly and introduces a residual tracker to make micro-adjustments for variations in the ball toss without breaking overall form.

In practice, standard decoupled planner-tracker pipelines suffer from severe sim-to-real degradation. Minor physical tracking lag accumulates over time, compounding with visual perception noise and leading to missed strikes.

AdaPT addresses this through explicit speed adaptation. During simulation training in Mjlab with Proximal Policy Optimization (PPO), low-level tracking policies are exposed to randomized temporal blend factors, forcing them to remain stable across varying execution tempos. Concurrently, the high-level planner outputs a continuous speed-adaptation variable, dynamically speeding up or slowing down the planned kinematic trajectory to match incoming ball timing and real-world hardware limits.

Real-World Deployment: From G1 to Dobot Atom

The team deployed AdaPT on both the compact, 1.35-meter Unitree G1 and the full-sized, 1.7-meter Dobot Atom humanoid. In controlled laboratory tests using a specialized ball-launching machine and reflective balls on a Tennis 10s court, the adaptive system delivered dramatic real-world improvements over non-adaptive baselines:

• Rally Consistency: On the Unitree G1, AdaPT achieved forehand and backhand hit rates of up to 64% under unseen ball trajectories, whereas baseline decoupled pipelines without speed adaptation dropped below 24% due to tracking drift.
• Preserving Technical Form: By incorporating keyframe-guided tracking rewards on backswings and tossing parabolas, the robot executed genuine wrist pronation and full kinetic chain whipping motions during serves, achieving real-world serve landing success rates up to 86.7%.
• In-the-Wild Serving: Beyond optical MoCap arenas, the researchers demonstrated autonomous serving on standard courts using a YOLO-based stereo vision system with a ZED X camera for ball tracking and waist-mounted HTC VIVE Ultimate Trackers for robot localization.

Hardware and Algorithmic Trade-Offs

The study reveals key engineering insights for dynamic humanoid robotics. The authors observed a fundamental algorithmic trade-off: tightly coupled end-to-end policies (such as those underpinning PULSE or LATENT) exhibit faster reflexive responsiveness to immediate sensory input, but often sacrifice biomechanical naturalness. In contrast, decoupled planner-tracker systems preserve high-fidelity human style, but place far greater demands on long-horizon future ball trajectory forecasting.

Hardware constraints also remain a factor. While the team successfully demonstrated amateur-level serves using a Unitree Dex-3 multi-fingered hand, its substantial weight created balance risks during high-speed arm swings, necessitating a lightweight parallel gripper for professional, high-velocity serving demonstrations.

By proving that stylistic motor intelligence can be extracted directly from unstructured public video and transferred reliably to physical humanoids, AdaPT provides a practical blueprint for training agile, whole-body robotic athletic skills without requiring specialized laboratory capture environments.