Beyond Mechanical Chatbots: Galbot Unveils RoboGesture to Tackle Real-Time Nonverbal Communication

Humanoid robots have made dramatic strides in whole-body locomotion and contact-rich industrial manipulation over the past two years, but face-to-face social interaction remains an uncanny valley. When standard speech-language models are wired into humanoid hardware, the resulting conversation often feels disjointed: mechanical figures pause awkwardly, recite colloquial text in a monotone cadence, or flap their arms in rhythmic, canned loops that bear little connection to what is actually being said.

To tackle this communicative disconnect, Beijing-based robotics firm Galbot—working in collaboration with Tsinghua University, Peking University, and other academic partners—has introduced RoboGesture. Set for presentation at ECCV 2026 on September 10, the project provides an end-to-end framework designed to synthesize natural, semantically aligned, and physically safe co-speech gestures in real time.

The release comes during a period of rapid product diversification for Galbot. Having built its initial commercial foundation on wheeled logistics machines for warehouse and pharmacy inventory backed by investors like CATL, the startup has aggressively expanded into the bipedal arena. Just this week, Galbot opened pre-orders for its bipedal ET1 humanoid, an agile 32 kg platform tailored specifically for storefront reception, hospitality, and live stage performance. Natural, real-time social gesturing is a foundational prerequisite for exactly those use cases.

Solving "Modality Eclipse" in Robot Space

Historically, co-speech gesture generation has lived mostly in the digital domain, driving virtual characters and 3D avatars. Applying these methods to physical humanoids introduces severe challenges. Most prior frameworks generate gestures in a human skeletal space and then retarget the motion to the robot online via inverse kinematics and PD controllers. In practice, this human-first pipeline often results in severe self-collisions—such as mechanical hands slamming into the chest—or unnatural high-frequency motor jerk.

RoboGesture departs from this paradigm by learning directly inside the robot's kinematic action space. Targeting a 41-degree-of-freedom (DoF) upper-body configuration (17 DoFs across the torso and arms, paired with 24 DoFs across dual BrainCo dexterous hands), the architecture circumvents inference-time retargeting bottlenecks.

+-------------------------------------------------------------------------+
|                       ROBOGESTURE ARCHITECTURE                          |
+-------------------------------------------------------------------------+
                                                                           
  [ User Audio ]                                                           
         │                                                                 
         ▼                                                                 
  ┌───────────────┐     RVQ Audio Tokens      ┌───────────────────────────┐
  │  Mimi Codec   │ ────────────────────────> │ Hierarchical Aligner      │
  │   Tokenizer   │                           │ - Shallow: Beat Pulses    │
  └───────────────┘                           │ - Deep: 300 Semantic Cls  │
                                              └─────────────┬─────────────┘
                                                            │ (FiLM & Cross-Attn)
                                                            ▼
  ┌───────────────┐  15% Anti-Inertia Masking ┌───────────────────────────┐
  │  Past Motion  │ ────────────────────────> │ Continuous DiT Generator  │
  │  Trajectory   │                           │ (Conditional Flow Match)  │
  └───────────────┘                           └─────────────┬─────────────┘
                                                            │ (120 FPS Chunk)
                                                            ▼
                                              ┌───────────────────────────┐
                                              │  MPC Kinematic Filter     │
                                              │  (QP Collision Avoidance) │
                                              └─────────────┬─────────────┘
                                                            │
                                                            ▼
                                              [ Safe Humanoid Execution ]

At the core of the framework is an effort to solve what the authors term "modality eclipse". In streaming environments, deep motion models frequently fall into a lazy kinematic shortcut: because previous joint positions offer a strong mathematical prior for smoothness, the system simply continues the physical momentum of historical motion while ignoring newly incoming audio signals.

To overcome this, RoboGesture implements a multi-layered countermeasure:

• Hierarchical Semantic-Acoustic Aligner: Instead of relying strictly on text embeddings, the pipeline tokenizes raw streaming audio using the Mimi codec. Shallow transformer layers track rhythmic transients and speech onsets, while deeper layers compress high-level semantic meaning through an auxiliary 300-class gesture classifier. This design captures pre-phonetic acoustic cues—such as a speaker tensing up or inhaling prior to an emphatic exclamation—allowing the robot to initiate preparatory body language before a keyword is finished.
• Anti-Inertia CFG Masking: During the second-stage joint training of the continuous Diffusion Transformer (DiT) parameterized with Conditional Flow Matching, the network randomly masks past motion context with a 15% probability. This forces the generator to perform "cold starts" driven purely by the speech cues rather than relying on past inertia.
• Delayed History Injection: Past-motion context is injected only in the later transformer blocks of the DiT generator, ensuring the early layers prioritize incoming audio and semantic tokens.

To guarantee hardware integrity, synthesized 1-second chunks are routed through an online Model Predictive Control (MPC) safety filter. Formulated as a convex quadratic program solved on a frame-by-frame basis, the filter reduces upper-body self-collision frame ratios from 4.16% down to 0.13%.

The Data Engine Behind the Hands

Because physical humanoid data remains scarce, the research team constructed a semi-synthetic data generation pipeline. Expanding on taxonomic foundations from datasets like SeG and EgoGesture, the team assembled the RoboGesture Dataset, comprising over 300 verified semantic gesture categories recorded via marker-based motion capture.

Using large language models to generate scripts and paired text-to-speech engines to generate audio, the team synthesized roughly 1,000 hours of collision-vetted, robot-specific audio-motion pairs. Crucially, semantic actions are timed to initiate roughly 400 ms prior to their corresponding spoken keyword, mimicking natural human cadence.

In quantitative testing against baselines including LivelySpeaker, DiffSHEG, and Semantic Gesticulator, RoboGesture outperformed the field on most BEAT metrics, logging a Fréchet Gesture Distance (FGD) of 0.8452 compared to 2.2316 for DiffSHEG and 3.0147 for Semantic Gesticulator. The one exception is Diversity, where Semantic Gesticulator scores higher—though the authors argue that raw diversity scores can be inflated by unstable or jittery motion. In pairwise human evaluations across rhythm alignment, semantic accuracy, and physical hand consistency, evaluators favored the RoboGesture trajectories, with the clearest margins on semantic accuracy and physical consistency.

Data source: Wang et al. / Galbot Inc. / Tsinghua University. Collision figures are BEAT-set values; on the out-of-domain SemanticBEAT set RoboGesture logs 0.13%. Separately, the MPC filter alone cuts self-collision frames from 4.16% to 0.13%.

The Latency Reality Check

Galbot demonstrated the software running on a Unitree G1 humanoid outfitted with BrainCo dexterous hands. In project demonstration videos, the robot exhibits expressive nonverbal cues: it forms an "OK" gesture with its fingers during agreement and pats its chest during emphatic statements. Across 200 real-world deployment trials, the team reports zero falls and zero emergency stops.

However, the demonstration also surfaces the primary bottleneck facing real-time social humanoid deployments: overall turn-around latency.

Galbot advertises closing the "listen → respond → gesture" loop in approximately 2 seconds. The paper's own per-module breakdown—which the authors caution should be read as streaming latency per component rather than a strict end-to-end total—shows the gesture generation subsystem itself is remarkably lean:

• The continuous DiT motion generator sustains an effective throughput of 120 FPS, processing a 1-second motion chunk in roughly 250 milliseconds.
• The frame-by-frame MPC safety filter executes in just 5.6 milliseconds per frame.

Instead, the noticeable delay visible in live demonstrations stems from the upstream conversational pipeline. Waiting for an automatic speech recognition (ASR) module to verify conversational pause endpoints (~0.5 seconds), running a LoRA-tuned Qwen language model to draft a response (~0.14 seconds), streaming the synthesized audio through text-to-speech (~0.41 seconds), and tokenizing that audio (~0.16 seconds) consume the bulk of the response window before the motion stack can begin streaming its first usable gesture chunk. The authors themselves frame total latency as governed mainly by how quickly the first audio chunk arrives—typically under 1.5 seconds—plus roughly 0.25 seconds of first-chunk inference.

While a two-second latency envelope is manageable for a stage performance, it remains on the edge of conversational comfort for intimate, one-on-one human interactions.

Expanding Beyond Wheeled Utilities

The technical scope of RoboGesture remains intentionally constrained: the current policy only governs stationary upper-body gesturing and dexterous hand poses, leaving lower-body balance to the host robot's underlying standing controller. Full-body expressive gestures that shift the robot's support polygon or blend locomotion with social expression remain an unsolved target for future reinforcement learning controllers.

Even so, RoboGesture marks a significant conceptual evolution for humanoid control architectures. As companies transition from proving bare hardware stability to chasing real-world adoption, social embodiment cannot remain an afterthought driven by pre-recorded animations.

For Galbot—which has been building investor momentum toward a potential $4 billion Hong Kong listing—proving it has the software stack to handle nuanced human interaction is just as critical as showcasing athletic tennis rallies. If humanoids are truly destined to work among people in hospitality, retail, and caregiving, teaching them how to carry a conversation with their hands may prove just as vital as teaching them to walk.