As AI assistants shift from text interfaces to multimodal companions, there is increasing demand for proactive, situated assistance. Systems such as Project Astra and Gemini 3.1 Flash Live already let users discuss their physical surroundings in real time, often using 2D bounding boxes over camera feeds. While these overlays work well on flat screens, immersive platforms like Android XR raise a new question: how can we move beyond planar UIs to create embodied, spatially aware dialogues?
AgentHands, published at CHI 2026, is a research prototype addressing that question by bringing co‑speech hand gestures into 3D XR. In human communication, hands do more than point: they depict shapes, mimic actions, and emphasize concepts in sync with speech. Using Extended Reality (XR)’s spatial understanding, AgentHands reproduces that synergy. Building on prior work such as Human I/O and Sensible Agent, AgentHands equips AI agents with expressive, synchronized hand gestures that turn abstract verbal instructions into intuitive, physical demonstrations.
A taxonomy for embodied hand agents in XR
A formative study with XR and human–computer interaction (HCI) experts at Google identified what makes a virtual hand legible in 3D. The team developed a multi‑dimensional taxonomy that guides how an agent should use hands to ground a conversation in the user’s physical space:
- Handedness & gesture: choose one or two hands and select from a library of forms (for example, a “palm” to signal caution or a “cylindrical grip” to mimic holding a tool).
- Spatiality: exploit XR depth to place hands appropriately — mid‑air for general conversation, object‑anchored for marking specific parts, or user‑relative for social cues.
- Temporal dynamics & visual effects (VFX): gestures include animated motions such as “pouring” or “tracing,” and XR visual layers add effects (for instance, a red glow to indicate a heat warning).
The AgentHands workflow
AgentHands maps high‑level LLM reasoning to precise, real‑time physical motions that align with the agent’s speech and the XR scene. The key steps are:
- Environment awareness
A lightweight object registration module starts the pipeline. Using eye gaze and scene reconstruction, users can tag items (e.g., an orchid or a laptop), creating a spatial registry of 3D bounding boxes the agent can reference.
- Hand gesture event library
The team built a library of hand gesture behaviors across three semantic categories: a) deictic for referencing, b) iconic for depicting actions or forms, and c) expression for social cues and emotion.
- Gesture‑embedded reasoning
When a user asks a question, the backend LLM generates a response that includes inline GestureEvents. Each event attaches to specific trigger words and encodes gesture primitives according to the taxonomy dimensions.
- Synchronized XR execution
A local parser on the XR headset coordinates text‑to‑speech (TTS) playback with the animation engine. Using word‑level timestamps, the agent’s hands perform co‑speech gestures precisely in sync with spoken words, providing clear spatial references.
By integrating these modules, AgentHands converts a standard LLM output into a rich, multimodal performance: generated responses are realized through both speech and spatially accurate movement, enabling complex instructions to be demonstrated exactly where they apply in the user’s environment.
Application scenarios
Demonstrations showed how embodied gestures combined with XR spatial awareness improve understanding of physical surroundings:
- Interactive tutoring: in an orchid care scenario the agent not only says “check the roots” but moves its hands to the plant base and outlines air roots while explaining their function.
- Technical walkthroughs: for 3D printer operations the agent can demonstrate the exact “turn and click” sequence to manipulate knobs and select files, making complex interface steps intuitive.
- Lifestyle companionship: as a wellness coach the agent can interact with physical choices — for example, perform an interactive warning by holding the user’s hand and applying a visual effect to caution against unhealthy behavior.
User study
To evaluate impact, the team ran a within‑subjects study (N = 12) comparing AgentHands to a speech‑only baseline. Both conditions used the same researcher‑scripted verbal content, so the only difference was the presence of embodied hands and synchronized gestures. Participants completed two procedural tasks balancing everyday care and technical operation:
- Orchid care task: identify plant parts (air roots, stems) and perform multi‑step care activities including targeted watering and proper fertilizer application.
- 3D printer operation task: identify hardware components (nozzle, print bed) and follow a workflow to power on the device, insert an SD card, and navigate the control panel.
Results
The results indicate that combining XR with co‑speech gestures significantly improves spatially grounded interactions across several metrics:
- Significant gains in spatial grounding: participants found it easier to locate specific objects and identify directions referenced by the agent (p < 0.05). The pointing gesture’s precise timing when the agent said “this” removed much of the guesswork present in verbal instructions alone.
- Enhanced understanding of complex actions: required activities were rated easier to follow (p < 0.05). One participant stated that only when the agent showed a lifting gesture did they realize they needed to lift the orchid to let water drain.
- Salient safety cues: warnings were more effective when paired with gestures and visual effects. The “burn” effect used during the 3D printer task was described as “really impressive,” ensuring users noticed the hot nozzle risk.
- Reduced cognitive load: participants found instructions easier to understand and remember. Qualitative feedback described the hands as feeling like “a partner guiding me,” shifting the interaction from tool‑centered search to an embodied dialogue.
Conclusion and future directions
AgentHands advances a model where AI systems not only analyze our world but operate dynamically within it. By grounding conversation in co‑speech gestures and XR spatiality, the approach can lower cognitive load for complex tasks and make spatial computing more human‑centric and accessible. Future work on the Android XR ecosystem will explore personalization, such as adapting gestures to a user’s dominant hand or learning individual spatial routines to enable even smoother human–AI collaboration.
Acknowledgements
This research was primarily conducted by Ziyi Liu during his Student Researcher tenure at Google, as part of a joint collaboration across multiple teams. The authors thank David Li, Zhongyi Zhou, and David Kim for their support, and Adarsh Kowdle, Guru Somadder, and Shahram Izadi for strategic guidance and reviews.
