GUIRoboTron-Speech: Meituan and Zhejiang University's First Voice-Interactive GUI Agent 212

Meituan and Zhejiang University collaborated to develop GUIRoboTron-Speech, a GUI agent that can make decisions directly using voice commands and device screenshots. It enables users to control computers through voice and device screenshots, avoiding the inconvenience of traditional text input.

GUIRoboTron-Speech addresses the limitations of traditional text-based approaches. The team successfully trained this agent by creating high-quality speech datasets, using staged training, and a hybrid instruction training strategy, enabling it to perform excellently on multiple tasks. The research found that when user intent is clear, voice commands achieve higher task completion success rates than text commands, and the ability to handle complex instructions will continue to be improved in the future.

Core Features

• Voice Command Driven: GUIRoboTron-Speech can directly receive voice commands and device screenshots to predict and execute actions. Users don't need to manually type text—they can control GUI agents through voice.

• Multimodal Interaction: It combines both voice and visual modalities, accurately understanding the correspondence between voice commands and visual elements of the GUI interface.

Research Methodology

• Dataset Generation: Due to the lack of voice command datasets, the team used advanced random-voice text-to-speech (TTS) models to convert large-scale text command datasets into voice command datasets with various speaking styles and voice timbres.

• Progressive Training Framework: Training is divided into two stages:

• Grounding Stage: The model learns the precise correspondence between voice commands and GUI visual elements, such as understanding "click the 'OK' button" and locating the "OK" button in a screenshot.

• Planning Stage: The model learns to execute multi-step complex tasks, such as "first log into the account, then find the latest email and open the attachment," demonstrating logical reasoning and task planning capabilities.

• Heuristic Hybrid Instruction Training Strategy: To address the modality imbalance issue in pre-trained models, the team mixed voice and text instructions during training, enabling the model to process both types of input equally.

Performance

• Instruction Modality Comparison: In the ScreenSpot task, the average grounding accuracy of the model trained with only voice commands was 1.6% lower than that trained with only text commands, but after adopting the hybrid instruction strategy, accuracy improved by 1.9%.

• GUI Visual Grounding: In the Mobile and Web domains, GUIRoboTron-Speech's performance with voice command input is comparable to advanced text-command-based methods.

• Multi-step Task Execution: On multi-step task datasets such as AndroidControl and GUIOdyssey, its performance is comparable to some advanced text-command-based models.

Advantages and Significance

• Enhanced Interaction Convenience: Users can complete various operations through voice without manually typing text when interacting with devices, greatly improving interaction convenience.

• Expanded Application Scenarios: This technology is not only suitable for convenient daily operations by ordinary users, but also for special scenarios such as driving and healthcare where hands-free operation is required, providing safer and more efficient interaction methods.

• Advancing Technology Development: GUIRoboTron-Speech provides new ideas and methods for the application of voice interaction technology in the GUI field, promoting the development of related technologies.

Project Links

Technical Paper:https://arxiv.org/abs/2506.11127

Github Repository:https://github.com/GUIRoboTron/GUIRoboTron-Speech