Mobile-Agent-v2: A Powerful Mobile Device Operation Assistant 198

Alibaba and Beijing Jiaotong University have released Mobile-Agent-v2, a mobile device operation assistant that achieves effective navigation through multi-agent collaboration. It enables automated operation and visual perception of mobile devices through multi-agent collaboration, allowing AI to simulate clicks, swipes, typing, and other operations on your phone just like a real person, thereby performing various tasks.

Background of Mobile-Agent-v2

Traditionally, mobile device operation tasks have been insufficient in multitasking, primarily due to limitations in navigation and focus management within current multimodal LLMs (MLLMs). The consequences of this limitation include performance degradation with long sequences and mixed data formats, making effective navigation of task progress and focus content particularly challenging. The emergence of MLLM-based agents, which enhance the extended capabilities of MLLMs through tool invocation, has brought novel solutions, but they fail to address the inherent navigation issues in mobile device operations.

Mobile-Agent-v, on the other hand, is a pure visual solution based on a multi-agent architecture, with stronger capabilities in task decomposition, cross-application operation, and multilingual support.

Architecture and Methodology of Mobile-Agent-v2

Mobile-Agent-v2 introduces a multi-agent system composed of three specialized agents: the planning agent, the decision agent, and the reflection agent. Each agent plays a distinct role, collectively enhancing navigation and decision-making processes:

Planning Agent: This agent addresses the complexity of lengthy operation histories by summarizing and compressing them into manageable plain-text task progress. This task progress is handed off to the decision agent, facilitating easier navigation and decision-making by reducing context length.

Decision Agent: Operating within the visual perception module, the decision agent processes the condensed task progress and makes informed operational decisions. It is responsible for updating memory units with relevant focus content for reference in future steps. This functionality ensures that the agent can maintain an accurate focus context based on past screens.

Reflection Agent: To manage and correct potential erroneous operations, the reflection agent evaluates the outcome of each operation relative to the expected result. By analyzing screen changes before and after operations, erroneous and invalid operations are identified, classified, and responded to, thereby enhancing the reliability of task execution.

Scenario Applications of Mobile-Agent-v2

Product Search and Purchase: Automatically search for products in shopping apps, add them to the shopping cart, and complete purchases.

Email Sending: Automatically compose and send emails in Gmail.

Navigation: Automatically perform navigation operations in Google Maps.

Video Watching: Search for and comment on specific videos on YouTube.

Compared to Mobile-Agent, which relies on a single agent to perform tasks, Mobile-Agent-v2 introduces a multi-agent collaborative architecture where multiple agents work together to achieve more efficient navigation and task execution.

Experimental results demonstrate substantial improvements over previous mobile agent architectures. The task completion rate of Mobile-Agent-v2 has increased by more than 30%, highlighting the efficacy of multi-agent collaboration. Tasks involving multi-step and interleaved modalities can benefit significantly from the new architecture, as it can effectively navigate and manage the complexities of mobile device operations.

Mobile-Agent-v2 Paper: https://arxiv.org/abs/2406.01014

Mobile-Agent-v2 Code: https://github.com/X-PLUG/MobileAgent

Modelscope-agent: https://github.com/modelscope/modelscope-agent