ByteDance UI-TARS: An Open-Source Self-Learning GUI Agent 242

ByteDance open-sourced a native graphical user interface (GUI) agent model on January 22, 2025: UI-TARS. The name UI-TARS is derived from the TARS robot in the movie Interstellar. It is designed for automating graphical interface interactions, signaling a high degree of intelligence and autonomous thinking capability.

What is UI-TARS?

UI-TARS can observe interfaces through screenshots like a human, and perform keyboard and mouse operations to simulate human interaction with phones and computers to complete tasks. It possesses advanced perception, reasoning, and interaction capabilities.

The goal of UI-TARS is to control and interact with user interfaces through a highly intelligent model, moving toward a more natural and efficient way of human-computer interaction.

UI-TARS Features:

• Natural language control powered by a vision-language model

• Screenshot and visual recognition support

• Precise mouse and keyboard control

• Cross-platform support (Windows/macOS)

• Real-time feedback and status display

UI-TARS Capabilities:

• Self-learning ability: UI-TARS has the ability to learn on its own, optimizing its operations through continuous interaction and feedback.

• Multi-platform support: The model is not only applicable to PC and macOS but also runs on mobile phones and web pages, offering broad applicability.

• Natural language processing: UI-TARS can take screenshots and natural language instructions as input and accurately predict the next action needed to complete the instruction, helping users operate UI interfaces.

• Image recognition: UI-TARS can process screenshots, recognize interface elements, and perform corresponding actions, making it highly effective for UI automation testing and operations.

• Enhanced perception: Leverages large-scale GUI screenshot datasets to achieve context-aware understanding and precise annotation of UI elements.

• Unified action modeling: Standardizes cross-platform actions into a unified space and achieves precise localization and interaction through large-scale action trajectories.

• System-2 reasoning: Incorporates deliberate reasoning into multi-step decision-making, involving multiple reasoning modes such as task decomposition, reflective thinking, and milestone recognition.

• Iterative training: Resolves data bottlenecks by automatically collecting, filtering, and reflectively refining new interaction trajectories on hundreds of virtual machines.

UI-TARS Application Scenarios

• Automated testing: UI-TARS can be used for software automated testing, helping developers quickly verify the functionality and stability of user interfaces.

• Web automation: When combined with UI-TARS, Midscene.js supports driving AI large models through natural language, performing a series of actions in the current page state to approximate human command goals.

• Task automation: In daily work, UI-TARS can help users automatically perform repetitive tasks such as data entry and information retrieval, improving work efficiency.

• Cross-platform operation: The UI-TARS Desktop version supports Windows and macOS systems, demonstrating its broad applicability across different operating systems and providing users with a consistent interactive experience.

• Daily use and office automation: Through natural language control, UI-TARS can simplify daily office tasks such as file management and application operations, improving work efficiency.

Unlike traditional systems that rely on modular frameworks or manually optimized prompts, UI-TARS adopts an end-to-end architecture that relies purely on visual input, achieving full automation of complex tasks. UI-TARS is not limited to testing scenarios; it places greater emphasis on the intelligentization of everyday user interactions, making computer interaction more human-centered through the power of AI.

UI-TARS project address: https://github.com/bytedance/UI-TARS