Cradle: Multimodal AI Agent Framework with Direct Keyboard & Mouse Control 158

A general-purpose multimodal AI framework for computer control, enabling AI Agents to directly operate the keyboard and mouse like a human and interact with any open-source or proprietary software.

#Ai工具箱 #Ai开源项目

Visit Cradle: Multimodal AI Agent Framework with Direct Keyboard & Mouse Control

Cradle: Multimodal AI Agent Framework with Direct Keyboard & Mouse ControlIntroduction

What is Cradle?

Cradle is a multimodal AI framework for General Computer Control (GCC), jointly launched by Kunlun Tech, Beijing Academy of Artificial Intelligence (BAAI), and other institutions. It uses screenshots as input and keyboard and mouse operations as output, enabling AI agents to directly control computers and complete various tasks just like a human. Cradle features core capabilities including information gathering, self-reflection, task inference, skill planning, action planning, and memory management, and can be widely applied in scenarios such as gaming, everyday software operations, and professional software operations.

Cradle：多模态AI Agent框架，能够直接控制键盘和鼠标.webp

Cradle Core Features

Information Gathering: Extracts multimodal information from screen images and audio for decision-making.
Self-Reflection: Evaluates the success of previous actions and analyzes the causes of failure to guide subsequent actions.
Task Inference: Infers and selects the best next task based on the current environment and historical information.
Skill Planning: Generates and updates skills related to a given task to adapt to different computer operation needs.
Action Planning: Generates specific operations for keyboard and mouse control, translating strategies into executable commands.
Memory and Knowledge Management: Possesses long-term and short-term memory systems that store past experiences and skills, retrieved and applied as needed.

Cradle Application Scenarios

Game Environments: Such as Red Dead Redemption 2, Stardew Valley, Cities: Skylines, etc.
Everyday Software Operations: Such as browsers, email clients, office software, etc.
Professional Software Operations: Such as graphic design or video editing software.

Cradle Technical Principles

Multimodal Input Processing: Receives and processes multimodal inputs including screen images and audio, simulating human perception to understand computer interfaces and environments.
Information Extraction and Understanding: Uses large multimodal models (such as GPT-4V) to identify visual elements, text information in images, and instructions or feedback in audio.
Self-Reflection Mechanism: Evaluates whether previously executed actions were successful through a reflection module and analyzes the reasons for failure.
Task Inference and Planning: Determines current priority tasks and formulates new actions to complete them.
Skill Generation and Update: Generates new skills or updates existing skills based on the current task.

Cradle：多模态AI Agent框架，能够直接控制键盘和鼠标.webp

Cradle Features

Generality: Does not rely on any internal APIs and can interact with any open-source or closed-source software.
Strong Reasoning Capabilities: Based on large language models, capable of complex task planning and execution.
Self-Improvement: Continuously optimizes performance through a self-reflection mechanism.
Skill Accumulation: Can progressively accumulate and reuse various operational skills.

Cradle has been successfully applied in games such as Red Dead Redemption 2, Stardew Valley, Cities: Skylines, as well as software like Chrome browser and Outlook email.