Amazon Nova Act: Amazon's AI Agent That Controls a Web Browser and Autonomously Performs Simple Tasks 143

What is Nova Act?

Nova Act is an AI agent launched by Amazon that is capable of controlling a web browser and autonomously executing simple tasks. It can drive a web browser and independently complete simple tasks such as searching, filling out forms, and checking out. It also supports recognition of detailed instructions and decomposition of complex tasks, helping users streamline everyday operations.

Nova Act Features

• Browser Control Capability: Nova Act can automate operations by breaking down complex tasks into atomic instructions (such as searching, checkout, and filling out forms) and calling browser APIs or Playwright tools. Examples include ordering salads online, making restaurant reservations, or setting up auto-reply emails. In its internal testing, it achieved a success rate of over 90% on UI interaction tasks such as date selection and pop-up handling.

• Complex Task Decomposition: The Nova Act SDK allows developers to decompose complex tasks into a series of executable simple commands, such as "find the most convenient route from my home to these three stores, then go to a movie around 6 PM."

• Detailed Instruction Recognition: It can recognize more detailed instructions, such as declining insurance upsells while shopping.

• Multi-modal Interaction Architecture: Built on Amazon's proprietary Nova foundation models (such as Nova Micro/Lite/Pro) to parse web content, combined with a high ScreenSpot Web Text score of 94% to simulate human operating logic, achieving a closed loop of visual recognition, behavior simulation, and process control.

Performance

In Amazon's internal ScreenSpot Web Text test, Nova Act scored 94%, outperforming OpenAI's CUA (88%) and Anthropic's Claude 3.7 Sonnet (90%). However, Amazon did not test it against more commonly used agent evaluation benchmarks (such as WebVoyager), so its performance comparison still requires more external validation.

Nova Act Use Cases

• Shopping Assistant: Nova Act can help users complete online shopping, such as ordering salads from Sweetgreen or making dinner reservations.

• Trip Planning: It can assist users in planning trips, such as finding the best route from home to multiple locations and arranging follow-up activities like going to a movie.

• Form Filling: Nova Act can automatically fill out various online forms.

• Scheduling: It can also select dates on a calendar, helping users arrange their schedules.

• Information Lookup: Nova Act can search for and answer questions related to content displayed on the screen based on user instructions.

• Integration with Alexa+: Nova Act will also power Amazon's upcoming Alexa+ upgrade, a generative-AI enhanced version of Amazon's popular voice assistant.

Nova Act is developed by Amazon's newly established San Francisco AGI (Artificial General Intelligence) Lab, and Amazon Nova Act is the lab's first publicly available product.

Detailed Introduction:https://labs.amazon.science/blog/nova-act

Official Website:https://nova.amazon.com/