Midscene.js: Let AI Be Your Browser Operator 191

Midscene is an automated browser assistant based on ai technology that can operate web pages through natural language. You simply describe the actions you want to perform, and Midscene will automatically handle web pages, such as posting tweets, verifying content, and searching for information.

Key Features of Midscene.js

• Natural Language Interaction: Users can control the browser using natural language commands, such as "click the login button" or "search for a keyword", and Midscene can understand and execute these commands.

• Data Extraction: Midscene can automatically extract information from web pages and output it in a structured format (such as JSON), for example, extracting search results or table data.

• Web Page Operations: It supports performing various operations on multiple websites, such as posting on Twitter, filling out forms, clicking links, etc., without requiring users to write complex code.

• Visual Reports: Each run generates a detailed visual report, including animation playback and step-by-step details, making it easy to debug and optimize the testing process.

• High Integration: Supports seamless integration with tools such as Puppeteer and Playwright, suitable for various automated testing scenarios.

• High Flexibility: Allows users to choose different large language models (LLMs), and even supports the open-source model UI-TARS.

• UI Automation Testing: Midscene can be used as a UI automation testing tool to help developers quickly write and maintain test scripts, improving testing efficiency.

Use Cases of Midscene.js

• Social Media Management: Users can directly give Midscene commands, such as "post a status update" or "reply to a comment", and Midscene will automatically complete these operations, saving time and effort.

• Data Collection: When conducting market research or data analysis, users can use Midscene to quickly scrape information from specific websites, such as "extract all product prices and descriptions", and Midscene will automatically process and provide the required structured data.

• Automated Testing: For developers, Midscene's natural language commands make writing UI tests much simpler. Developers only need to describe the behaviors to be tested, and Midscene can automatically execute the test steps and generate reports.

How to Use Midscene.js:

Example code (using Puppeteer):

const puppeteer = require('puppeteer');
(async () => {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();

    // Navigate to the target website
    await page.goto("http://example.com");

    // Execute Midscene.js actions
    await midscene.execute(page, '输入用户名 "testuser"');
    await midscene.execute(page, '点击登录按钮');

    // Data extraction
    let userData = await midscene.query(page, '提取用户信息为 JSON 格式');
    console.log(userData); 

    // Close the browser
    await browser.close();
})();

Using Midscene.js is super simple. Just install the Chrome extension and you can start using it. No programming background is required — just describe the task you want to perform, and it will understand your intent and automatically execute the corresponding browser operations. Isn't it amazing?

GitHub：https://github.com/web-infra-dev/midscene