Skip to content

Midscene.js

Describe a UI step and let AI drive the browser

Visit official site

What Midscene.js is

The tool is described as a UI automation tool that uses a multimodal large language model to read an interface and act on a plain description of the interaction steps or the wanted data format.

What you can do with it

  • Run actions, extract data and assert results with .ai, .aiQuery and .aiAssert
  • Try the main features on any page through a browser extension without code
  • Add the tool to Puppeteer or Playwright test frameworks
  • Inspect intermediate data in visual reports and a debug Playground

Example uses

  • Enter a keyword in a search box and run a search with .ai
  • Pull product titles and prices from a list with .aiQuery
  • Try quick UI automation on any page through the Chrome extension

Getting started

  • Visit the site and download the matching browser extension
  • Open any page and describe the action in the natural language box
  • Write scripts with the methods or describe steps inside the extension
  • Run the script and let the AI carry out the UI actions

Pros & cons

✓ What we like

  • Data is sent directly to the chosen model with no third-party service
  • OpenAI GPT-4o or another multimodal model can be selected

! What to watch out for

  • Actions are driven by a written description of steps or data format
  • Using the tool means connecting to a chosen multimodal model

FAQ

What methods are named?

The feature list names .ai, .aiQuery and .aiAssert for actions, queries and assertions.

How can it be integrated?

The material lists a browser extension, Puppeteer and Playwright, plus a visual report and debug Playground.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review