Skip to content

OmniParser V2

Turn UI screenshots into structured elements for an LLM

Visit official site

What OmniParser V2 is

The model is described as a Microsoft Research technique that helps an LLM understand and operate a graphical user interface. It converts UI screenshots from pixel space into interpretable structured elements so icons can be identified and actions executed on screen.

What you can do with it

  • Convert UI screenshots into structured elements for an LLM
  • Detect small icons and link them to interactive screen regions
  • Combine the model with several LLMs such as OpenAI, DeepSeek and Qwen
  • Use OmniTool to speed up experiments and development

Application scenarios

  • Recognise interface elements and run test scripts in automated testing
  • Parse a user interface and give operation advice in customer support
  • Work with GPT-4o on GUI grounding tasks for high-resolution screens

Getting started

  • Download the OmniParser V2 code from GitHub
  • Install OmniTool and configure the required LLM environment
  • Parse UI screenshots with the model to extract structured elements
  • Feed the parsed results into the chosen LLM to produce interaction instructions

Pros & cons

✓ What we like

  • Works with several LLMs including OpenAI, DeepSeek and Qwen
  • Image size reduction for the icon caption model lowers inference latency

! What to watch out for

  • Parsing is described as starting from UI screenshots
  • The reported accuracy figures are tied to GPT-4o on ScreenSpot Pro

FAQ

What does it do with a screenshot?

The material says it converts UI screenshots into interpretable structured elements.

Which LLMs are named?

The material names OpenAI, DeepSeek and Qwen among the supported models.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review