Skip to content

OmniParser-v2.0

Turn UI screenshots into structured element lists

Visit official site

What OmniParser-v2.0 is

The tool is described as a screen parsing technique from Microsoft that turns irregular UI screenshots into structured element lists. The lists are stated to include the positions of interactive regions and descriptions of what icons do.

What you can do with it

  • Convert a UI screenshot into a structured format
  • Extract interactive regions and icon function descriptions
  • Integrate with LLMs such as OpenAI, DeepSeek and Qwen
  • Parse screenshots from PC and mobile devices

Application scenarios

  • Identify interface elements quickly in automated testing
  • Give an intelligent assistant more accurate interface information
  • Control a Windows 11 virtual machine with a chosen vision model

Model notes

  • YOLOv8 and Florence-2 are named as the deep learning models
  • An average latency as low as 0.6 seconds per frame on A100 is listed
  • A larger and cleaner icon description dataset is listed
  • Open-source code and detailed documentation are provided

Pros & cons

✓ What we like

  • Works with several LLMs such as OpenAI, DeepSeek and Qwen
  • Handles screenshots from PC and mobile devices

! What to watch out for

  • Parsing is described through YOLOv8 and Florence-2 models
  • Integration is listed with a chosen large language model

FAQ

What does it output?

A structured element list with interactive region positions and icon function descriptions.

How fast is parsing stated to be?

The features list an average latency as low as 0.6 seconds per frame on A100.

Last reviewed: 2026-11-15

More AI design tools tools

View all →

How we review