Skip to content

Ferret-UI

Understand and reason about mobile UI screens

Visit official site

What Ferret-UI is

The model is described as a new MLLLM designed to improve understanding of mobile UI screens. It handles tasks on UI screens with referring, grounding and reasoning, and shows dialogue ability on advanced tasks.

What you can do with it

  • Refer to, ground and reason about elements on UI screens
  • Enlarge fine details to recognise small objects with any-resolution technology
  • Hold dialogues on advanced tasks
  • Work on tasks framed for mobile UI screens

Application scenarios

  • Automate actions on iOS 18 app interfaces
  • Recognise and operate mobile app UI screens
  • Demonstrate dialogue ability on advanced tasks

Model notes

  • Task formulation and training sample collection are covered
  • Model architecture and dataset construction are covered
  • The model is built specifically for mobile UI screens

Pros & cons

✓ What we like

  • Enlarged detail views address small object recognition
  • Referring, grounding and reasoning are combined in one model

! What to watch out for

  • Recognition of small objects depends on the enlarged detail views
  • The documented tasks are framed around mobile UI screens

FAQ

What is any-resolution technology?

The material describes it as enlarging details to solve small object recognition on UI screens.

What abilities are named?

Referring, grounding, reasoning and dialogue ability on advanced tasks are named.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review