Skip to content

Magma

A multimodal foundation model for vision, language and action

Visit official site

What magma is

The model is described as a multimodal foundation model from Microsoft Research that joins vision, language and action. It is pretrained on large vision-language data for planning tasks.

What you can do with it

  • Navigate a web or mobile UI from an instruction
  • Plan robot pick-and-place actions from visual input
  • Answer questions about video content

Key features

  • Set-of-Mark and Trace-of-Mark techniques
  • Zero-shot adaptation to unseen tasks
  • Multimodal input of image, video and language

Getting started

  • Get the model and code from the official site or GitHub
  • Choose a pretrained model version for the task
  • Fine-tune with a small amount of labeled data if needed
  • Pass images, video or text instructions into the model

Pros & cons

✓ What we like

  • Spatial reasoning and action planning
  • Video question answering is supported

! What to watch out for

  • Some tasks use a small amount of labeled data for fine-tuning
  • A pretrained model version must be chosen per task

FAQ

Who released it?

The material names Microsoft Research.

Which tasks are named?

UI navigation, robot manipulation and video QA.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review