Skip to content

MMStar

A benchmark for visual language models

Visit official site

What MMStar is

The benchmark is described as a set for evaluating the multimodal ability of large vision language models.

What you can do with it

  • Evaluate models on many tasks
  • Detect leakage problems
  • Guide model improvements

Benchmark design

  • Review samples by hand
  • Cover six core abilities
  • Add leakage and gain metrics

Pros & cons

✓ What we like

  • Two new metrics are proposed
  • Sixteen models are stated as benchmarked

! What to watch out for

  • Samples are stated to require visual dependence
  • Leakage metrics rest on the sample design

FAQ

How many samples?

The material states 1500 curated samples.

What is added beyond accuracy?

The material states metrics for data leakage and multimodal gain.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review