Skip to content

Humanity's Last Exam

A multimodal academic benchmark for language models

Visit official site

What Humanity's Last Exam is

The benchmark is described as a multimodal academic test built by experts from many institutions. It holds 3000 questions across more than 100 subjects with both public and private sets.

What you can do with it

  • Compare language models on academic questions
  • Find model weaknesses from test results
  • Use results as a reference for AI progress

Getting started

  • Visit the site for the test rules
  • Download the public dataset for an initial check
  • Submit results for accuracy and calibration error

Pros & cons

✓ What we like

  • A private test set is kept to limit overfitting
  • Accuracy and calibration error are reported

! What to watch out for

  • A private question set is withheld from public release
  • Evaluation is reported through the listed metrics

FAQ

How many questions are there?

The material states 3000 questions covering more than 100 subjects.

What is measured?

The material lists accuracy and calibration error.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review