What Humanity's Last Exam is
The benchmark is described as a multimodal academic test built by experts from many institutions. It holds 3000 questions across more than 100 subjects with both public and private sets.
What you can do with it
- Compare language models on academic questions
- Find model weaknesses from test results
- Use results as a reference for AI progress
Getting started
- Visit the site for the test rules
- Download the public dataset for an initial check
- Submit results for accuracy and calibration error
Pros & cons
✓ What we like
- A private test set is kept to limit overfitting
- Accuracy and calibration error are reported
! What to watch out for
- A private question set is withheld from public release
- Evaluation is reported through the listed metrics
FAQ
How many questions are there?
The material states 3000 questions covering more than 100 subjects.
What is measured?
The material lists accuracy and calibration error.
Last reviewed: 2026-11-15
More LLM API platform tools
View all →-
302.AI Aggregated cloud platform that hosts hundreds of large models behind one API Developer tools API integration Multi model platform Model deployment LLM API platform -
360智脑开放平台 360 model API platform API integration Multi model platform Knowledge base Q&A Enterprise automation LLM API platform -
4o Image API Image generation API gateway Free tools Freemium tools Developer tools API integration LLM API platform -
AI-Flow No-code multi-model workflows Workflow automation Developer tools API integration Knowledge base Q&A LLM API platform -
AI/ML API Single gateway to many models Developer tools API integration Multimodal Multi model platform LLM API platform -
Amazon Bedrock AWS code suggestions with security scans Free tools Code assistant IDE App builder LLM API platform