Skip to content

Cheating LLM Benchmarks

Probe LLM benchmarks with null models

Visit official site

What Cheating LLM Benchmarks is

The project is described as a study that uses null models to question the reliability of automated LLM benchmarks.

What you can do with it

  • Build a null model for a benchmark
  • Run experiments in notebooks
  • Compute win rates and errors

Getting started

  • Get the code from the GitHub page
  • Install Jupyter Notebook and AlpacaEval
  • Run the notebooks and evaluate outputs

Pros & cons

✓ What we like

  • Win rates and standard errors are computed
  • Experiment data can be reassessed

! What to watch out for

  • Focus is on automated benchmark tests
  • Setup needs Jupyter Notebook and AlpacaEval

FAQ

What are built?

Null models for automated language model benchmarks.

Which tool evaluates outputs?

AlpacaEval is named for evaluating outputs.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review