Skip to content

Turtle Benchmark

A cheat-resistant benchmark built on Turtle Soup

Visit official site

What Turtle Benchmark is

The tool is a benchmark built on the Turtle Soup game to test logical reasoning and context understanding of LLMs. It aims for objective, quantifiable results without background knowledge.

What you can do with it

  • Compare LLMs on specific reasoning tasks
  • Test whether a model answers user questions
  • Use it as a teaching tool for evaluation

Getting started

  • Enter the project directory
  • Rename .env.example to .env and add a key
  • Run python evaluate.py for a 2-shot test

Pros & cons

✓ What we like

  • Runs through simple command line operations
  • Results are described as objective and unbiased

! What to watch out for

  • An API key is added to a .env file
  • A 2-shot command is the default evaluation

FAQ

What does it measure?

Logical reasoning and context understanding of large language models are named.

How much data is included?

32 unique stories and 1,537 human-annotated labels are stated.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review