Skip to content

lmsys

Open models, arenas and evaluation from LMSYS Org

Visit official site

What LMSYS is

The organisation is described as working to democratise large model technology and the systems infrastructure around it. Its work spans chatbots, evaluation arenas, datasets, runtimes and open platforms.

What you can do with it

  • Develop large models and their systems infrastructure
  • Train, serve and evaluate LLM-based chatbots with FastChat
  • Run complex LLM programs with the SGLang interface and runtime
  • Evaluate models at scale through crowdsourced battles

Projects and resources

  • Vicuna is described at 7B, 13B and 33B scales, with the material citing 90% of ChatGPT quality
  • Chatbot Arena combines crowdsourcing with an Elo rating system
  • LMSYS-Chat-1M is a large real-world LLM conversation dataset
  • MT-Bench is a set of challenging multi-turn open-ended questions

Example uses

  • Build a high quality chatbot application with Vicuna
  • Hold an LLM evaluation competition on Chatbot Arena
  • Train and serve a chatbot with FastChat

Pros & cons

✓ What we like

  • Models, datasets and evaluation tools in one organisation
  • FastChat covers training, serving and evaluation

! What to watch out for

  • Arena evaluation is based on crowdsourcing and Elo ratings
  • MT-Bench relies on multi-turn open-ended questions

FAQ

What is Chatbot Arena?

The material describes large-scale, gamified LLM evaluation through crowdsourcing and an Elo rating system.

Which projects are listed?

Vicuna, Chatbot Arena, SGLang, LMSYS-Chat-1M, FastChat and MT-Bench are named.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review