Skip to content

MInference 1.0

Dynamic sparse attention for long-context pre-filling

Visit official site

What MInference 1.0 is

The method is described as a sparse computing approach that accelerates the pre-fill stage for long-context language models while keeping retrieval ability.

What you can do with it

  • Apply dynamic sparse attention to pre-fill
  • Split attention into A-shape, Vertical-Slash and Block-Sparse patterns
  • Use online approximation and tuned GPU kernels

Application scenarios

  • Retrieve and answer questions in QA tasks
  • Help write and understand code in programming tasks
  • Handle complex context in multi-hop QA

Pros & cons

✓ What we like

  • Speed gains of up to 10 times are stated for pre-fill
  • Retrieval ability is reported as preserved

! What to watch out for

  • Environment and dependencies must be configured from the docs
  • The demo is described for a single A100 with a LLaMA-style model

FAQ

Which patterns are named?

The material names A-shape, Vertical-Slash and Block-Sparse.

How long a prompt is discussed?

A 1M token pre-fill inference on a single A100 is described.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review