Skip to content

FlashInfer

A GPU kernel library built for LLM serving

Visit official site

What FlashInfer is

The library is described as a GPU kernel collection built for large language model serving.

What you can do with it

  • Speed up LLM inference with efficient kernels
  • Tune attention for machine translation
  • Generate text in a question answering system

Getting started

  • Install a prebuilt wheel matched to the CUDA version
  • Import the FlashInfer module in a Python script
  • Call the FlashInfer API for attention computation

Pros & cons

✓ What we like

  • Works with CUDAGraph and torch.compile
  • Fused Top-P, Top-K and Min-P sampling kernels are listed

! What to watch out for

  • Named APIs are PyTorch, TVM and C++
  • Custom attention relies on JIT compilation

FAQ

Which APIs are named?

PyTorch, TVM and C++ APIs are listed.

What sampling kernels are included?

Top-P, Top-K and Min-P fused sampling kernels are named.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review