Skip to content

FlashAttention

Exact attention with lower memory use

Visit official site

What FlashAttention is

The library is an open-source attention mechanism for Transformer models with an IO-aware design.

What you can do with it

  • Run exact attention with less memory
  • Use causal or non-causal attention
  • Apply sliding window local attention

Pros & cons

✓ What we like

  • Causal and non-causal attention are supported
  • Sliding window attention is included

! What to watch out for

  • CUDA 11.6 and PyTorch 1.12 are required
  • Install runs through setup.py

FAQ

Which GPUs are named?

Ampere, Ada and Hopper.

What data types?

FP16 and BF16.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review