Skip to content

FlexHeadFA

Flexible head dimensions for exact attention

Visit official site

What FlexHeadFA is

The model is described as a FlashAttention-based improvement that delivers fast, memory-efficient exact attention with flexible head dimensions.

What you can do with it

  • Use all FlashAttention-2 and FlashAttention-3 configurations
  • Combine QKHeadDim and VHeadDim settings
  • Set unequal query, key and value head counts
  • Generate code for head dimensions not preset
  • Run efficient forward and backward passes

Getting started

  • Install the package with pip
  • Replace flash_attn with flex_head_fa in code
  • Configure QKHeadDim and VHeadDim
  • Call the attention function for a forward pass

Application scenarios

  • Use a 32 and 64 head dimension pair on an A100 GPU
  • Tune head dimensions for a specific task
  • Handle long sequence data with less memory

Pros & cons

✓ What we like

  • Unequal query, key and value head counts are supported
  • Generated code covers head dimensions not preset

! What to watch out for

  • Setup replaces flash_attn with flex_head_fa in code
  • Head dimensions must be configured with parameters

FAQ

Which attention versions are supported?

FlashAttention-2 and FlashAttention-3 configurations are listed.

What dimensions can be set?

QKHeadDim and VHeadDim are named as configurable parameters.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review