Skip to content

torchao

Quantise and sparsify PyTorch models

Visit official site

What torchao is

torchao is described as a PyTorch library for custom data types that quantises and sparsifies weights, gradients, optimisers and activations.

What you can do with it

  • Apply post training quantisation or QAT
  • Quantise a KV cache for long context inference
  • Train with scaled float8 or 2:4 sparsity

Example uses

  • Quantise an image segmentation model for faster inference
  • Apply QAT to a language model
  • Cut memory use in diffusion inference with sparsity

Pros & cons

✓ What we like

  • Compatible with torch.compile() and FSDP2
  • A developer API allows custom quantisation algorithms

! What to watch out for

  • Quantisation strategy is chosen per model
  • Local work starts with installing the library

FAQ

What does it quantise?

Weights, gradients, optimisers and activations are named.

Is long context covered?

KV cache quantisation is listed for long context inference.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review