Skip to content

T-MAC

A kernel library for low-bit LLM inference on CPUs

Visit official site
Category AI models

What T-MAC is

The library is described as a kernel set that uses lookup tables to speed up low-bit LLM inference on CPUs without dequantisation.

What you can do with it

  • Run mixed-precision matrix multiplication directly
  • Work with 1 to 4 bit quantised Llama models
  • Deploy natively on Windows ARM
  • Lower CPU core use and power draw

Getting started

  • Install Python 3.8 for TVM
  • Add cmake 3.22 or newer and dependencies
  • Install T-MAC with pip in a virtual environment
  • Run the provided scripts or integrate with llama.cpp

Model notes

  • Stated throughput is 20 tokens per second on one core
  • Four cores are stated at 48 tokens per second
  • The source compares it with llama.cpp at 4 to 5 times

Pros & cons

✓ What we like

  • The material states 4 to 5 times the speed of llama.cpp
  • CPU core use and power draw are reduced

! What to watch out for

  • Python 3.8 and cmake 3.22 or newer are needed
  • Setup uses a virtual environment and pip

FAQ

What speeds are stated?

The material states 20 tokens per second on a single core and 48 on four cores.

Which platforms are named?

The material names OSX, Linux, Windows ARM and Intel CPUs.

Last reviewed: 2026-11-15

More LLM API platform tools

View all →

How we review