Skip to content

ReDrafter

Speculative decoding for faster LLM inference on NVIDIA GPUs

Visit official site

What ReDrafter is

The method is described as speculative decoding that combines an RNN draft model with dynamic tree attention to speed up token generation on NVIDIA GPUs.

What you can do with it

  • Run speculative decoding on supported models
  • Cut the delay a user experiences
  • Reduce GPU usage and energy consumption
  • Work with several open-source LLMs

Getting started

  • Install and configure NVIDIA TensorRT-LLM
  • Get the open-source code from GitHub
  • Follow the docs to integrate the method
  • Pick an open-source LLM for testing

Pros & cons

✓ What we like

  • The method is described as reducing user latency
  • Several open-source LLMs are supported

! What to watch out for

  • A TensorRT-LLM environment must be set up first
  • Integration follows the project documentation on GitHub

FAQ

Who developed it?

The material states Apple machine learning research developed it with NVIDIA.

What speed figures are given?

The material states up to 3.5 tokens per generation step and a 2.7x lift in tokens per second.

Last reviewed: 2026-11-15

More AI coding tools tools

View all →

How we review