Skip to content

whisper-diarization

Transcribe speech and label each speaker in audio

Visit official site

What whisper-diarization is

The project is described as an open-source combination of Whisper ASR, voice activity detection and speaker embeddings, extracting voiced parts of audio to improve speaker embedding accuracy. The tool produces transcripts with Whisper, corrects timestamps with WhisperX, and links word-level speaker detection to those timestamps.

What you can do with it

  • Generate high-quality speech transcripts with Whisper ASR
  • Exclude silence with voice activity detection
  • Identify speakers with speaker embeddings
  • Correct and align timestamps with WhisperX

How the pipeline works

  • MarbleNet performs VAD and segmentation to drop silent parts
  • TitaNet extracts speaker embeddings for each segment
  • Results are linked to WhisperX timestamps to detect the speaker of each word
  • A punctuation model realigns the transcript to compensate for small time shifts

Getting started

  • Install FFMPEG and Cython as prerequisites
  • Clone or download the code base
  • Adjust WhisperX and NeMo parameters in diarize.py and helpers.py
  • Run the model with a command line tool, passing parameters and an audio file name

Pros & cons

✓ What we like

  • Batch inference is supported for more efficient processing
  • Voiced parts are extracted before speaker embedding to raise accuracy

! What to watch out for

  • FFMPEG and Cython are listed as prerequisites
  • WhisperX and NeMo parameters need editing in the project files

FAQ

What components are involved?

The material names Whisper, WhisperX, MarbleNet, TitaNet, a punctuation model and NeMo parameters.

What must be installed first?

The material lists FFMPEG and Cython as prerequisites, then the code base.

Last reviewed: 2026-11-15

More AI audio tools tools

View all →

How we review