How to Add Speaker Diarization to Whisper Transcriptions

Before you start

Whisper alone transcribes speech but does not identify who is speaking. Speaker diarization is a separate step that assigns labels such as Speaker 1 or Speaker 2 to segments of the transcript. The most common local method is to combine Whisper with another speech-processing model in a Python environment. This runs on your own machine, so the recording stays under your control instead of being uploaded to a third-party service.

Use a spoken recording with clearly separated voices. A meeting recorded through a microphone that picks up everyone at a similar input level works better than audio where one speaker is far from the input device. Check that the file is not heavily compressed, because clipping and excessive background noise make both transcription and speaker separation less reliable.

The process is easiest to reproduce when all audio uses the same sample rate. Audio from video calls, voice memos, or camera recordings can arrive with different rates, so plan to convert the file to one format before running the pipeline.

Step-by-step instructions

The procedure below describes a Python workflow. It uses Whisper for transcription and a separate diarization model for speaker boundaries. The speaker model identifies changes in voice characteristics, while Whisper supplies the words for those segments.

1. Set up a Python environment

Create a clean project folder and install the packages you need. The exact package names and versions can change, so inspect the current documentation for each tool before copying commands from an older guide.

python -m venv venv
source venv/bin/activate
pip install whisper
pip install pyannote.audio

Some diarization models require a separate access token or agreement before the model weights can be downloaded. Treat that as an account and licensing step, not as a sign that the transcription itself must be sent to a service.

2. Convert the audio to a consistent format

Use a single channel or a selected input device track. If the source is a video call recording that saves each participant as a separate track, decide whether to analyze the mixed output device track or one input at a time.

ffmpeg -i recording.m4a -ac 1 -ar 16000 recording_16k.wav

The command above produces mono PCM audio at a 16 kHz sample rate. This is only an input-preparation step; it does not promise that a particular model version accepts that exact rate. Match the value to the requirements of the specific tools you installed.

3. Transcribe with Whisper

import whisper

model = whisper.load_model('small.en')
result = model.transcribe('recording_16k.wav')
segments = result['segments']
print(segments)

The segment list contains text and start and end times. Keep this list in memory or write it to a temporary file. The next step aligns each Whisper segment with a speaker label.

4. Run speaker diarization

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained('pyannote/speaker-diarization')
diarization = pipeline('recording_16k.wav')

# Each item has start time, end time, and a speaker label.
for turn, _, speaker in diarization.itertracks(yield_label=True):
    print(start, end, speaker)

This creates speaker turns separately from the transcript. A speaker turn says when a voice changed and which label applies, but it does not know the words. The final step combines the two outputs by time overlap.

5. Match Whisper segments to speaker labels

Align the diarization turns with the Whisper segments. Where a Whisper segment overlaps mostly with one speaker turn, assign that speaker label. If a segment spans a speaker change, split it at the nearest turn boundary or assign the speaker with the largest overlap.

output = []
for seg in segments:
    speaker = None
    for turn, _, label in diarization.itertracks(yield_label=True):
        if turn.start <= seg['start'] < turn.end:
            speaker = label
            break
    output.append({'start': seg['start'], 'end': seg['end'], 'speaker': speaker, 'text': seg['text']})
print(output)

The example is deliberately simple. Boundary mismatches are common because Whisper and the diarization model can disagree about where a sentence starts or ends. A refinement step can reduce mismatches, but start with the direct overlap method so the result remains easy to inspect and reverse.

Check the output

Open the final transcript and confirm that every Whisper segment has a speaker label. Look for two expected limitations: the labels are usually generic, such as Speaker 00 and Speaker 01, and the diarization timeline may cut a sentence a few tenths of a second before or after the words.

Play short excerpts around speaker changes to hear whether the label switch matches the actual voice change. If the recording has one person nearer the microphone and another farther away, expect the quieter speaker to be merged into the louder speaker or missed. This is not necessarily a software failure; it can reflect the input level difference in the source file.

When the source is a video call with separate audio tracks, compare the diarization output with the track names if the recording format exposes them. That gives a rough reference point without claiming any file metadata is preserved by every application.

Common problems and next steps

Speakers keep getting merged. This often happens when the voices are similar, when speakers overlap, or when one person is much quieter. Reduce unnecessary background noise, keep the sample rate consistent, and try recordings where people speak one at a time. Increasing separation at the input stage is more effective than changing only a model parameter.

One speaker is labeled as several. A single voice that moves around the room or changes distance from the microphone can look like multiple speakers to the model. Conversation-heavy audio with laughter, coughs, or phone audio can also split a speaker. Check the boundaries and merge adjacent labels manually if the words clearly belong to the same person.

The transcript text is correct but the speaker assignment is wrong. This is a diarization error, not a Whisper transcription error. Isolate the part of the script that assigns labels and test it against known short clips where you already know the speaker order.

No diarization model can be downloaded. Verify the authentication or license requirement for the chosen model. This is the point at which an otherwise local workflow may require an online account for model access. Keep the audio file separate from that account unless the specific service states that the file is uploaded.

For a less code-heavy path, look for a maintained open-source project that combines Whisper with speaker diarization. Project names and command-line options change, so use the current repository documentation rather than relying on an old installation snippet. The same cautions apply to privacy: check whether the tool downloads models only or sends audio elsewhere.