How to Use SpeechBrain for Speaker Diarization

Before you start

SpeechBrain is an open-source toolkit that can run speaker diarization on a local machine. In this workflow the audio stays on your computer unless you change the code to upload or store it somewhere else. You need a Python environment with SpeechBrain installed, an audio file such as a meeting or interview recording, and enough disk space for extracted features and the diarization output. The pipeline works best with spoken audio at a clear input level and a sample rate supported by the pretrained model, typically 16 kHz mono for the standard SpeechBrain diarization pipeline. Very low input level or strong background noise can reduce the reliability of speaker boundaries.

Step-by-step instructions

Use these steps as a reversible local test. You can delete the generated RTTM file and the output folder without changing the source recording.

python -m venv speechbrain-env
# On Linux or macOS:
source speechbrain-env/bin/activate
# On Windows:
speechbrain-env\Scripts\activate

python -m pip install speechbrain

Create a virtual environment and install SpeechBrain. Using a project-local environment keeps the installed dependencies separate from the system Python and makes the test easy to remove.

Prepare a short recording of two or more speakers. Put the input audio file in your working folder, for example meeting.wav. If the recording is stereo, convert it to mono first; many diarization models expect mono input.

from speechbrain.inference.speaker import Speech_Encoder
from speechbrain.inference.diarization import Speech_Acc_Diarization

diarization = Speech_Acc_Diarization.from_hparams(
    source="speechbrain/spkrec-ecapa-voxceleb",
    savedir="pretrained_models/spkrec-ecapa-voxceleb",
)

segments = diarization.diarize_file("meeting.wav")
print(segments)

Run the pretrained diarization pipeline from a script. In this example the savedir is relative to the current folder, so model files are downloaded into your project rather than the global environment. The printed result contains time ranges with speaker labels such as spk1 and spk2.

Save the output if you need to review it later. You can write the returned segments to an RTTM file or a simple text table. Keep the original naming from SpeechBrain so the speaker IDs match the printed result.

If you want to change the expected speaker count, check whether the loaded pipeline or recipe exposes a max_speakers or num_speakers option in your installed version. SpeechBrain releases can differ, so inspect the pretrained model card for the version you are using before changing the default behavior.

Check the output

Compare the returned segments with the recording. Speaker diarization answers who spoke when, but the labels are not real names; they are assigned during the run and may change between runs. Verify that each segment has a start time, end time, and speaker ID, and check whether the speaker changes occur near pauses or turn-taking points. A short clean recording should show clearly separated segments, while overlapping speech and heavy background noise can make the result less accurate.

Common problems and next steps

If the recording has constant background noise, clean it before diarization or use an audio editor to reduce noise and normalize the input level. Avoid clipping while raising a quiet signal, because distortion can create false speaker changes.

If the model does not find enough speakers, check the input sample rate. Many SpeechBrain diarization pipelines require 16 kHz mono; resample other input devices or recordings before inference. Also confirm that speech is present in the file and not only unvoiced sections such as silence or music.

If installed versions conflict, isolate the test in a new virtual environment and pin the SpeechBrain version shown in the pretrained model documentation. Reinstall only the local environment if a dependency error appears, rather than changing the system Python.

For a meeting with many speakers or variable microphone levels, the standard pipeline may need more tuning. Keep the same input file while testing one change at a time: mono conversion, sample rate, noise reduction, and any available speaker-count setting. Remove generated model and output folders when you no longer need the local test results.