Speaker Diarization with PLDA I-Vector Scoring and Unsupervised Calibration Explained

What is speaker diarization with PLDA i-vector scoring and unsupervised calibration?

Speaker diarization is the process of answering the question 'who spoke when' in a multi-speaker audio recording. It does not identify people by name, transcribe what they said, or separate overlapping audio sources. Instead, it segments the recording and assigns speaker-turn labels such as Speaker A, Speaker B, and Speaker C to audio regions.

PLDA i-vector scoring is a widely used approach for measuring whether two speech segments are likely to come from the same anonymous speaker. Unsupervised calibration adjusts the scoring output so that scores can be interpreted more consistently without requiring manually labeled training data for the specific recording.

How i-vectors represent speech segments

An i-vector is a compact numerical representation of a speech segment. It captures speaker-related variation while reducing the influence of the spoken words and channel effects. In diarization, short segments of audio are converted into i-vectors so their similarity can be compared mathematically.

Because i-vectors are created from audio, the quality of the comparison is sensitive to segment length, noise, reverberation, and the amount of speech present in each segment. Short or silence-heavy segments generally produce less stable representations.

The role of PLDA scoring

Probabilistic linear discriminant analysis, or PLDA, is a generative model that compares two i-vectors and produces a likelihood score for the hypothesis that they belong to the same speaker. In diarization, clustering algorithms use these scores to decide whether neighboring or candidate segments should be merged under the same speaker label.

PLDA scoring is designed to separate speaker variability from other variation in the i-vector space. A higher PLDA score indicates stronger support for the same-speaker hypothesis, but the raw score alone does not provide a direct probability without calibration.

Why unsupervised calibration matters

Calibration maps raw scores to more interpretable values, often log-likelihood ratios or calibrated probabilities. Supervised calibration requires labeled examples of same-speaker and different-speaker pairs. Unsupervised calibration instead estimates the necessary parameters from the unlabeled recording itself.

This is useful for real-world audio where speaker identities are unknown in advance. It allows the diarization system to adapt scoring behavior to the characteristics of a particular recording without needing manually annotated turns.

How speaker turns are labeled

The typical pipeline begins with segmentation, followed by i-vector extraction, PLDA scoring, calibration, and clustering. The final result is a sequence of time intervals with associated speaker labels.

• Segmentation divides the audio into short regions that are likely to contain speech from one speaker.
• I-vector extraction converts each region into a speaker-discriminative vector.
• PLDA scoring measures pairwise similarity between regions or against cluster representations.
• Calibration adjusts scores to improve decision consistency.
• Clustering groups regions and assigns anonymous labels such as Speaker A and Speaker B.

What speaker diarization does not do

Speaker diarization using this pipeline does not reveal a person's identity. It produces anonymous speaker-turn labels limited to the current recording. The same label is not guaranteed to refer to the same person across different recordings without an additional speaker recognition or identity comparison step.

This approach also differs from speech transcription, which produces words, and source separation, which isolates audio streams. A diarization label can mark when a speaker is active, but it does not recover isolated speech audio or the content of the speech.

Practical limitations to consider

The documented behavior of PLDA i-vector diarization depends on the quality of segmentation, i-vector estimation, and clustering. Overlapping speech, rapid speaker changes, and short utterances complicate accurate turn boundaries. Unsupervised calibration can reduce score mismatch, but it requires enough audio to estimate parameters reliably and is not a substitute for a well-matched PLDA model.

This article describes the general mechanism of these components rather than independently measured accuracy figures. Reported performance varies by recording conditions and system configuration. Evaluation should therefore measure the chosen implementation on audio conditions similar to the intended use, with attention to whether labels, boundaries, and speaker counts match the actual recording.