diarization_error_rate()

audmetric.diarization_error_rate(truth, prediction, *, individual_file_mapping=False, num_workers=1, multiprocessing=False)

Diarization error rate.

$$\text{DER} = \frac{\text{confusion}+\text{false alarm}+\text{miss}} {\text{total}}$$

where $\text{confusion}$ is the total confusion duration, $\text{false alarm}$ is the total duration of predictions without an overlapping ground truth, $\text{miss}$ is the total duration of ground truth without an overlapping prediction, and $\text{total}$ is the total duration of ground truth segments.

The diarization error rate can used when the labels are not known by the prediction model, e.g. for the task of speaker diarization on unknown speakers.

This metric is computed the same way as audmetric.identification_error_rate(), but first creates a one-to-one mapping between truth and prediction labels.

This implementation uses the ‘greedy’ method to compute the one-to-one mapping between truth and predicted labels. This method is faster than other implementations that optimize the confusion term, but may slightly over-estimate the diarization error rate. 1

1

Hervé Bredin. pyannote.metrics: a toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems. In Interspeech 2017, 18th Annual Conference of the International Speech Communication Association. Stockholm, Sweden, August 2017.

Parameters

• truth (Series) – ground truth labels with a segmented index conform to audformat

• prediction (Series) – predicted labels with a segmented index conform to audformat

• individual_file_mapping (bool) – whether to create the mapping between truth and prediction labels individually for each file. If False, all segments are taken into account to compute the mapping

• num_workers (int) – number of threads or 1 for sequential processing

• multiprocessing (bool) – use multiprocessing instead of multithreading

Return type

float

Returns

diarization error rate

Raises

ValueError – if truth or prediction do not have a segmented index conform to audformat

Examples

>>> import pandas as pd
>>> import audformat
>>> truth = pd.Series(
...     index=audformat.segmented_index(
...         files=["f1.wav", "f1.wav"],
...         starts=[0.0, 0.1],
...         ends=[0.1, 0.2],
...     ),
...     data=["a", "b"],
... )
>>> prediction = pd.Series(
...     index=audformat.segmented_index(
...         files=["f1.wav", "f1.wav", "f1.wav"],
...         starts=[0, 0.1, 0.1],
...         ends=[0.1, 0.15, 0.2],
...     ),
...     data=["0", "1", "0"],
... )
>>> diarization_error_rate(truth, prediction)
0.5