event_fscore_per_class()

audmetric.event_fscore_per_class(truth, prediction, labels=None, *, zero_division=0.0, propagate_nans=False, onset_tolerance=0.0, offset_tolerance=0.0, duration_tolerance=None)[source]

Event-based F-score per class.

fscorek=true positivektrue positivek+12(false positivek+false negativek)\text{fscore}_k = \frac{\text{true positive}_k} {\text{true positive}_k + \frac{1}{2} (\text{false positive}_k + \text{false negative}_k)}

This metric compares not only the labels of prediction and ground truth, but also the time windows they occur in.

Each event is considered to be correctly identified if the predicted label is the same as the ground truth label, and if the onset is within the given onset_tolerance (in seconds) and the offset is within the given offset_tolerance (in seconds). Additionally to the offset_tolerance, one can also specify the duration_tolerance, to ensure that the offset occurs within a certain proportion of the reference event duration. If a prediction fulfills the duration_tolerance but not the offset_tolerance (or vice versa), it is still considered to be an overlapping segment. 1

1

Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. Metrics for polyphonic sound event detection. Applied Sciences, 2016. doi:10.3390/app6060162.

Parameters

  • truth (Series) – ground truth values/classes

  • prediction (Series) – predicted values/classes

  • labels (Optional[Sequence[object]]) – included labels in preferred ordering. If no labels are supplied, they will be inferred from {prediction,truth}\{\text{prediction}, \text{truth}\} and ordered alphabetically.

  • zero_division (float) – set the value to return when there is a zero division

  • propagate_nans (bool) – whether to set the F-score to NaN when recall or precision are NaN. If False, the F-score is only set to NaN when both recall and precision are NaN

  • onset_tolerance (Optional[float]) – the onset tolerance in seconds. If the predicted segment’s onset does not occur within this time window compared to the ground truth segment’s onset, it is not considered correct

  • offset_tolerance (Optional[float]) – the offset tolerance in seconds. If the predicted segment’s offset does not occur within this time window compared to the ground truth segment’s offset, it is not considered correct, unless the duration_tolerance is specified and fulfilled

  • duration_tolerance (Optional[float]) – the duration tolerance as a measure of proportion of the ground truth segment’s total duration. If the offset_tolerance is not fulfilled, and the predicted segment’s offset does not occur within this time window compared to the ground truth segment’s offset, it is not considered correct

Return type

dict[str, float]

Returns

dictionary with label as key and F-score as value

Raises

ValueError – if truth or prediction do not have a segmented index conform to audformat

Examples

>>> import pandas as pd
>>> import audformat
>>> truth = pd.Series(
...     index=audformat.segmented_index(
...         files=["f1.wav", "f1.wav"],
...         starts=[0.0, 0.1],
...         ends=[0.1, 0.2],
...     ),
...     data=["a", "b"],
... )
>>> prediction = pd.Series(
...     index=audformat.segmented_index(
...         files=["f1.wav", "f1.wav"],
...         starts=[0, 0.09],
...         ends=[0.1, 0.2],
...     ),
...     data=["a", "a"],
... )
>>> event_fscore_per_class(
...     truth, prediction, onset_tolerance=0.02, offset_tolerance=0.02
... )
{'a': 0.6666666666666666, 'b': 0.0}