Usage

The basic idea of audinterface is to provide easy and standardized interfaces to apply a machine learning model, or other digital signal processing algorithms to audio files. The only prerequisite is the algorithm provides a callable that takes at least the signal as a numpy.ndarray and the sampling rate as input.

The interface can then apply the algorithm on a list of files, a folder, or an index conform to the audformat database specification. Results are always returned containing a segmented index. In the following we load three files from the emodb database and define a list of files, a folder, and an index.

import audb
import os

media = [
    "wav/03a01Fa.wav",
    "wav/03a01Nc.wav",
    "wav/16b10Wb.wav",
]
db = audb.load(
    "emodb",
    version="1.3.0",
    media=media,
    verbose=False,
)

files = list(db.files)
folder = os.path.dirname(files[0])
index = db["emotion"].index

Processing interface

Let’s assume we want to calculate the root mean square (RMS) value in dB. We first define the function and create an interface for it using audinterface.Process.

import audinterface
import numpy as np

def rms(signal, sampling_rate):
    return 20 * np.log10(np.sqrt(np.mean(signal ** 2)))

interface = audinterface.Process(process_func=rms)

The following three commands apply the algorithm and all return the same result as a pandas.Series.

y = interface.process_files(files)
y = interface.process_folder(folder)
y = interface.process_index(index)
y

file	start	end
.../03a01Fa.wav	0 days	0 days 00:00:01.898250	-21.690142
.../03a01Nc.wav	0 days	0 days 00:00:01.611250	-18.040704
.../16b10Wb.wav	0 days	0 days 00:00:02.522499999	-20.394533

To calculate RMS with a sliding window, we create a new interface and set a window and hop duration.

interface = audinterface.Process(
    process_func=rms,
    win_dur=1.0,
    hop_dur=0.5,
)
y = interface.process_files(files)
y

file	start	end
.../03a01Fa.wav	0 days 00:00:00	0 days 00:00:01	-20.165248
	0 days 00:00:00.500000	0 days 00:00:01.500000	-23.472970
.../03a01Nc.wav	0 days 00:00:00	0 days 00:00:01	-16.386614
...	...	...	...
.../16b10Wb.wav	0 days 00:00:00.500000	0 days 00:00:01.500000	-20.233054
	0 days 00:00:01	0 days 00:00:02	-18.856522
	0 days 00:00:01.500000	0 days 00:00:02.500000	-20.403574

Feature interface

When the result of the processing function has multiple dimensions it is recommended to use audinterface.Feature, which returns a pandas.DataFrame and assigns names to the dimensions/features.

def features(signal, sampling_rate):
    return [signal.mean(), signal.std()]

interface = audinterface.Feature(
    ["mean", "std"],
    process_func=features,
)

df = interface.process_index(index)
df

			mean	std
file	start	end
.../03a01Fa.wav	0 days	0 days 00:00:01.898250	-0.000311	0.082317
.../03a01Nc.wav	0 days	0 days 00:00:01.611250	-0.000312	0.125304
.../16b10Wb.wav	0 days	0 days 00:00:02.522499999	-0.000464	0.095558

To calculate features with a sliding window, we create a new interface and set a window and hop duration. By setting process_func_applies_sliding_window=False the windowing is automatically handled and single frames are passed on to the processing function.

interface = audinterface.Feature(
    ["mean", "std"],
    process_func=features,
    process_func_applies_sliding_window=False,
    win_dur=1.0,
    hop_dur=0.5,
)
df = interface.process_files(files)
df

			mean	std
file	start	end
.../03a01Fa.wav	0 days 00:00:00	0 days 00:00:01	-0.000329	0.098115
	0 days 00:00:00.500000	0 days 00:00:01.500000	-0.000285	0.067042
.../03a01Nc.wav	0 days 00:00:00	0 days 00:00:01	0.000039	0.151590
...	...	...	...	...
.../16b10Wb.wav	0 days 00:00:00.500000	0 days 00:00:01.500000	-0.000461	0.097351
	0 days 00:00:01	0 days 00:00:02	-0.000469	0.114070
	0 days 00:00:01.500000	0 days 00:00:02.500000	-0.000447	0.095459

Feature interface for multi-channel input

By default, an interface will process the first channel of an audio signal. We can prove this by running the previous interface on the following multi-channel signal.

import audiofile

signal, sampling_rate = audiofile.read(
    files[0],
    always_2d=True,
)
signal_multi_channel = np.concatenate(
    [
        signal,
        signal * 0,
        signal - 0.5,
        signal + 0.5,
    ],
)
signal_multi_channel.shape

(4, 30372)

df = interface.process_signal(
    signal_multi_channel,
    sampling_rate,
)
df

		mean	std
start	end
0 days 00:00:00	0 days 00:00:01	-0.000329	0.098115
0 days 00:00:00.500000	0 days 00:00:01.500000	-0.000285	0.067042

To process the second and fourth channel, we create a new interface and set channels=[1, 3]. To reuse our processing function, we additionally set process_func_is_mono=True. This will apply the function on each channel and combine the results. Otherwise, the processing function must return an array with the correct number of channels (here 2).

interface_multi_channel = audinterface.Feature(
    ["mean", "std"],
    process_func=features,
    process_func_is_mono=True,
    process_func_applies_sliding_window=False,
    win_dur=1.0,
    hop_dur=0.5,
    channels=[1, 3],
)

df = interface_multi_channel.process_signal(
    signal_multi_channel,
    sampling_rate,
)
df

		1		3
		mean	std	mean	std
start	end
0 days 00:00:00	0 days 00:00:01	0.0	0.0	0.499671	0.098115
0 days 00:00:00.500000	0 days 00:00:01.500000	0.0	0.0	0.499715	0.067042

We can access the features of a specific channel by its index.

df[3]

		mean	std
start	end
0 days 00:00:00	0 days 00:00:01	0.499671	0.098115
0 days 00:00:00.500000	0 days 00:00:01.500000	0.499715	0.067042

Feature interface for external function

If we interface a function from an external library that already applies a sliding window, we again specfiy the win_dur and hop_dur arguments. However, by setting process_func_applies_sliding_window=True we still request that the whole signal is passed on. Now, the processing function is responsible for extracting the features in a framewise manner and returning the values in the correct shape, namely (num_channels, num_features, num_frames), whereas the first dimension is optionally.

import librosa

def features(signal, sampling_rate, win_dur, hop_dur, n_mfcc):
    hop_length = int(hop_dur * sampling_rate)
    win_length = int(win_dur * sampling_rate)
    mfcc = librosa.feature.mfcc(
        y=signal,
        sr=sampling_rate,
        n_mfcc=13,
        hop_length=hop_length,
        win_length=win_length,
    )
    return mfcc

n_mfcc = 13
interface = audinterface.Feature(
    [f"mfcc-{idx}" for idx in range(n_mfcc)],
    process_func=features,
    process_func_args={"n_mfcc": n_mfcc},  # "win_dur" and "hop_dur" can be omitted
    process_func_applies_sliding_window=True,
    win_dur=0.02,
    hop_dur=0.01,
)
df = interface.process_index(index)
df

			mfcc-0	mfcc-1	...	mfcc-11	mfcc-12
file	start	end
.../03a01Fa.wav	0 days 00:00:00	0 days 00:00:00.020000	-611.993286	47.627602	...	10.659678	1.151390
	0 days 00:00:00.010000	0 days 00:00:00.030000	-668.175842	35.650566	...	13.644274	14.068543
	0 days 00:00:00.020000	0 days 00:00:00.040000	-664.612793	43.068939	...	5.081633	7.949757
...	...	...	...	...	...	...	...
.../16b10Wb.wav	0 days 00:00:02.500000	0 days 00:00:02.520000	-644.156494	34.024551	...	2.472639	7.411011
	0 days 00:00:02.510000	0 days 00:00:02.530000	-618.545898	44.741577	...	12.690891	17.645359
	0 days 00:00:02.520000	0 days 00:00:02.540000	-666.805237	19.845566	...	4.124321	3.711080

Serializable feature interface

To use a feature extractor as an input transform of a machine learning model it is recommend to provide it in a serializable way so it can be stored as part of the model. One example of such a feature extractor is opensmile.Smile.

To create such a feature extractor, we create a class that inherits from audinterface.Feature and audobject.Object.

import audobject

class MeanStd(audinterface.Feature, audobject.Object):

    def __init__(self):
        super().__init__(
            ["mean", "std"],
            process_func=self.features,
        )

    def features(self, signal, sampling_rate):
        return [signal.mean(), signal.std()]

fex = MeanStd()
df = fex.process_index(index)
df

			mean	std
file	start	end
.../03a01Fa.wav	0 days	0 days 00:00:01.898250	-0.000311	0.082317
.../03a01Nc.wav	0 days	0 days 00:00:01.611250	-0.000312	0.125304
.../16b10Wb.wav	0 days	0 days 00:00:02.522499999	-0.000464	0.095558

The advantage of the feature extraction object is that we can save it to a YAML file and re-instantiate it from there.

fex.to_yaml("mean-std.yaml")
fex2 = audobject.from_yaml("mean-std.yaml")
df = fex2.process_index(index)
df

			mean	std
file	start	end
.../03a01Fa.wav	0 days	0 days 00:00:01.898250	-0.000311	0.082317
.../03a01Nc.wav	0 days	0 days 00:00:01.611250	-0.000312	0.125304
.../16b10Wb.wav	0 days	0 days 00:00:02.522499999	-0.000464	0.095558

Segmentation interface

When the result of the processing function is an index it is recommended to use audinterface.Segment, which returns a segmented index conform to audformat. An example for such a processing function would be a voice activity detection algorithm.

import auditok

def segments(signal, sampling_rate):

    # Convert floating point array to 16bit PCM little-endian
    ints = (signal[0, :] * 32767).astype(np.int16)
    little_endian = ints.astype("<u2")
    signal = little_endian.tobytes()

    regions = auditok.split(
        signal,
        sampling_rate=sampling_rate,
        sample_width=2,
        channels=1,
        min_dur=0.2,
        energy_threshold=70,
    )
    index = pd.MultiIndex.from_tuples(
        [
            (
                pd.Timedelta(region.meta.start, unit="s"),
                pd.Timedelta(region.meta.end, unit="s"),
            )
            for region in regions
        ],
        names=["start", "end"],
    )
    return index

interface = audinterface.Segment(process_func=segments)
idx = interface.process_file(files[0])
idx

file	start	end
.../03a01Fa.wav	0 days 00:00:00.150000	0 days 00:00:00.700000
	0 days 00:00:00.900000	0 days 00:00:01.600000

Special processing function arguments

There are some special arguments to the processing function, which will be automatically set if they are not specified in process_func_args:

argument	value
idx	running index
file	file path
root	root folder

The following processing function returns the values of "idx" and "file".

def special_args(signal, sampling_rate, idx, file):
    return idx, os.path.basename(file)

interface = audinterface.Process(process_func=special_args)
y = interface.process_files(files)
y

file	start	end
.../03a01Fa.wav	0 days	0 days 00:00:01.898250	(0, 03a01Fa.wav)
.../03a01Nc.wav	0 days	0 days 00:00:01.611250	(1, 03a01Nc.wav)
.../16b10Wb.wav	0 days	0 days 00:00:02.522499999	(2, 16b10Wb.wav)

For instance, we can pass a list with gender labels to the processing function and use the running index to select the appropriate f0 range.

gender = db["files"]["speaker"].get(map="gender")  # gender per file
f0_range = {
    "female": [160, 300],  # [fmin, fmax]
    "male": [60, 180],
}

def f0(signal, sampling_rate, idx, gender, f0_range):
    # extract mean f0 using a gender adapted range
    y = librosa.yin(
        signal,
        fmin=f0_range[gender.iloc[idx]][0],
        fmax=f0_range[gender.iloc[idx]][1],
        sr=sampling_rate,
    ).mean()
    return y, gender.iloc[idx]

interface = audinterface.Feature(
    ["f0", "gender"],
    process_func=f0,
    process_func_args={
        "gender": gender,
        "f0_range": f0_range,
    },
)
df = interface.process_index(gender.index)
df

			f0	gender
file	start	end
.../03a01Fa.wav	0 days	0 days 00:00:01.898250	128.8100011952554	male
.../03a01Nc.wav	0 days	0 days 00:00:01.611250	111.6334972484757	male
.../16b10Wb.wav	0 days	0 days 00:00:02.522499999	229.09341831755435	female