load_configuration()

audeer.load_configuration(default_config_file, user_configs=None, *, env_prefix=None, strict=False, types=None, validate=None, tracking=None)[source]

Load configuration from files and environment variables.

Configuration values are collected from, in order of increasing precedence:

  1. default_config_file, e.g. the configuration file shipped with a package

  2. user_configs, applied in the given order (a later entry overrides an earlier one)

  3. environment variables, if env_prefix is given

Configuration files or mappings are deep-merged: nested mappings are merged key by key, so a section in a user file only overrides the keys it defines and keeps the remaining keys from the default. Any non-mapping value (including a list) replaces the previous value as a whole. An integer is converted to a float where the default value is a float.

Environment variables are matched against the upper-cased configuration keys, prefixed with env_prefix and an underscore, e.g. the key cache_root is overridden by <env_prefix>_CACHE_ROOT. Keys inside nested mappings are addressed by joining the levels with __, e.g. model.device is overridden by <env_prefix>_MODEL__DEVICE. Several keys of a nested mapping can instead be set by a single variable holding a JSON object, e.g. <env_prefix>_MODEL='{"device": "cpu"}'; the object is deep-merged into the mapping like a user configuration file (keys it omits keep their values) and may introduce keys not present in the files. Nested variables are applied afterwards and therefore take precedence, e.g. <env_prefix>_MODEL__DEVICE overrides device from <env_prefix>_MODEL. The value of an environment variable is converted to the type of the corresponding default value: str values are used as they are, int/float values are cast, and list values are parsed as JSON. A bool value is True for "1", "true", "yes", "on" (case insensitive) and False for any other value; a boolean is therefore never rejected, whereas int, float and JSON conversions raise a ValueError on invalid input. A default value of any other type (e.g. a date parsed from YAML) cannot be overridden and raises a ValueError as well. Only keys already present in the merged configuration, whether from a file or a mapping, can be overridden by environment variables. Only string keys are matched; a non-string key (e.g. a numeric YAML key) is left untouched.

A default value of None carries no type, so an environment variable would be kept as a string. Use types to declare the intended type for such keys, or to override the inferred type of any key. types mirrors the (possibly nested) structure of the configuration; an entry that does not match a configuration key raises a ValueError.

Missing or empty configuration files are skipped.

Reading configuration files requires pyyaml, which is installed when depending on audeer[yaml].

Warning

load_configuration() is experimental and might have breaking changes in a future release.

Parameters:

  • default_config_file (str) – path to default configuration file. The file does not have to exist

  • user_configs (str | Mapping | Sequence[str | Mapping] | None) – path(s) to user configuration file(s), or already parsed mapping(s), applied in the given order. Files do not have to exist, and a given mapping is not modified

  • env_prefix (str | None) – prefix of environment variables used to override configuration values. If None, environment variables are ignored

  • strict (bool) – if True, every user configuration, and every environment variable holding a JSON object for a section, may only contain keys of default_config_file, with values of the same type as their default values

  • types (Mapping | None) – mapping that declares the type of configuration values, mirroring the (possibly nested) configuration structure. Used to cast environment variable overrides for keys whose default value is None, or to override the type inferred from the default value. Supported types are bool, int, float, str, list

  • validate (Callable[[dict], None] | None) – callable that receives the merged configuration dictionary and raises an error if it is invalid. It is applied once, after files and environment variables are merged

  • tracking (MutableMapping | None) – mutable mapping that records, for each configuration key, which layer set its effective value: f"file:{path}" for default_config_file or a user_configs entry that provided a file, "mapping[<i>]" for the user_configs entry at index <i> that provided an already parsed mapping (indices count every entry of the sequence, file or mapping alike), or f"env:{name}" naming the exact environment variable (e.g. "env:PKG_MODEL__DEVICE") for an environment variable override. A section JSON environment variable labels every key it sets; a more specific nested variable re-labels only the key it overrides. Entries are added to tracking in place. Reuse the same mapping across several calls to accumulate their entries. If None, no tracking is performed

Return type:

dict

Returns:

merged configuration dictionary

Raises:

  • ImportError – if pyyaml is not installed, and a configuration file exists

  • ValueError – if a user configuration, or an environment variable holding a JSON object for a section, sets a section of default_config_file to a value, or a value to a section

  • ValueError – if a configuration file does not contain a mapping of key-value pairs

  • ValueError – if strict is True and a user configuration, or an environment variable holding a JSON object for a section, contains a key that is not in default_config_file, or a value whose type does not match its default value

  • ValueError – if an environment variable cannot be converted to the type of the corresponding default value, or that type is not supported

  • ValueError – if a type declared in types is not a class, or not one of the supported types

  • ValueError – if types, or a types entry for a nested section, is not a mapping

  • ValueError – if a types entry does not match any configuration key

  • ValueError – if tracking is not a mutable mapping

Examples

>>> import tempfile
>>> config_file = audeer.path(tempfile.mkdtemp(), "config.yaml")
>>> with open(config_file, "w") as file:
...     _ = file.write("cache_root: ~/cache\n")
>>> audeer.load_configuration(config_file)
{'cache_root': '~/cache'}

A user configuration can also be given as an already parsed mapping.

>>> config_file = audeer.path(tempfile.mkdtemp(), "config.yaml")
>>> with open(config_file, "w") as file:
...     _ = file.write("model:\n  device: cpu\n  lora: false\n")
>>> audeer.load_configuration(config_file, {"model": {"device": "cuda"}})
{'model': {'device': 'cuda', 'lora': False}}

A key that defaults to None has no inferred type, so declare it in types.

>>> import os
>>> config_file = audeer.path(tempfile.mkdtemp(), "config.yaml")
>>> with open(config_file, "w") as file:
...     _ = file.write("hosts: null\n")
>>> os.environ["APP_HOSTS"] = '["host1", "host2"]'
>>> audeer.load_configuration(
...     config_file, env_prefix="APP", types={"hosts": list}
... )
{'hosts': ['host1', 'host2']}
>>> del os.environ["APP_HOSTS"]

tracking records which layer set each key’s effective value.

>>> config_file = audeer.path(tempfile.mkdtemp(), "config.yaml")
>>> with open(config_file, "w") as file:
...     _ = file.write("model:\n  device: cpu\n  lora: false\n")
>>> os.environ["PKG_MODEL__DEVICE"] = "cuda"
>>> tracking = {}
>>> audeer.load_configuration(config_file, env_prefix="PKG", tracking=tracking)
{'model': {'device': 'cuda', 'lora': False}}
>>> tracking
{'model': {'device': 'env:PKG_MODEL__DEVICE', 'lora': 'file:...config.yaml'}}
>>> del os.environ["PKG_MODEL__DEVICE"]