Minio¶

class audbackend.backend.Minio(host, repository, *, authentication=None, secure=None, **kwargs)[source]¶

Backend for MinIO.

Parameters

host (str) – host address
repository (str) – repository name
authentication (Optional[tuple[str, str]]) – username, password / access key, secret key token tuple. If None, it requests it by calling get_authentication()
secure (Optional[bool]) – if None, it looks in the config file for it, compare get_config(). If it cannot find a matching entry, it defaults to True. Needs to be True when using TLS for the connection, and False otherwise, e.g. when using a local MinIO server.
**kwargs – keyword arguments passed on to minio.Minio

Examples

>>> host = "play.min.io"  # playground provided by https://min.io
>>> auth = ("Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG")
>>> repository = "my-data" + audeer.uid()
>>> Minio.create(host, repository, authentication=auth)
>>> file = audeer.touch("file.txt")
>>> backend = Minio(host, repository, authentication=auth)
>>> try:
...     with backend:
...         backend.put_file(file, "/sub/file.txt")
...         backend.ls()
... finally:
...     Minio.delete(host, repository, authentication=auth)
['/sub/file.txt']

authentication¶

Minio.authentication¶: Object used for authentication, e.g. username, password tuple.

checksum()¶

Minio.checksum(path)¶

MD5 checksum for file on backend.

Parameters

path (str) – path to file on backend

Return type

str

Returns

MD5 checksum

Raises

BackendError – if an error is raised on the backend, e.g. path does not exist
ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

close()¶

Minio.close()[source]¶

Close connection to backend.

This will only change the status of audbackend.backend.Minio.opened as Minio handles closing the session itself.

copy_file()¶

Minio.copy_file(src_path, dst_path, *, validate=False, verbose=False)¶

Copy file on backend.

If dst_path exists and has a different checksum, it is overwritten. Otherwise, the operation is silently skipped.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters

src_path (str) – source path to file on backend
dst_path (str) – destination path to file on backend
validate (bool) – verify file was successfully copied
verbose (bool) – show debug messages

Raises

BackendError – if an error is raised on the backend
InterruptedError – if validation fails
ValueError – if src_path or dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

create()¶

classmethod Minio.create(host, repository, *, authentication=None)¶

Create repository.

Creates repository located at host on the backend.

Parameters

host (str) – host address
repository (str) – repository name
authentication (Optional[object]) – object used for authentication, e.g. a tuple with username and password

Raises

BackendError – if an error is raised on the backend, e.g. repository exists already or cannot be created

date()¶

Minio.date(path)¶

Last modification date of file on backend.

If the date cannot be determined, an empty string is returned.

Parameters

path (str) – path to file on backend

Return type

str

Returns

date in format 'yyyy-mm-dd'

Raises

BackendError – if an error is raised on the backend, e.g. path does not exist
ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

delete()¶

classmethod Minio.delete(host, repository, *, authentication=None)¶

Delete repository.

Deletes repository located at host on the backend.

Parameters

host (str) – host address
repository (str) – repository name
authentication (Optional[object]) – access token for possible authentication, e.g. username, password tuple

Raises

BackendError – if an error is raised on the backend, e.g. repository does not exist

exists()¶

Minio.exists(path, *, suppress_backend_errors=False)¶

Check if file exists on backend.

Parameters

path (str) – path to file on backend
suppress_backend_errors (bool) – if set to True, silently catch errors raised on the backend and return False

Return type

bool

Returns

True if file exists

Raises

BackendError – if suppress_backend_errors is False and an error is raised on the backend, e.g. due to a connection timeout
ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
ValueError – if version is empty or does not match '[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened

get_archive()¶

Minio.get_archive(src_path, dst_root, *, tmp_root=None, validate=False, verbose=False)¶

Get archive from backend and extract.

The archive type is derived from the extension of src_path. See audeer.extract_archive() for supported extensions.

If dst_root does not exist, it is created.

If validate is set to True, a final check is performed to assert that src_path and the retrieved archive have the same checksum. If it fails, the retrieved archive is removed and an InterruptedError is raised.

Parameters

src_path (str) – path to archive on backend
dst_root (str) – local destination directory
tmp_root (Optional[str]) – directory under which archive is temporarily extracted. Defaults to temporary directory of system
validate (bool) – verify archive was successfully retrieved from the backend
verbose (bool) – show debug messages

Return type

list[str]

Returns

extracted files

Raises

BackendError – if an error is raised on the backend, e.g. src_path does not exist
FileNotFoundError – if tmp_root does not exist
InterruptedError – if validation fails
NotADirectoryError – if dst_root is not a directory
PermissionError – if the user lacks write permissions for dst_path
RuntimeError – if extension of src_path is not supported or src_path is a malformed archive
ValueError – if src_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

get_authentication()¶

classmethod Minio.get_authentication(host)[source]¶

Access and secret tokens for given host.

Returns a authentication for MinIO server as tuple.

To get the authentication tuple, the function looks first for the two environment variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY. Otherwise, it tries to extract missing values from a config file, see get_config(). If no config file exists or if it has missing entries, None is returned for the missing entries.

Parameters: host (str) – hostname
Return type: tuple[str, str]
Returns: access token tuple

get_config()¶

classmethod Minio.get_config(host)[source]¶

Configuration of MinIO server.

The default path of the config file is ~/.config/audbackend/minio.cfg. It can be overwritten with the environment variable MINIO_CONFIG_FILE.

If no config file can be found, or no entry for the requested host, an empty dictionary is returned.

The config file expects one section per host, e.g.

[play.min.io]
access_key = "Q3AM3UQ867SPQQA43P2F"
secret_key = "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"

Parameters: host (str) – hostname
Return type: dict
Returns: config entries as dictionary

get_file()¶

Minio.get_file(src_path, dst_path, *, validate=False, verbose=False)¶

Get file from backend.

If the folder of dst_path does not exist, it is created.

If dst_path exists with a different checksum, it is overwritten, Otherwise, the operation is silently skipped.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters

src_path (str) – path to file on backend
dst_path (str) – destination path to local file
validate (bool) – verify file was successfully retrieved from the backend
verbose (bool) – show debug messages

Return type

str

Returns

full path to local file

Raises

BackendError – if an error is raised on the backend, e.g. src_path does not exist
InterruptedError – if validation fails
IsADirectoryError – if dst_path points to an existing folder
PermissionError – if the user lacks write permissions for dst_path
ValueError – if src_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

host¶

Minio.host¶: Host path.

join()¶

Minio.join(path, *paths)¶

Join to (sub-)path on backend.

Parameters

path (str) – first part of path
*paths – additional parts of path

Return type

str

Returns

path joined by Backend.sep

Raises

ValueError – if path contains invalid character or does not start with '/', or if joined path contains invalid character

ls()¶

Minio.ls(path='/', *, pattern=None, suppress_backend_errors=False)¶

List files on backend.

Returns a sorted list of tuples with path and version. If a full path (e.g. /sub/file.ext) is provided, all versions of the path are returned. If a sub-path (e.g. /sub/) is provided, all files that start with the sub-path are returned. When path is set to '/' a (possibly empty) list with all files on the backend is returned.

Parameters

path (str) – path or sub-path (if it ends with '/') on backend
pattern (Optional[str]) – if not None, return only files matching the pattern string, see fnmatch.fnmatch()
suppress_backend_errors (bool) – if set to True, silently catch errors raised on the backend and return an empty list

Return type

list[str]

Returns

list of tuples (path, version)

Raises

BackendError – if suppress_backend_errors is False and an error is raised on the backend, e.g. path does not exist
ValueError – if path does not start with '/' or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

move_file()¶

Minio.move_file(src_path, dst_path, *, validate=False, verbose=False)¶

Move file on backend.

If dst_path exists and has a different checksum, it is overwritten. Otherwise, src_path is removed and the operation silently skipped.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised. To ensure src_path still exists in this case it is first copied and only removed when the check has successfully passed.

Parameters

src_path (str) – source path to file on backend
dst_path (str) – destination path to file on backend
validate (bool) – verify file was successfully moved
verbose (bool) – show debug messages

Raises

BackendError – if an error is raised on the backend
InterruptedError – if validation fails
ValueError – if src_path or dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

open()¶

Minio.open()¶

Open connection to backend.

Repository must exist, use audbackend.backend.Base.create() to create it. Finally, use audbackend.backend.Base.close() to close the connection. Instead of explicitly calling audbackend.backend.Base.open() and audbackend.backend.Base.close() it is good practice to use a with statement.

Raises: BackendError – if an error is raised on the backend, e.g. repository does not exist

opened¶

Minio.opened¶: If a connection to the repository has been established.

owner()¶

Minio.owner(path)¶

Owner of file on backend.

If the owner of the file cannot be determined, an empty string is returned.

Parameters

path (str) – path to file on backend

Return type

str

Returns

owner

Raises

BackendError – if an error is raised on the backend, e.g. path does not exist
ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

path()¶

Minio.path(path)[source]¶

Convert to backend path.

Parameters: path (str) – path on backend
Return type: str
Returns: path

put_archive()¶

Minio.put_archive(src_root, dst_path, *, files=None, tmp_root=None, validate=False, verbose=False)¶

Create archive and put on backend.

The archive type is derived from the extension of dst_path. See audeer.create_archive() for supported extensions.

The operation is silently skipped, if an archive with the same checksum already exists on the backend.

If validate is set to True, a final check is performed to assert that the local archive and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters

src_root (str) – local root directory where files are located. By default, all files below src_root will be included into the archive. Use files to select specific files
dst_path (str) – path to archive on backend
files (Union[str, Sequence[str], None]) – file(s) to include into the archive. Must exist within src_root
tmp_root (Optional[str]) – directory under which archive is temporarily created. Defaults to temporary directory of system
validate (bool) – verify archive was successfully put on the backend
verbose (bool) – show debug messages

Raises

BackendError – if an error is raised on the backend
FileNotFoundError – if src_root, tmp_root, or one or more files do not exist
InterruptedError – if validation fails
NotADirectoryError – if src_root is not a folder
RuntimeError – if dst_path does not end with zip or tar.gz or a file in files is not below root
ValueError – if dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

put_file()¶

Minio.put_file(src_path, dst_path, *, validate=False, verbose=False)¶

Put file on backend.

The operation is silently skipped, if a file with the same checksum already exists on the backend.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters

src_path (str) – path to local file
dst_path (str) – path to file on backend
validate (bool) – verify file was successfully put on the backend
verbose (bool) – show debug messages

Raises

BackendError – if an error is raised on the backend
FileNotFoundError – if src_path does not exist
InterruptedError – if validation fails
IsADirectoryError – if src_path is a folder
ValueError – if dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

remove_file()¶

Minio.remove_file(path)¶

Remove file from backend.

Parameters

path (str) – path to file on backend

Raises

BackendError – if an error is raised on the backend, e.g. path does not exist
ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened

repository¶

Minio.repository¶: Repository name.

sep¶

Minio.sep¶

File separator on backend.

Returns: file separator

split()¶

Minio.split(path)¶

Split path on backend into sub-path and basename.

Parameters: path (str) – path containing Backend.sep as separator
Return type: tuple[str, str]
Returns: tuple containing (root, basename)
Raises: ValueError – if path does not start with '/' or does not match '[A-Za-z0-9/._-]+'