FileSystem

class audbackend.backend.FileSystem(host, repository)

Backend for file system.

Parameters:

• host (str) – host directory

• repository (str) – repository name

authentication

FileSystem.authentication

Object used for authentication, e.g. username, password tuple.

checksum()

FileSystem.checksum(path)

MD5 checksum for file on backend.

Parameters:

path (str) – path to file on backend

Return type:

str

Returns:

MD5 checksum

Raises:

• BackendError – if an error is raised on the backend, e.g. path does not exist

• ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

close()

FileSystem.close()

Close connection to backend.

Raises:

BackendError – if an error is raised on the backend

copy_file()

FileSystem.copy_file(src_path, dst_path, *, num_workers=1, validate=False, verbose=False)

Copy file on backend.

If dst_path exists and has a different checksum, it is overwritten. Otherwise, the operation is silently skipped.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters:

• src_path (str) – source path to file on backend

• dst_path (str) – destination path to file on backend

• num_workers (int) – number of parallel jobs

• validate (bool) – verify file was successfully copied

• verbose (bool) – show debug messages

Raises:

• BackendError – if an error is raised on the backend

• InterruptedError – if validation fails

• ValueError – if src_path or dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

create()

classmethod FileSystem.create(host, repository, *, authentication=None)

Create repository.

Creates repository located at host on the backend.

Parameters:

• host (str) – host address

• repository (str) – repository name

• authentication (object) – object used for authentication, e.g. a tuple with username and password

Raises:

BackendError – if an error is raised on the backend, e.g. repository exists already or cannot be created

date()

FileSystem.date(path)

Last modification date of file on backend.

If the date cannot be determined, an empty string is returned.

Parameters:

path (str) – path to file on backend

Return type:

str

Returns:

date in format 'yyyy-mm-dd'

Raises:

• BackendError – if an error is raised on the backend, e.g. path does not exist

• ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

delete()

classmethod FileSystem.delete(host, repository, *, authentication=None)

Delete repository.

Deletes repository located at host on the backend.

Parameters:

• host (str) – host address

• repository (str) – repository name

• authentication (object) – access token for possible authentication, e.g. username, password tuple

Raises:

BackendError – if an error is raised on the backend, e.g. repository does not exist

exists()

FileSystem.exists(path, *, suppress_backend_errors=False)

Check if file exists on backend.

Parameters:

• path (str) – path to file on backend

• suppress_backend_errors (bool) – if set to True, silently catch errors raised on the backend and return False

Return type:

bool

Returns:

True if file exists

Raises:

• BackendError – if suppress_backend_errors is False and an error is raised on the backend, e.g. due to a connection timeout

• ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• ValueError – if version is empty or does not match '[A-Za-z0-9._-]+'

• RuntimeError – if backend was not opened

get_archive()

FileSystem.get_archive(src_path, dst_root, *, tmp_root=None, num_workers=1, validate=False, verbose=False)

Get archive from backend and extract.

For ZIP archives, streaming extraction is used if stream-unzip is installed, and num_workers=1. It extracts files during download without storing the archive locally.

For uncompressed archives or archives that do not support streaming extraction, the archive is downloaded first and then extracted. In that case we recommend setting num_workers=5 for best performance.

If dst_root does not exist, it is created.

If validate is set to True, a final check is performed to assert that src_path and the retrieved archive have the same checksum. If it fails, the extracted files are removed and an InterruptedError is raised.

Parameters:

• src_path (str) – path to archive on backend

• dst_root (str) – local destination directory

• tmp_root (str) – directory under which archive is temporarily extracted. Defaults to temporary directory of system. Not used for ZIP archives when stream-unzip is available.

• num_workers (int) – number of parallel jobs

• validate (bool) – verify archive was successfully retrieved from the backend

• verbose (bool) – show debug messages

Return type:

list[str]

Returns:

extracted files

Raises:

• BackendError – if an error is raised on the backend, e.g. src_path does not exist

• FileNotFoundError – if tmp_root does not exist

• InterruptedError – if validation fails

• NotADirectoryError – if dst_root is not a directory

• PermissionError – if the user lacks write permissions for dst_path

• RuntimeError – if extension of src_path is not supported or src_path is a malformed archive

• ValueError – if src_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

get_file()

FileSystem.get_file(src_path, dst_path, *, num_workers=1, validate=False, verbose=False)

Get file from backend.

If the folder of dst_path does not exist, it is created.

If dst_path exists with a different checksum, it is overwritten, Otherwise, the operation is silently skipped.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters:

• src_path (str) – path to file on backend

• dst_path (str) – destination path to local file

• num_workers (int) – number of parallel jobs

• validate (bool) – verify file was successfully retrieved from the backend

• verbose (bool) – show debug messages

Return type:

str

Returns:

full path to local file

Raises:

• BackendError – if an error is raised on the backend, e.g. src_path does not exist

• InterruptedError – if validation fails

• IsADirectoryError – if dst_path points to an existing folder

• PermissionError – if the user lacks write permissions for dst_path

• ValueError – if src_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

host

FileSystem.host

Host path.

join()

FileSystem.join(path, *paths)

Join to (sub-)path on backend.

Parameters:

• path (str) – first part of path

• *paths – additional parts of path

Return type:

str

Returns:

path joined by Backend.sep

Raises:

ValueError – if path contains invalid character or does not start with '/', or if joined path contains invalid character

ls()

FileSystem.ls(path='/', *, pattern=None, suppress_backend_errors=False)

List files on backend.

Returns a sorted list of tuples with path and version. If a full path (e.g. /sub/file.ext) is provided, all versions of the path are returned. If a sub-path (e.g. /sub/) is provided, all files that start with the sub-path are returned. When path is set to '/' a (possibly empty) list with all files on the backend is returned.

Parameters:

• path (str) – path or sub-path (if it ends with '/') on backend

• pattern (str) – if not None, return only files matching the pattern string, see fnmatch.fnmatch()

• suppress_backend_errors (bool) – if set to True, silently catch errors raised on the backend and return an empty list

Return type:

list[str]

Returns:

list of tuples (path, version)

Raises:

• BackendError – if suppress_backend_errors is False and an error is raised on the backend, e.g. path does not exist

• ValueError – if path does not start with '/' or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

move_file()

FileSystem.move_file(src_path, dst_path, *, num_workers=1, validate=False, verbose=False)

Move file on backend.

If dst_path exists and has a different checksum, it is overwritten. Otherwise, src_path is removed and the operation silently skipped.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised. To ensure src_path still exists in this case it is first copied and only removed when the check has successfully passed.

Parameters:

• src_path (str) – source path to file on backend

• dst_path (str) – destination path to file on backend

• num_workers (int) – number of parallel jobs

• validate (bool) – verify file was successfully moved

• verbose (bool) – show debug messages

Raises:

• BackendError – if an error is raised on the backend

• InterruptedError – if validation fails

• ValueError – if src_path or dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

open()

FileSystem.open()

Open connection to backend.

Repository must exist, use audbackend.backend.Base.create() to create it. Finally, use audbackend.backend.Base.close() to close the connection. Instead of explicitly calling audbackend.backend.Base.open() and audbackend.backend.Base.close() it is good practice to use a with statement.

Raises:

BackendError – if an error is raised on the backend, e.g. repository does not exist

opened

FileSystem.opened

If a connection to the repository has been established.

owner()

FileSystem.owner(path)

Owner of file on backend.

If the owner of the file cannot be determined, an empty string is returned.

Parameters:

path (str) – path to file on backend

Return type:

str

Returns:

owner

Raises:

• BackendError – if an error is raised on the backend, e.g. path does not exist

• ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

put_archive()

FileSystem.put_archive(src_root, dst_path, *, files=None, tmp_root=None, validate=False, verbose=False)

Create archive and put on backend.

The archive type is derived from the extension of dst_path. See audeer.create_archive() for supported extensions.

The operation is silently skipped, if an archive with the same checksum already exists on the backend.

If validate is set to True, a final check is performed to assert that the local archive and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters:

• src_root (str) – local root directory where files are located. By default, all files below src_root will be included into the archive. Use files to select specific files

• dst_path (str) – path to archive on backend

• files (str | Sequence[str]) – file(s) to include into the archive. Must exist within src_root

• tmp_root (str) – directory under which archive is temporarily created. Defaults to temporary directory of system

• validate (bool) – verify archive was successfully put on the backend

• verbose (bool) – show debug messages

Raises:

• BackendError – if an error is raised on the backend

• FileNotFoundError – if src_root, tmp_root, or one or more files do not exist

• InterruptedError – if validation fails

• NotADirectoryError – if src_root is not a folder

• RuntimeError – if dst_path does not end with zip or tar.gz or a file in files is not below root

• ValueError – if dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

put_file()

FileSystem.put_file(src_path, dst_path, *, validate=False, verbose=False)

Put file on backend.

The operation is silently skipped, if a file with the same checksum already exists on the backend.

If validate is set to True, a final check is performed to assert that src_path and dst_path have the same checksum. If it fails, dst_path is removed and an InterruptedError is raised.

Parameters:

• src_path (str) – path to local file

• dst_path (str) – path to file on backend

• validate (bool) – verify file was successfully put on the backend

• verbose (bool) – show debug messages

Raises:

• BackendError – if an error is raised on the backend

• FileNotFoundError – if src_path does not exist

• InterruptedError – if validation fails

• IsADirectoryError – if src_path is a folder

• ValueError – if dst_path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

remove_file()

FileSystem.remove_file(path)

Remove file from backend.

Parameters:

path (str) – path to file on backend

Raises:

• BackendError – if an error is raised on the backend, e.g. path does not exist

• ValueError – if path does not start with '/', ends on '/', or does not match '[A-Za-z0-9/._-]+'

• RuntimeError – if backend was not opened

repository

FileSystem.repository

Repository name.

sep

FileSystem.sep

File separator on backend.

Returns: file separator

split()

FileSystem.split(path)

Split path on backend into sub-path and basename.

Parameters:

path (str) – path containing Backend.sep as separator

Return type:

tuple[str, str]

Returns:

tuple containing (root, basename)

Raises:

ValueError – if path does not start with '/' or does not match '[A-Za-z0-9/._-]+'