Versioned¶
- class audbackend.interface.Versioned(backend)[source]¶
Interface for versioned file access.
Use this interface if you care about versioning. For each file on the backend path one or more versions may exist.
- Parameters
backend (
Base
) – backend object
Examples
>>> host = audeer.mkdir("host") >>> audbackend.backend.FileSystem.create(host, "repo") >>> backend = audbackend.backend.FileSystem(host, "repo") >>> backend.open() >>> interface = Versioned(backend) >>> file = "src.txt" >>> interface.put_archive(".", "/sub/archive.zip", "1.0.0", files=[file]) >>> for version in ["1.0.0", "2.0.0"]: ... interface.put_file(file, "/file.txt", version) >>> interface.ls() [('/file.txt', '1.0.0'), ('/file.txt', '2.0.0'), ('/sub/archive.zip', '1.0.0')] >>> interface.get_file("/file.txt", "dst.txt", "2.0.0") '...dst.txt'
backend¶
- Versioned.backend¶
Backend object.
- Returns
backend object
Examples
>>> interface.backend audbackend.backend.FileSystem('host', 'repo')
checksum()¶
- Versioned.checksum(path, version)[source]¶
MD5 checksum for file on backend.
- Parameters
- Return type
- Returns
MD5 checksum
- Raises
BackendError – if an error is raised on the backend, e.g.
path
does not existValueError – if
path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> import audeer >>> audbackend.checksum(file) 'd41d8cd98f00b204e9800998ecf8427e' >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.checksum("/file.txt", "1.0.0") 'd41d8cd98f00b204e9800998ecf8427e'
copy_file()¶
- Versioned.copy_file(src_path, dst_path, *, version=None, validate=False, verbose=False)[source]¶
Copy file on backend.
If
version
isNone
all versions ofsrc_path
will be copied.If
dst_path
exists and has a different checksum, it is overwritten. Otherwise, the operation is silently skipped.If
validate
is set toTrue
, a final check is performed to assert thatsrc_path
anddst_path
have the same checksum. If it fails,dst_path
is removed and anInterruptedError
is raised.- Parameters
- Raises
BackendError – if an error is raised on the backend
InterruptedError – if validation fails
ValueError – if
src_path
ordst_path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.exists("/copy.txt", "1.0.0") False >>> interface.copy_file("/file.txt", "/copy.txt", version="1.0.0") >>> interface.exists("/copy.txt", "1.0.0") True
date()¶
- Versioned.date(path, version)[source]¶
Last modification date of file on backend.
If the date cannot be determined, an empty string is returned.
- Parameters
- Return type
- Returns
date in format
'yyyy-mm-dd'
- Raises
BackendError – if an error is raised on the backend, e.g.
path
does not existValueError – if
path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.date("/file.txt", "1.0.0") '1991-02-20'
exists()¶
- Versioned.exists(path, version, *, suppress_backend_errors=False)[source]¶
Check if file exists on backend.
- Parameters
- Return type
- Returns
True
if file exists- Raises
BackendError – if
suppress_backend_errors
isFalse
and an error is raised on the backend, e.g. due to a connection timeoutValueError – if
path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.exists("/file.txt", "1.0.0") False >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.exists("/file.txt", "1.0.0") True
get_archive()¶
- Versioned.get_archive(src_path, dst_root, version, *, tmp_root=None, validate=False, verbose=False)[source]¶
Get archive from backend and extract.
The archive type is derived from the extension of
src_path
. Seeaudeer.extract_archive()
for supported extensions.If
dst_root
does not exist, it is created.If
validate
is set toTrue
, a final check is performed to assert thatsrc_path
and the retrieved archive have the same checksum. If it fails, the retrieved archive is removed and anInterruptedError
is raised.- Parameters
src_path (
str
) – path to archive on backenddst_root (
str
) – local destination directoryversion (
str
) – version stringtmp_root (
Optional
[str
]) – directory under which archive is temporarily extracted. Defaults to temporary directory of systemvalidate (
bool
) – verify archive was successfully retrieved from the backendverbose (
bool
) – show debug messages
- Return type
- Returns
extracted files
- Raises
BackendError – if an error is raised on the backend, e.g.
src_path
does not existFileNotFoundError – if
tmp_root
does not existInterruptedError – if validation fails
NotADirectoryError – if
dst_root
is not a directoryPermissionError – if the user lacks write permissions for
dst_path
RuntimeError – if extension of
src_path
is not supported orsrc_path
is a malformed archiveValueError – if
src_path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_archive(".", "/sub/archive.zip", "1.0.0", files=[file]) >>> os.remove(file) >>> interface.get_archive("/sub/archive.zip", ".", "1.0.0") ['src.txt']
get_file()¶
- Versioned.get_file(src_path, dst_path, version, *, validate=False, verbose=False)[source]¶
Get file from backend.
If the folder of
dst_path
does not exist, it is created.If
dst_path
exists with a different checksum, it is overwritten, or otherwise, the operation is silently skipped.If
validate
is set toTrue
, a final check is performed to assert thatsrc_path
anddst_path
have the same checksum. If it fails,dst_path
is removed and anInterruptedError
is raised.- Parameters
- Return type
- Returns
full path to local file
- Raises
BackendError – if an error is raised on the backend, e.g.
src_path
does not existInterruptedError – if validation fails
IsADirectoryError – if
dst_path
points to an existing folderPermissionError – if the user lacks write permissions for
dst_path
ValueError – if
src_path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.get_file("/file.txt", "dst.txt", "1.0.0") '...dst.txt'
join()¶
- Versioned.join(path, *paths)¶
Join to path on backend.
- Parameters
path (
str
) – first part of path*paths – additional parts of path
- Return type
- Returns
path joined by
Backend.sep
- Raises
ValueError – if
path
contains invalid character or does not start with'/'
, or if joined path contains invalid character
Examples
>>> interface.join("/", "file.txt") '/file.txt' >>> interface.join("/sub", "file.txt") '/sub/file.txt' >>> interface.join("//sub//", "/", "", None, "/file.txt") '/sub/file.txt'
latest_version()¶
- Versioned.latest_version(path)[source]¶
Latest version of a file.
- Parameters
path (
str
) – path to file on backend- Return type
- Returns
version string
- Raises
BackendError – if an error is raised on the backend, e.g.
path
does not existValueError – if
path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.put_file(file, "/file.txt", "2.0.0") >>> interface.latest_version("/file.txt") '2.0.0'
ls()¶
- Versioned.ls(path='/', *, latest_version=False, pattern=None, suppress_backend_errors=False)[source]¶
List files on backend.
Returns a sorted list of tuples with path and version. If a full path (e.g.
/sub/file.ext
) is provided, all versions of the path are returned. If a sub-path (e.g./sub/
) is provided, all files that start with the sub-path are returned. Whenpath
is set to'/'
a (possibly empty) list with all files on the backend is returned.- Parameters
path (
str
) – path or sub-path (if it ends with'/'
) on backendlatest_version (
bool
) – if multiple versions of a file exist, only include the latestpattern (
Optional
[str
]) – if notNone
, return only files matching the pattern string, seefnmatch.fnmatch()
suppress_backend_errors (
bool
) – if set toTrue
, silently catch errors raised on the backend and return an empty list
- Return type
- Returns
list of tuples (path, version)
- Raises
BackendError – if
suppress_backend_errors
isFalse
and an error is raised on the backend, e.g.path
does not existValueError – if
path
does not start with'/'
or does not match'[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.put_file(file, "/file.txt", "2.0.0") >>> interface.put_archive(".", "/sub/archive.zip", "1.0.0", files=[file]) >>> interface.ls() [('/file.txt', '1.0.0'), ('/file.txt', '2.0.0'), ('/sub/archive.zip', '1.0.0')] >>> interface.ls(latest_version=True) [('/file.txt', '2.0.0'), ('/sub/archive.zip', '1.0.0')] >>> interface.ls("/file.txt") [('/file.txt', '1.0.0'), ('/file.txt', '2.0.0')] >>> interface.ls(pattern="*.txt") [('/file.txt', '1.0.0'), ('/file.txt', '2.0.0')] >>> interface.ls(pattern="archive.*") [('/sub/archive.zip', '1.0.0')] >>> interface.ls("/sub/") [('/sub/archive.zip', '1.0.0')]
move_file()¶
- Versioned.move_file(src_path, dst_path, *, version=None, validate=False, verbose=False)[source]¶
Move file on backend.
If
version
isNone
all versions ofsrc_path
will be moved.If
dst_path
exists and has a different checksum, it is overwritten. Otherwise,src_path
is removed and the operation silently skipped.If
validate
is set toTrue
, a final check is performed to assert thatsrc_path
anddst_path
have the same checksum. If it fails,dst_path
is removed and anInterruptedError
is raised. To ensuresrc_path
still exists in this case it is first copied and only removed when the check has successfully passed.- Parameters
- Raises
BackendError – if an error is raised on the backend
InterruptedError – if validation fails
ValueError – if
src_path
ordst_path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.exists("/move.txt", "1.0.0") False >>> interface.move_file("/file.txt", "/move.txt", version="1.0.0") >>> interface.exists("/move.txt", "1.0.0") True >>> interface.exists("/file.txt", "1.0.0") False
owner()¶
- Versioned.owner(path, version)[source]¶
Owner of file on backend.
If the owner of the file cannot be determined, an empty string is returned.
- Parameters
- Return type
- Returns
owner
- Raises
BackendError – if an error is raised on the backend, e.g.
path
does not existValueError – if
path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.owner("/file.txt", "1.0.0") 'doctest'
put_archive()¶
- Versioned.put_archive(src_root, dst_path, version, *, files=None, tmp_root=None, validate=False, verbose=False)[source]¶
Create archive and put on backend.
The archive type is derived from the extension of
dst_path
. Seeaudeer.create_archive()
for supported extensions.The operation is silently skipped, if an archive with the same checksum already exists on the backend.
If
validate
is set toTrue
, a final check is performed to assert that the local archive anddst_path
have the same checksum. If it fails,dst_path
is removed and anInterruptedError
is raised.- Parameters
src_root (
str
) – local root directory where files are located. By default, all files belowsrc_root
will be included into the archive. Usefiles
to select specific filesdst_path (
str
) – path to archive on backendversion (
str
) – version stringfiles (
Union
[str
,Sequence
[str
],None
]) – file(s) to include into the archive. Must exist withinsrc_root
tmp_root (
Optional
[str
]) – directory under which archive is temporarily created. Defaults to temporary directory of systemvalidate (
bool
) – verify archive was successfully put on the backendverbose (
bool
) – show debug messages
- Raises
BackendError – if an error is raised on the backend
FileNotFoundError – if
src_root
,tmp_root
, or one or morefiles
do not existInterruptedError – if validation fails
NotADirectoryError – if
src_root
is not a folderRuntimeError – if
dst_path
does not end withzip
ortar.gz
or a file infiles
is not belowroot
ValueError – if
dst_path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.exists("/sub/archive.tar.gz", "1.0.0") False >>> interface.put_archive(".", "/sub/archive.tar.gz", "1.0.0") >>> interface.exists("/sub/archive.tar.gz", "1.0.0") True
put_file()¶
- Versioned.put_file(src_path, dst_path, version, *, validate=False, verbose=False)[source]¶
Put file on backend.
The operation is silently skipped, if a file with the same checksum already exists on the backend.
If
validate
is set toTrue
, a final check is performed to assert thatsrc_path
anddst_path
have the same checksum. If it fails,dst_path
is removed and anInterruptedError
is raised.- Parameters
- Returns
file path on backend
- Raises
BackendError – if an error is raised on the backend
FileNotFoundError – if
src_path
does not existInterruptedError – if validation fails
IsADirectoryError – if
src_path
is a folderValueError – if
dst_path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.exists("/file.txt", "3.0.0") False >>> interface.put_file(file, "/file.txt", "3.0.0") >>> interface.exists("/file.txt", "3.0.0") True
remove_file()¶
- Versioned.remove_file(path, version)[source]¶
Remove file from backend.
- Parameters
- Raises
BackendError – if an error is raised on the backend, e.g.
path
does not existValueError – if
path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
ValueError – if
version
is empty or does not match'[A-Za-z0-9._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.exists("/file.txt", "1.0.0") True >>> interface.remove_file("/file.txt", "1.0.0") >>> interface.exists("/file.txt", "1.0.0") False
repository¶
- Versioned.repository¶
Repository name.
- Returns
repository name
Examples
>>> interface.repository 'repo'
sep¶
- Versioned.sep¶
File separator on backend.
- Returns
file separator
Examples
>>> interface.sep '/'
split()¶
- Versioned.split(path)¶
Split path on backend into sub-path and basename.
- Parameters
path (
str
) – path containingBackend.sep
as separator- Return type
- Returns
tuple containing (root, basename)
- Raises
ValueError – if
path
does not start with'/'
or does not match'[A-Za-z0-9/._-]+'
Examples
>>> interface.split("/") ('/', '') >>> interface.split("/file.txt") ('/', 'file.txt') >>> interface.split("/sub/") ('/sub/', '') >>> interface.split("/sub//file.txt") ('/sub/', 'file.txt')
versions()¶
- Versioned.versions(path, *, suppress_backend_errors=False)[source]¶
Versions of a file.
- Parameters
- Return type
- Returns
list of versions in ascending order
- Raises
BackendError – if
suppress_backend_errors
isFalse
and an error is raised on the backend, e.g.path
does not existValueError – if
path
does not start with'/'
, ends on'/'
, or does not match'[A-Za-z0-9/._-]+'
RuntimeError – if backend was not opened
Examples
>>> file = "src.txt" >>> interface.put_file(file, "/file.txt", "1.0.0") >>> interface.put_file(file, "/file.txt", "2.0.0") >>> interface.versions("/file.txt") ['1.0.0', '2.0.0']