yellowdog_client.object_store – Object Store

All functionality is accessed via the yellowdog_client.object_store.ObjectStoreClient.

class yellowdog_client.object_store.ObjectStoreClient(proxy)

The API interface exposed by the YellowDog Object Store Service

Parameters:

proxy (ObjectStoreServiceProxy)

DEFAULT_DOWNLOAD_CHUNK_SIZE = 5242880
DEFAULT_DOWNLOAD_THREAD_COUNT = 10
DEFAULT_DOWNLOAD_THROTTLE_PERIOD_SEC = 10
DEFAULT_UPLOAD_CHUNK_SIZE = 5242880
DEFAULT_UPLOAD_THREAD_COUNT = 10
DEFAULT_UPLOAD_THROTTLE_PERIOD_SEC = 10
ON_ERROR = 'on_error'
abort_all_transfers()

Aborts all running transfers in object store client

Return type:

None

property active_sessions: List[AbstractSession]
Getter:

List of active sessions for both upload and download

Return type:

List of yellowdog_client.object_store.abstracts.AbstractSession

property all_sessions: List[AbstractSession]
Getter:

List of all sessions (active and inactive) for both upload and download

Return type:

List of yellowdog_client.object_store.abstracts.AbstractSession

build_download_batch()

Retrieves a batch builder for downloads, which is used to create multiple download sessions

Returns:

download batch builder

Return type:

yellowdog_client.object_store.download.DownloadBatchBuilder

build_upload_batch()

Retrieves a batch builder for uploads, which is used to create multiple upload sessions

Returns:

upload batch builder

Return type:

yellowdog_client.object_store.upload.UploadBatchBuilder

check_object_exists(namespace, name)
Parameters:
Return type:

bool

clear_inactive_sessions()

Removes any inactive sessions from the list for both upload and download.

Return type:

None

close()

Aborts all transfers for upload and download. When executing __exit__(), method close() is invoked

Return type:

None

create_download_session(file_namespace, file_name, destination_folder_path, destination_file_name=None, transfer_properties=None)

Creates a new upload session for the file

Parameters:

  • file_namespace (str) – Namespace for file storage

  • file_name (str) – path of the file to upload

  • destination_folder_path (str) – directory path to store downloaded file

  • destination_file_name (Optional[str]) – new name of the file to use within object store namespace

  • transfer_properties (Optional[yellowdog_client.object_store.model.TransferProperties]) – custom properties to use for file transfer

Returns:

upload session, which, upon return, can be started or aborted

Return type:

yellowdog_client.object_store.upload.UploadSession

create_upload_session(file_namespace, source_file_path, destination_file_name=None, transfer_properties=None)

Creates a new upload session for the file

Parameters:

  • file_namespace (str) – Namespace for file storage

  • source_file_path (str) – path of the file to upload

  • destination_file_name (Optional[str]) – new name of the file to use within object store namespace

  • transfer_properties (Optional[yellowdog_client.object_store.model.TransferProperties]) – custom properties to use for file transfer

Returns:

upload session, which, upon return, can be started or aborted

Return type:

yellowdog_client.object_store.upload.UploadSession

delete_namespace_storage_configuration(namespace)

Removes all cloud provider storage configurations within namespace in YellowDog object store

Parameters:

namespace (str) – namespace, holding storage configurations

Return type:

None

delete_objects(namespace, object_paths)

Delete a collection of files within object store namespace

Parameters:
Return type:

None

property download_chunk_size: int
Getter:

chunk size in bytes, used for downloads

Setter:

chunk size in bytes for download to use

Type:

int

Return type:

int

property download_file_retry_count: int
Getter:

number of retries, which will be attempted in case of download failure

Setter:

number of retries for downloads to use

Type:

int

Return type:

int

property download_max_bytes_per_second: int
Getter:

limit of bytes per second allowed for download. If equal to 0, no throttle is applied

Setter:

sets a number of max bytes per second for download engine to allow transferring

Type:

int

Return type:

int

property downloads_active: bool
Getter:

True, if any of download threads are actively transferring chunks

Return type:

bool

get_namespace_object_paths(request)

Returns a list of all stored objects within namespace in object store

Parameters:

request (ObjectPathsRequest) – the request

Return type:

List[ObjectPath]

Returns:

object paths within the namespace

get_namespace_object_paths_slice(request)

Returns a slice of stored objects within namespace in object store

Parameters:

request (ObjectPathsSliceRequest) – the request

Return type:

Slice[ObjectPath]

Returns:

object paths within the namespace

get_namespace_storage_configurations()

Returns all available storage configurations found within YellowDog object store

Returns:

cloud provider storage configurations

Return type:

List[yellowdog_client.model.NamespaceStorageConfiguration]

get_namespaces()

Returns a list of all namespaces in object store

Return type:

List[str]

Returns:

all namespaces

get_object_detail(namespace, name)

Returns details of stored file within the namespace in object store

Parameters:

  • namespace (str) – file name of the file to query

  • name (str) – storage namespace within object store

Return type:

ObjectDetail

Returns:

details about the remote file

put_namespace_storage_configuration(namespace_storage_configuration)

Stores cloud provider storage configuration within YellowDog object store

Parameters:

namespace_storage_configuration (yellowdog_client.model.NamespaceStorageConfiguration) – storage configuration for cloud provider

Return type:

NamespaceStorageConfiguration

start_transfers()

Enables both uploads and downloads for yellowdog object store. Required to make any transfers using the client

Return type:

None

stop_transfers()

Stops yellowdog object store client from transferring any new chunks for both upload and download. New transfers will be put on hold until start() is used

Return type:

None

property upload_chunk_size: int
Getter:

chunk size in bytes, used for uploads

Setter:

chunk size in bytes for upload to use

Type:

int

Return type:

int

property upload_file_retry_count: int
Getter:

number of retries, which will be attempted in case of upload failure

Setter:

number of retries for uploads to use

Type:

int

Return type:

int

property upload_max_bytes_per_second: int
Getter:

limit of bytes per second allowed for upload. If equal to 0, no throttle is applied

Setter:

sets a number of max bytes per second for upload engine to allow transferring

Type:

int

Return type:

int

property uploads_active: bool
Getter:

True, if any of upload threads are actively transferring chunks

Return type:

bool

class yellowdog_client.object_store.ObjectStoreServiceProxy(proxy, request_timeout_ms=10000, request_upload_chunk_timeout_ms=1200000, request_download_chunk_timeout_ms=1200000)
Parameters:

  • proxy (Proxy)

  • request_timeout_ms (int)

  • request_upload_chunk_timeout_ms (int)

  • request_download_chunk_timeout_ms (int)

ON_ERROR = 'on_error'
abort_transfer(session_id)
Parameters:

session_id (str)

Return type:

None

check_object_exists(namespace, object_name)
Parameters:

  • namespace (str)

  • object_name (str)

Return type:

bool

complete_transfer(session_id, summary_hash)
Parameters:

  • session_id (str)

  • summary_hash (str)

Return type:

None

delete_namespace_storage_configuration(namespace)
Parameters:

namespace (str)

Return type:

None

delete_objects(namespace, object_paths)
Parameters:
Return type:

None

download_chunk(session_id, chunk_number, chunk_size, chunk_hash)
Parameters:

  • session_id (str)

  • chunk_number (int)

  • chunk_size (int)

  • chunk_hash (str)

Return type:

Tuple[bytes, Optional[str]]

get_namespace_object_paths(request)
Parameters:

request (ObjectPathsRequest)

Return type:

List[ObjectPath]

get_namespace_object_paths_slice(request)
Parameters:

request (ObjectPathsSliceRequest)

Return type:

Slice[ObjectPath]

get_namespace_storage_configurations()
Return type:

List[NamespaceStorageConfiguration]

get_namespaces()
Return type:

List[str]

get_object_detail(namespace, object_name)
Parameters:

  • namespace (str)

  • object_name (str)

Return type:

ObjectDetail

get_transfer_status(session_id)
Parameters:

session_id (str)

Return type:

TransferStatusResponse

put_namespace_storage_configuration(conf)
Parameters:

conf (NamespaceStorageConfiguration)

Return type:

NamespaceStorageConfiguration

start_download_session(namespace, object_download_request)
Parameters:
Return type:

ObjectDownloadResponse

start_upload_session(namespace, object_upload_request)
Parameters:
Return type:

str

upload_chunk(session_id, chunk_number, chunk_data, chunk_hash)
Parameters:

  • session_id (str)

  • chunk_number (str)

  • chunk_data (str)

  • chunk_hash (str)

Return type:

None

class yellowdog_client.object_store.ServiceSessionFacade
abort()
Return type:

None

complete(summary_hash)
Parameters:

summary_hash (str)

Return type:

None

create_thread(target)
Parameters:

target (Callable)

Return type:

BackgroundThread

dispatch_notification(event_handler, event_args)
Parameters:
Return type:

None

download_engine: AbstractTransferEngine = None
enqueue_chunk_download(chunk_download_task)
Parameters:

chunk_download_task (AbstractChunkDownloadTask)

Return type:

None

enqueue_chunk_upload(chunk_upload_task)
Parameters:

chunk_upload_task (AbstractChunkUploadTask)

Return type:

None

property server_chunks_transferred: List[int]
Return type:

List[int]

upload_engine: AbstractTransferEngine = None

yellowdog_client.object_store.upload – Upload services

class yellowdog_client.object_store.upload.ChunkUploadTask
class yellowdog_client.object_store.upload.UploadBatch(sessions)

Transfer batch used for upload

Added in version 0.5.0.

See also

Inherits from yellowdog_client.object_store.abstracts.AbstractTransferBatch

Parameters:

sessions (List[AbstractSession])

class yellowdog_client.object_store.upload.UploadBatchBuilder(upload_engine)

Builder class for creating batch upload for multiple files in directory

Added in version 0.5.0.

See also

Inherits from yellowdog_client.object_store.upload.abstracts.AbstractUploadBatchBuilder

Parameters:

upload_engine (AbstractUploadEngine)

find_source_objects(source_directory_path, source_file_pattern)

Iterates through all files found in directory and its subdirectories. If files match the pattern, they are appended to list of files for upload:

builder.find_source_objects("C:/my_files_for_upload", "*.txt")
Parameters:

  • source_directory_path (str) – Directory path to collect files for upload

  • source_file_pattern (str) – String, containing unix shell-style wildcards. See more at fnmatch

Return type:

None

get_batch_if_objects_found()

Creates an upload batch once namespace is set and upload objects found using find_source_objects(). Raises exception, if namespace is not set. Returns None, if no files are found for upload

Returns:

Upload batch, containing all upload sessions

Return type:

Optional[yellowdog_client.object_store.upload.UploadBatch]

Raises:

ValueError if namespace is not set

class yellowdog_client.object_store.upload.UploadEngine(service_proxy, thread_factory, file_reader_factory, notification_dispatcher, chunk_transfer_throttle, upload_thread_count)
Parameters:
build_upload_batch()
Return type:

AbstractUploadBatchBuilder

create_upload_session(file_namespace, source_file_path, destination_file_name, transfer_properties=None)
Parameters:
Return type:

AbstractSession

class yellowdog_client.object_store.upload.UploadSession(file_reader_factory, service_session_facade, file_path, file_size, chunk_size, chunk_count, file_retry_count)

Transfer session used for upload

Added in version 0.5.0.

See also

Inherits from yellowdog_client.object_store.abstracts.AbstractSession

Parameters:

  • file_reader_factory (MemoryMappedFileReaderFactory)

  • service_session_facade (ServiceSessionFacade)

  • file_path (str)

  • file_size (int)

  • chunk_size (int)

  • chunk_count (int)

  • file_retry_count (int)

yellowdog_client.object_store.upload.abstracts – Upload service abstracts

class yellowdog_client.object_store.upload.abstracts.AbstractUploadBatchBuilder

Abstract class for yellowdog_client.object_store.upload.UploadBatchBuilder

find_source_objects(source_directory_path, source_file_pattern)
Parameters:

  • source_directory_path (str)

  • source_file_pattern (str)

Return type:

None

get_batch_if_objects_found()
Return type:

UploadBatch

namespace: str = None

Object store namespace to use for transfer

object_name_mapper: Callable[[str], str] = None

Function, which renames file name for transfer

transfer_properties: TransferProperties = None

Transfer properties to use when creating a new batch

class yellowdog_client.object_store.upload.abstracts.AbstractUploadEngine(service_proxy, thread_factory, notification_dispatcher, chunk_transfer_throttle, transfer_thread_count)
Parameters:
build_upload_batch()
Return type:

AbstractUploadBatchBuilder

create_upload_session(file_namespace, source_file_path, destination_file_name, transfer_properties=None)
Parameters:
Return type:

AbstractSession

yellowdog_client.object_store.download – Download services

class yellowdog_client.object_store.download.ChunkDownloadTask
class yellowdog_client.object_store.download.DownloadBatch(sessions)

Transfer batch used for download

Added in version 0.5.0.

See also

Inherits from yellowdog_client.object_store.abstracts.AbstractTransferBatch

Parameters:

sessions (List[AbstractSession])

class yellowdog_client.object_store.download.DownloadBatchBuilder(download_engine, service_proxy)

Builder class for creating batch download from object store namespace

Added in version 0.5.0.

See also

Inherits from yellowdog_client.object_store.download.abstracts.AbstractDownloadBatchBuilder

Parameters:
find_source_objects(namespace, object_name_pattern)

Iterates through all files found in remote namespace and its subdirectories. If files match the pattern, they are appended to list of files for download:

builder.find_source_objects("MY_REMOTE_NAMESPACE", "*.txt")
Parameters:

  • namespace (str) – Object store namespace, containing required files

  • object_name_pattern (str) – String, containing unix shell-style wildcards. See more at fnmatch

Return type:

None

get_batch_if_objects_found()

Creates an download batch once download directory is set and download objects found using find_source_objects(). Raises exception, if download directory is not set. Returns None, if no files are found for download

Returns:

Download batch, containing all download sessions

Return type:

Optional[yellowdog_client.object_store.download.DownloadBatch]

Raises:

ValueError if destination_folder is not set

set_flatten_file_name_mapper(value)

Sets the method, which is used to rename files once they are downloaded in target directory

Parameters:

value (Optional[yellowdog_client.model.FlattenPath]) – enum value, indicating the method of path flattening

Return type:

None

class yellowdog_client.object_store.download.DownloadEngine(service_proxy, thread_factory, file_writer_factory, notification_dispatcher, chunk_transfer_throttle, download_thread_count)
Parameters:
build_download_batch()
Return type:

AbstractDownloadBatchBuilder

create_download_session(file_namespace, file_name, destination_folder_path, destination_file_name=None, transfer_properties=None)
Parameters:
Return type:

AbstractSession

class yellowdog_client.object_store.download.DownloadSession(file_reader_factory, service_session_facade, file_path, file_size, chunk_size, chunk_count, file_retry_count)

Transfer session used for download

Added in version 0.5.0.

See also

Inherits from yellowdog_client.object_store.abstracts.AbstractSession

Parameters:

  • file_reader_factory (MemoryMappedFileWriterFactory)

  • service_session_facade (ServiceSessionFacade)

  • file_path (str)

  • file_size (int)

  • chunk_size (int)

  • chunk_count (int)

  • file_retry_count (int)

ON_PROGRESS_EXTENSION: str = '.inprogress'

yellowdog_client.object_store.download.abstracts – Download service abstracts

class yellowdog_client.object_store.download.abstracts.AbstractDownloadBatchBuilder

Abstract class for yellowdog_client.object_store.download.DownloadBatchBuilder

Added in version 0.5.0.

destination_folder: str = None

Target directory, where files need to be downloaded

file_name_mapper: Callable[[str], str] = None

Custom function, which renames file name after download

find_source_objects(namespace, object_name_pattern)
Parameters:

  • namespace (str)

  • object_name_pattern (str)

Return type:

None

flatten_file_name_mapper: Callable[[str], str] | None = None

Custom function, which flattens file name after download

get_batch_if_objects_found()
Return type:

Optional[AbstractTransferBatch]

set_flatten_file_name_mapper(value)
Parameters:

value (Optional[FlattenPath])

Return type:

None

transfer_properties: TransferProperties = None

Transfer properties to use when creating a new batch

class yellowdog_client.object_store.download.abstracts.AbstractDownloadEngine(service_proxy, thread_factory, notification_dispatcher, chunk_transfer_throttle, transfer_thread_count)
Parameters:
build_download_batch()
Return type:

AbstractDownloadBatchBuilder

create_download_session(file_namespace, file_name, destination_folder_path, destination_file_name=None, transfer_properties=None)
Parameters:
Return type:

AbstractSession

yellowdog_client.object_store.model – Object store models

class yellowdog_client.object_store.model.BatchTransferEventArgs(status)

Arguments for batch transfer status

Parameters:

status (FileTransferStatus)

transfer_status: FileTransferStatus = None

Status of batch transfer

Type:

yellowdog_client.object_store.model.FileTransferStatus

class yellowdog_client.object_store.model.ClientErrorEventArgs(error_type, message, detail=None)

Arguments for client error, which occurred when using service

Parameters:
detail: Set[str] = None

Further details about the exception

Type:

Set[str]

error_type: ErrorType = None

Type of the error

Type:

yellowdog_client.model.ErrorType

message: str = None

Message, describing the exception

Type:

str

class yellowdog_client.object_store.model.FileTransferDirection(value)

Enum for file transfer direction

Download = 'Download'
Upload = 'Upload'
class yellowdog_client.object_store.model.FileTransferErrorEventArgs(full_path, file_name, transfer_status, error_type, message, detail=None)

Arguments for transfer error, which occurred when file was being uploaded or downloaded

See also

Inherits from yellowdog_client.object_store.model.FileTransferEventArgs

Parameters:
detail: Set[str] = None

Further details of the exception

Type:

Set[str]

error_type: ErrorType = None

Type of exception

Type:

yellowdog_client.model.ErrorType

message: str = None

Message of the exception

Type:

str

class yellowdog_client.object_store.model.FileTransferEventArgs(full_path, file_name, transfer_status)

Arguments for file transfer status

Parameters:
file_name: str = None

File name of file being transferred

Type:

str

full_path: str = None

Full file path to the file being transferred

Type:

str

transfer_status: FileTransferStatus = None

Current status of file transfer

Type:

yellowdog_client.object_store.model.FileTransferStatus

exception yellowdog_client.object_store.model.FileTransferException(error_type, message, detail=None)

Custom exception type for errors, related to object store. Inherits Exception

Parameters:
detail: Set[str] = None

Details of exception

Type:

Set[str]

error_type: ErrorType = None

Type of exception

Type:

yellowdog_client.model.ErrorType

class yellowdog_client.object_store.model.FileTransferProgressEventArgs(full_path, file_name, transfer_status, bytes_transferred, total_file_bytes, elapsed_time_delta)

Arguments for file transfer progress

See also

Inherits from yellowdog_client.object_store.model.FileTransferEventArgs

Parameters:
bytes_transferred: int = None

Total number of bytes transferred

Type:

int

elapsed_time: timedelta = None

Elapsed time since the beginning of transfer

Type:

datetime.timedelta

estimated_time_left: timedelta = None

Estimated remaining duration of transfer, based on transfer speed in transfer_speed_in_bits

Type:

datetime.timedelta

total_file_bytes: int = None

Total file size in bytes

Type:

int

transfer_percentage: float = None

Percentage of file transfer

Type:

float

transfer_speed_in_bits: float = None

Transfer speed in bits per second

Type:

float

class yellowdog_client.object_store.model.FileTransferStatus(value)

Enum for file transfer status

Aborted = 'Aborted'
Completed = 'Completed'
Downloading = 'Downloading'
Failed = 'Failed'
Ready = 'Ready'
Uploading = 'Uploading'
Validating = 'Validating'
is_active()

Method, which returns True, if status is considered active. The statuses are the following:

Returns:

True, if status is active

Return type:

bool

is_finished()

Method, which returns True, if status is considered finished. The statuses are the following:

Returns:

True, if status is finished

Return type:

bool

class yellowdog_client.object_store.model.TransferProperties
chunk_size: int = None
file_max_attempts: int = None
class yellowdog_client.object_store.model.TransferStatistics(bytes_transferred=None, total_bytes=None, elapsed_millis=None)

Calculated file transfer statistics

Parameters:
static build_new_empty()
Return type:

TransferStatistics

bytes_transferred: int = None

A number of bytes transferred

Type:

int

estimated_seconds_remaining: int = None

Estimated time remaining for transfer to finish

Type:

int

percentage_transferred: float = None

Percentage of batch transferred

Type:

float

total_bytes: int = None

A total number of bytes

Type:

int

transfer_speed_bits_per_second: int = None

Total transfer speed in bits per second

Type:

int

yellowdog_client.object_store.abstracts – Object store abstract classes

class yellowdog_client.object_store.abstracts.AbstractChunkDownloadTask
write_chunk_data: Callable[[str], None] = None
class yellowdog_client.object_store.abstracts.AbstractChunkTransferTask
abort_lock: allocate_lock = None
abort_token: CancellationToken = None
chunk_number: int = None
chunk_size: int = None
notify_chunk_transferred: Callable[[str], None] = None
notify_exception: Callable[[Exception], None] = None
session_id: str = None
transfer_countdown: CountdownEvent = None
class yellowdog_client.object_store.abstracts.AbstractChunkUploadTask
read_chunk_data: Callable[[], str] = None
class yellowdog_client.object_store.abstracts.AbstractNotificationDispatcher
dispatch(event_handler, event_args)
Parameters:
Return type:

None

class yellowdog_client.object_store.abstracts.AbstractObjectStoreServiceProxy(*args, **kwargs)
ON_ERROR = 'on_error'
abort_transfer(session_id)
Parameters:

session_id (str)

Return type:

None

check_object_exists(namespace, name)
Parameters:
Return type:

bool

complete_transfer(session_id, summary_hash)
Parameters:

  • session_id (str)

  • summary_hash (str)

Return type:

None

delete_namespace_storage_configuration(namespace)
Parameters:

namespace (str)

Return type:

None

delete_objects(namespace, object_paths)
Parameters:
Return type:

None

download_chunk(session_id, chunk_number, chunk_size, chunk_hash)
Parameters:

  • session_id (str)

  • chunk_number (int)

  • chunk_size (int)

  • chunk_hash (str)

Return type:

Tuple[bytes, Optional[str]]

get_namespace_object_paths(request)
Parameters:

request (ObjectPathsRequest)

Return type:

List[ObjectPath]

get_namespace_object_paths_slice(request)
Parameters:

request (ObjectPathsSliceRequest)

Return type:

List[ObjectPath]

get_namespace_storage_configurations()
Return type:

List[NamespaceStorageConfiguration]

get_object_detail(namespace, object_name)
Parameters:

  • namespace (str)

  • object_name (str)

Return type:

ObjectDetail

get_transfer_status(session_id)
Parameters:

session_id (str)

Return type:

TransferStatusResponse

put_namespace_storage_configuration(namespace_storage_configuration)
Parameters:

namespace_storage_configuration (NamespaceStorageConfiguration)

Return type:

None

start_download_session(namespace, object_download_request)
Parameters:
Return type:

ObjectDownloadResponse

start_upload_session(namespace, object_upload_request)
Parameters:
Return type:

str

upload_chunk(session_id, chunk_number, chunk_data, chunk_hash)
Parameters:

  • session_id (str)

  • chunk_number (int)

  • chunk_data (str)

  • chunk_hash (str)

Return type:

None

class yellowdog_client.object_store.abstracts.AbstractServiceSessionFacade
abort()
Return type:

None

complete(summary_hash)
Parameters:

summary_hash (str)

Return type:

None

create_thread(target)
Parameters:

target (Callable)

Return type:

BackgroundThread

dispatch_notification(event_handler, event_args)
Parameters:
Return type:

None

enqueue_chunk_download(chunk_download_task)
Parameters:

chunk_download_task (AbstractChunkTransferTask)

Return type:

None

enqueue_chunk_upload(chunk_upload_task)
Parameters:

chunk_upload_task (AbstractChunkTransferTask)

Return type:

None

notification_dispatcher: AbstractNotificationDispatcher = None
object_store_service_proxy: AbstractObjectStoreServiceProxy = None
property server_chunks_transferred: List[int]
Return type:

List[int]

session_id: str = None
thread_factory: BackgroundThreadFactory = None
class yellowdog_client.object_store.abstracts.AbstractSession(direction, service_session_facade, file_path, file_size, chunk_size, chunk_count, file_retry_count)

Base class for transfer session (upload and download). Inherits pydispatch.Dispatcher

Can be binded with callbacks on exceptions:

Parameters:

event_args (yellowdog_client.object_store.model.FileTransferErrorEventArgs) – session error arguments

Callback:

on_error

session.bind(on_error=lambda event_args: print(event_args.message))

Can be binded with callbacks on status changes:

Parameters:

event_args (yellowdog_client.object_store.model.FileTransferEventArgs) – session status change arguments

Callback:

on_status_changed

session.bind(on_status_changed=lambda event_args: print(event_args.transfer_status))

Can be binded with callbacks on session progress:

Parameters:

event_args (yellowdog_client.object_store.model.FileTransferProgressEventArgs) – session progress arguments

Callback:

on_progress

session.bind(on_progress=lambda event_args: print(event_args.transfer_status.transfer_percentage))

Added in version 0.5.0.

Parameters:
ON_ERROR = 'on_error'
ON_PROGRESS = 'on_progress'
ON_STATUS_CHANGED = 'on_status_changed'
abort()

Aborts any ongoing chunk transfers and prevents from continuing with new chunk transfers

Return type:

None

property bytes_transferred: int
Returns:

A number of bytes transferred during the session

Return type:

int

static calculate_chunk_offset(chunk_number, chunk_size)
static calculate_last_chunk_size(chunk_size, chunk_count, file_size)
Parameters:

  • chunk_size (int)

  • chunk_count (int)

  • file_size (int)

Return type:

int

close()

Closes the session and releases any resources related to file transfer

Return type:

None

direction: FileTransferDirection = None

File transfer direction

Type:

yellowdog_client.object_store.model.FileTransferDirection

property elapsed: timedelta
Returns:

Time duration, since the session was started

Return type:

datetime.timedelta

file_name: str = None

File name of object being transferred

Type:

str

file_path: str = None

File path of object being transferred

Type:

str

file_size: int = None

File size in bytes of object being transferred

Type:

int

get_statistics()

Calculates transfer statistics for transfer session

Returns:

Calculated statistics of file upload or download

Return type:

yellowdog_client.object_store.model.TransferStatistics

start()

Starts transfer session, allowing transfer engine to proceed with chunk upload or download

Return type:

None

property status: FileTransferStatus
Returns:

Latest status of transfer session

Return type:

yellowdog_client.object_store.model.FileTransferStatus

transfer_chunks_with_retries(enqueue_chunk_task_method)
Parameters:

enqueue_chunk_task_method (Callable[[AbstractChunkTransferTask], None])

Return type:

None

when_status_matches(status_predicate)

Assigns a session status predicate, which, when evaluates to True, sets a value for concurrent.futures.Future:

from concurrent import futures
from yellowdog_client.object_store.model import FileTransferStatus

future = session.when_status_matches(lambda status: status == FileTransferStatus.Completed)
futures.wait(fs=(future,))  # Wait for session status to match Completed
Parameters:

status_predicate (Callable[[yellowdog_client.object_store.model.FileTransferStatus], bool]) – file transfer status predicate to wait for

Returns:

a future to wait for session status to match a predicate

Return type:

concurrent.futures.Future

class yellowdog_client.object_store.abstracts.AbstractTransferBatch(transfer_direction, sessions)

Base class for transfer batch (a collection of multiple uploads or downloads). Inherits pydispatch.Dispatcher

Can be binded with callbacks on status changes:

Parameters:

event_args (yellowdog_client.object_store.model.BatchTransferEventArgs) – transfer batch status change arguments

Callback:

on_status_changed

batch.bind(on_status_changed=lambda event_args: print(event_args.transfer_status))

Added in version 0.5.0.

Parameters:
ON_STATUS_CHANGED = 'on_status_changed'
abort()

Aborts all transfer sessions, stopping any further chunk uploads or downloads

Return type:

None

add_session_error_listener(listener)

Binds event callback to all sessions within batch for errors

Parameters:

listener (Callable[[yellowdog_client.object_store.model.FileTransferErrorEventArgs], None]) – callback, which executes when any of session encounter error

batch.add_session_error_listener(lambda event_args: print(event_args.message))
Return type:

None

add_session_progress_listener(listener)

Binds event callback to all sessions within batch for progress changes

Parameters:

listener (Callable[[yellowdog_client.object_store.model.FileTransferProgressEventArgs], None]) – callback, which executes when any of session progress changes

batch.add_session_progress_listener(lambda event_args: print(event_args.transfer_status.transfer_percentage))
Return type:

None

add_session_status_listener(listener)

Binds event callback to all sessions within batch for status changes

Parameters:

listener (Callable[[yellowdog_client.object_store.model.FileTransferEventArgs], None]) – callback, which executes when any of session status changes

batch.add_session_status_listener(lambda event_args: print(event_args.transfer_status))
Return type:

None

direction: FileTransferDirection = None

Batch transfer direction

Type:

yellowdog_client.object_store.model.FileTransferDirection

get_statistics()

Calculates transfer statistics for batch transfer

Returns:

Calculated statistics of file upload or download for all sessions withing batch transfer

Return type:

yellowdog_client.object_store.model.TransferStatistics

get_transfer_sessions()
Returns:

A collection of all transfer sessions within the batch

Return type:

List[yellowdog_client.object_store.abstracts.AbstractSession]

start()

Starts batch transfer for uploads or downloads

Return type:

None

status: FileTransferStatus = None

Status of batch transfer

Return type:

yellowdog_client.object_store.model.FileTransferStatus

when_status_matches(status_predicate)

Assigns a transfer batch status predicate, which, when evaluates to True, sets a value for concurrent.futures.Future:

from concurrent import futures
from yellowdog_client.object_store.model import FileTransferStatus

future = batch.when_status_matches(lambda status: status == FileTransferStatus.Completed)
futures.wait(fs=(future,))  # Wait for batch status to match Completed
Parameters:

status_predicate (Callable[[yellowdog_client.object_store.model.FileTransferStatus], bool]) – file transfer status predicate to wait for

Returns:

a future to wait for batch status to match a predicate

Return type:

concurrent.futures.Future

class yellowdog_client.object_store.abstracts.AbstractTransferEngine(service_proxy, thread_factory, notification_dispatcher, chunk_transfer_throttle, transfer_thread_count)
Parameters:
DEFAULT_CHUNK_SIZE: int = 5242880
DEFAULT_FILE_RETRY_COUNT: int = 3
THREAD_TASK_MULTIPLIER: int = 5
TRANSFER_THREAD_STOP_WAIT_S: float = 0.1
abort_all_transfers()
Return type:

None

property active_sessions: List[AbstractSession]
Return type:

List[AbstractSession]

property all_sessions: List[AbstractSession]
Return type:

List[AbstractSession]

property chunk_size: int
Return type:

int

clear_inactive_sessions()
Return type:

None

enqueue_chunk_transfer_task(chunk_task)
Parameters:

chunk_task (AbstractChunkTransferTask)

Return type:

None

property file_retry_count: int
Return type:

int

property max_bytes_per_second: int
Return type:

int

start_transfers()
Return type:

None

stop_transfers()
Return type:

None

property transfer_threads_alive: bool
Return type:

bool