Welcome to S3 Downloader’s documentation!

Read Me:

S3 Downloader

Main Module

class s3.dumper.Downloader(bucket_name: str, download_dir: Optional[str] = None, region_name: Optional[str] = None, profile_name: Optional[str] = None, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, logger: Optional[Logger] = None, prefix: Optional[str] = None)

Initiates Downloader object to download an entire S3 bucket.

>>> Downloader

Initiates all the necessary args and creates a boto3 session with retry logic.

Parameters:

  • bucket_name – Name of the bucket.

  • download_dir – Name of the download directory. Defaults to bucket name.

  • region_name – Name of the AWS region.

  • profile_name – AWS profile name.

  • aws_access_key_id – AWS access key ID.

  • aws_secret_access_key – AWS secret access key.

  • logger – Bring your own logger.

  • prefix – Specific path from which the objects have to be downloaded.

RETRY_CONFIG: Config = <botocore.config.Config object>
init() → None

Instantiates the bucket instance.

Raises:

  • ValueError – If no bucket name was passed.

  • BucketNotFound – If bucket name was not found.

exit() → NoReturn

Logs if there were any failures.

get_objects() → List[str]

Get all the objects in the target s3 bucket.

Raises:
Returns:

List of objects in the bucket.

Return type:

list

downloader(file: str) → None

Download the files in the exact path replacing spaces with underscores for the directory names.

Parameters:

file – Takes the filename as an argument.

run() → None

Initiates bucket download in a traditional loop.

run_in_parallel(threads: int = 5) → None

Initiates bucket download in multi-threading.

Parameters:

threads – Number of threads to use for downloading using multi-threading.

get_bucket_structure() → str

Gets all the objects in an S3 bucket and forms it into a hierarchical folder like representation.

Returns:

Returns a hierarchical folder like representation of the chosen bucket.

Return type:

str

print_bucket_structure() → None

Prints all the objects in an S3 bucket with a folder like representation.

Exceptions

Module to store all the custom exceptions and formatters.

>>> S3Error
exception s3.exceptions.S3Error

Custom error for base exception to the s3-downloader module.

exception s3.exceptions.BucketNotFound

Custom error for bucket not found.

exception s3.exceptions.NoObjectFound

Custom error for no objects found.

s3.exceptions.convert_to_folder_structure(sequence: Set[str]) → str

Convert objects in a s3 buckets into a folder like representation.

Parameters:

sequence – Takes either a mutable or immutable sequence as an argument.

Returns:

String representation of the architecture.

Return type:

str

exception s3.exceptions.InvalidPrefix(prefix: str, bucket_name: str, available: Set[str])

Custom exception for invalid prefix value.

Initialize an instance of InvalidPrefix object inherited from S3Error

Parameters:

  • prefix – Prefix to limit the objects.

  • bucket_name – Name of the S3 bucket.

  • available – Available objects in the s3.

format_error_message()

Returns the formatter error message as a string.

Logger

Loads a default logger with StreamHandler set to DEBUG mode.

>>> logging.Logger
s3.logger.default_handler() → StreamHandler

Creates a StreamHandler and assigns a default format to it.

Returns:

Returns an instance of the StreamHandler object.

Return type:

logging.StreamHandler

s3.logger.default_format() → Formatter

Creates a logging Formatter with a custom message and datetime format.

Returns:

Returns an instance of the Formatter object.

Return type:

logging.Formatter

s3.logger.default_logger() → Logger

Creates a default logger with debug mode enabled.

Returns:

Returns an instance of the Logger object.

Return type:

logging.Logger

Indices and tables