Welcome to PyS3Uploader’s documentation!

Contents:

PyS3Uploader

class pys3uploader.uploader.Uploader(bucket_name: str, upload_dir: str, s3_prefix: str = None, exclude_prefix: str = None, skip_dot_files: bool = True, overwrite: bool = False, file_exclusion: ~typing.Iterable[str] = None, folder_exclusion: ~typing.Iterable[str] = None, region_name: str = None, profile_name: str = None, aws_access_key_id: str = None, aws_secret_access_key: str = None, retry_config: ~botocore.config.Config = <botocore.config.Config object>, logger: ~logging.Logger = None, log_handler: ~pys3uploader.logger.LogHandler = LogHandler.stdout, log_level: ~pys3uploader.logger.LogLevel = LogLevel.debug, env_file: str = None)

Initiates Uploader object to upload entire directory to S3.

>>> Uploader
__init__(bucket_name: str, upload_dir: str, s3_prefix: str = None, exclude_prefix: str = None, skip_dot_files: bool = True, overwrite: bool = False, file_exclusion: ~typing.Iterable[str] = None, folder_exclusion: ~typing.Iterable[str] = None, region_name: str = None, profile_name: str = None, aws_access_key_id: str = None, aws_secret_access_key: str = None, retry_config: ~botocore.config.Config = <botocore.config.Config object>, logger: ~logging.Logger = None, log_handler: ~pys3uploader.logger.LogHandler = LogHandler.stdout, log_level: ~pys3uploader.logger.LogLevel = LogLevel.debug, env_file: str = None)

Initiates all the necessary args and creates a boto3 session with retry logic.

Parameters:

  • bucket_name – Name of the bucket.

  • upload_dir – Full path of the directory to be uploaded.

  • s3_prefix – Particular bucket prefix within which the upload should happen.

  • exclude_prefix – Full directory path to exclude from S3 object prefix.

  • skip_dot_files – Boolean flag to skip dot files.

  • overwrite – Boolean flag to overwrite files in S3.

  • file_exclusion – Sequence of files to exclude during upload.

  • folder_exclusion – Sequence of directories to exclude during upload.

  • region_name – Name of the AWS region.

  • profile_name – AWS profile name.

  • aws_access_key_id – AWS access key ID.

  • aws_secret_access_key – AWS secret access key.

  • logger – Bring your own logger.

  • log_handler – Default log handler, can be file or stdout.

  • log_level – Default log level, can be debug, info, warning or error.

  • env_file – Dotenv file (.env) filepath to load environment variables.

See also

s3_prefix:

If provided, s3_prefix will always be attached to each object.

If s3_prefix is set to: 2025, then the file path /home/ubuntu/Desktop/S3Upload/sub/photo.jpg will be uploaded as 2025/S3Upload/sub/photo.jpg

exclude_prefix:

When upload directory is “/home/ubuntu/Desktop/S3Upload”, each file will naturally have the full prefix. However, this behavior can be avoided by specifying the exclude_prefix parameter.

If exclude_prefix is set to: /home/ubuntu/Desktop, then the file path /home/ubuntu/Desktop/S3Upload/sub-dir/photo.jpg will be uploaded as S3Upload/sub-dir/photo.jpg

env_file:

Environment variables can be loaded from a .env file. The filepath can be set as env_file during object instantiation or as an environment variable. If a filepath is provided, PyS3Uploader loads it directly or searches the root directory for the file. If no filepath is provided, PyS3Uploader searches the current directory for a .env file.

init() → None

Instantiates the bucket instance.

Raises:

  • ValueError – If no bucket name was passed.

  • BucketNotFound – If bucket name was not found.

exit() → None

Exits after printing results, and run time.

filesize(filepath: str) → int

Gets the file size of a given filepath.

Parameters:

filepath – Full path of the file.

Returns:

Returns the file size in bytes.

Return type:

int

_proceed_to_upload(filepath: str, objectpath: str) → bool

Compares file size if the object already exists in S3.

Parameters:

  • filepath – Source filepath.

  • objectpath – S3 object path.

Returns:

Returns a boolean flag to indicate upload flag.

Return type:

bool

_uploader(filepath: str, objectpath: str, callback: ProgressPercentage) → None

Uploads the filepath to the specified S3 bucket.

Parameters:

  • filepath – Filepath to upload.

  • objectpath – Object path ref in S3.

_get_files() → Dict[str, str]

Get a mapping for all the file path and object paths in upload directory.

Returns:

Returns a key-value pair of filepath and objectpath.

Return type:

Dict[str, str]

run() → None

Initiates object upload in a traditional loop.

run_in_parallel(max_workers: int = 5) → None

Initiates upload in multi-threading.

Parameters:

max_workers – Number of maximum threads to use.

get_bucket_structure() → str

Gets all the objects in an S3 bucket and forms it into a hierarchical folder like representation.

Returns:

Returns a hierarchical folder like representation of the chosen bucket.

Return type:

str

print_bucket_structure() → None

Prints all the objects in an S3 bucket with a folder like representation.

Exceptions

Module to store all the custom exceptions and formatters.

>>> S3Error
exception pys3uploader.exceptions.S3Error

Custom error for base exception to the PyS3Uploader module.

exception pys3uploader.exceptions.BucketNotFound

Custom error for bucket not found.

exception pys3uploader.exceptions.NoObjectFound

Custom error for no objects found.

pys3uploader.exceptions.convert_to_folder_structure(sequence: Set[str]) → str

Convert objects in a s3 buckets into a folder like representation.

Parameters:

sequence – Takes either a mutable or immutable sequence as an argument.

Returns:

String representation of the architecture.

Return type:

str

exception pys3uploader.exceptions.InvalidPrefix(prefix: str, bucket_name: str, available: Set[str])

Custom exception for invalid prefix value.

__init__(prefix: str, bucket_name: str, available: Set[str])

Initialize an instance of InvalidPrefix object inherited from S3Error

Parameters:

  • prefix – Prefix to limit the objects.

  • bucket_name – Name of the S3 bucket.

  • available – Available objects in the s3.

format_error_message()

Returns the formatter error message as a string.

Logger

Loads a default logger with StreamHandler set to DEBUG mode.

>>> logging.Logger
class pys3uploader.logger.LogHandler(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Logging handlers to choose from when default logger is used.

>>> LogHandler
file = 'file'
stdout = 'stdout'
class pys3uploader.logger.LogLevel(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Logging levels to choose from when default logger is used.

>>> LogLevel
debug = 10
info = 20
warning = 30
error = 40
classmethod _missing_(value)

Allow constructing from string names.

pys3uploader.logger.stream_handler() → StreamHandler

Creates a StreamHandler and assigns a default format to it.

Returns:

Returns an instance of the StreamHandler object.

Return type:

logging.StreamHandler

pys3uploader.logger.file_handler() → FileHandler

Creates a StreamHandler and assigns a default format to it.

Returns:

Returns an instance of the StreamHandler object.

Return type:

logging.StreamHandler

pys3uploader.logger.default_format() → Formatter

Creates a logging Formatter with a custom message and datetime format.

Returns:

Returns an instance of the Formatter object.

Return type:

logging.Formatter

pys3uploader.logger.setup_logger(handler: LogHandler, level: LogLevel)

Creates a default logger with debug mode enabled.

Returns:

Returns an instance of the Logger object.

Return type:

logging.Logger

Progress

class pys3uploader.progress.ProgressPercentage(filename: str, size: int, bar: alive_bar)

Tracks progress of a file upload to S3 and updates the alive_bar.

>>> ProgressPercentage
__init__(filename: str, size: int, bar: alive_bar)

Initializes the progress tracker.

Parameters:

  • filename – Name of the file being uploaded.

  • size – Total size of the file in bytes.

  • bar – alive_bar instance to update progress.

__call__(bytes_amount: int) → None

Callback method to update progress.

Parameters:

bytes_amount – Number of bytes transferred in the last chunk.

Tree

class pys3uploader.tree.Tree(skip_dot_files: bool)

Root tree formatter for a particular directory location.

This class allows the creation of a visual representation of the file system hierarchy of a specified directory. It can optionally skip files that start with a dot (hidden files).

>>> Tree
__init__(skip_dot_files: bool)

Instantiates the tree object.

Parameters:

skip_dot_files (bool) – If True, skips files with a dot prefix (hidden files).

scan(path: Path, last: bool = True, header: str = '') → List[str]

Returns contents for a folder as a root tree.

Parameters:

  • path – Directory path for which the root tree is to be extracted.

  • last – Indicates if the current item is the last in the directory.

  • header – The prefix for the current level in the tree structure.

Returns:

A list of strings representing the directory structure.

Return type:

List[str]

Utils

class pys3uploader.utils.UploadResults

Object to store results of S3 upload.

>>> UploadResults
success: int = 0
failed: int = 0
pys3uploader.utils.getenv(*args, default: str = None) → str

Returns the key-ed environment variable or the default value.

pys3uploader.utils.urljoin(*args) → str

Joins given arguments into a url. Trailing but not leading slashes are stripped for each argument.

Returns:

Joined url.

Return type:

str

pys3uploader.utils.convert_to_folder_structure(sequence: Set[str]) → str

Convert objects in a s3 buckets into a folder like representation.

Parameters:

sequence – Takes either a mutable or immutable sequence as an argument.

Returns:

String representation of the architecture.

Return type:

str

pys3uploader.utils.convert_seconds(seconds: int | float, n_elem: int = 2) → str

Calculate years, months, days, hours, minutes, seconds, and milliseconds from given input.

Parameters:

  • seconds – Number of seconds to convert (supports float values).

  • n_elem – Number of elements required from the converted list.

Returns:

Returns a humanized string notion of the number of seconds.

Return type:

str

pys3uploader.utils.format_nos(input_: float) → int | float

Removes .0 float values.

Parameters:

input_ – Strings or integers with .0 at the end.

Returns:

Int if found, else returns the received float value.

Return type:

int | float

pys3uploader.utils.size_converter(byte_size: int | float) → str

Gets the current memory consumed and converts it to human friendly format.

Parameters:

byte_size – Receives byte size as argument.

Returns:

Converted understandable size.

Return type:

str

Indices and tables