Versions Supported

Language Stats

Repo Stats

Activity

Build Status

PyS3Uploader

Python module to upload an entire directory to an S3 bucket.

Bucket Policy Required

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListBucketsForExistenceCheck",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        },
        {
            "Sid": "ListAndUploadToSpecificBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::bucketname"
        },
        {
            "Sid": "UploadObjectsToBucket",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::bucketname/*"
        }
    ]
}

Installation

pip install PyS3Uploader

Usage

Upload objects in parallel

import pys3uploader

if __name__ == '__main__':
    wrapper = pys3uploader.Uploader(
        bucket_name="BUCKET_NAME",
        upload_dir="FULL_PATH_TO_UPLOAD",
        exclude_prefix="PART_OF_UPLOAD_DIR_TO_EXCLUDE"
    )
    wrapper.run_in_parallel()

Upload objects in sequence

import pys3uploader

if __name__ == '__main__':
    wrapper = pys3uploader.Uploader(
        bucket_name="BUCKET_NAME",
        upload_dir="FULL_PATH_TO_UPLOAD",
        exclude_prefix="PART_OF_UPLOAD_DIR_TO_EXCLUDE"
    )
    wrapper.run()

Mandatory arg

• bucket_name - Name of the s3 bucket.

• upload_dir - Directory to upload.

Optional kwargs

• s3_prefix - S3 object prefix for each file. Defaults to None

• exclude_prefix - Path in upload_dir that has to be excluded in object keys. Defaults to None

• skip_dot_files - Boolean flag to skip dot files. Defaults to True

• overwrite - Boolean flag to overwrite files present in S3. Defaults to False

• file_exclusion - Sequence of files to exclude during upload. Defaults to None

• folder_exclusion - Sequence of directories to exclude during upload. Defaults to None

• metadata_upload_interval - Interval in seconds to upload metadata file. Defaults to 300

• metadata_filename - Metadata filename to upload periodically. Defaults to METADATA.json

• logger - Bring your own custom pre-configured logger. Defaults to on-screen logging.

• log_handler - Choose between stdout vs file logging. Defaults to pys3uploader.LogHandler.stdout

• log_level - Choose the logging level. Defaults to pys3uploader.LogLevel.debug

• env_file – Path to a .env file for loading environment variables. Defaults to scanning the current directory.
  
  

• region_name - AWS region name. Defaults to the env var AWS_DEFAULT_REGION

• profile_name - AWS profile name. Defaults to the env var PROFILE_NAME

• aws_access_key_id - AWS access key ID. Defaults to the env var AWS_ACCESS_KEY_ID

• aws_secret_access_key - AWS secret access key. Defaults to the env var AWS_SECRET_ACCESS_KEY

AWS values are loaded from env vars or the default config at ~/.aws/config / ~/.aws/credentials

Coding Standards

Docstring format: Google
Styling conventions: PEP 8
Clean code with pre-commit hooks: flake8 and isort

Release Notes

Requirement

python -m pip install gitverse

Usage

gitverse-release reverse -f release_notes.rst -t 'Release Notes'

Linting

pre-commit will ensure linting, run pytest, generate runbook & release notes, and validate hyperlinks in ALL markdown files (including Wiki pages)

Requirement

pip install sphinx==5.1.1 pre-commit recommonmark

Usage

pre-commit run --all-files

Pypi Package

https://pypi.org/project/PyS3Uploader/

Runbook

https://thevickypedia.github.io/PyS3Uploader/

License & copyright

© Vignesh Rao

Licensed under the MIT License