gompress

CLI tool to compress the contents of AWS S3 bucket.

Gompress takes files from the original location, compresses them on the fly using gzip, and uploads to their destination, keeping the original names (only adding .gz).

Gompress never keeps large files completely in memory, and never writes them to disk. It streams data from source into destination, adding a compression layer in the middle.

It is recommended to become familiar with AWS S3 pricing before using this utility. Moving files around costs money. To reduce the cost, use EC2 instances in the same region as the buckets!

Usage

Running in a docker container:

docker run --rm \
    -e AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY \
    orlangure/gompress \
    -src-region us-east-1 \
    -src-bucket uncompressed-files \
    -src-prefix some-folder/and-sub-folder \
    -dst-region us-east-1 \
    -dst-bucket compressed-files \
    -dst-prefix a-folder \
    -keep

AWS credentials with sufficient permissions to list, get, create and delete objects are required. AWS key and secret should be provided using environment variables.

When -keep flag is provided, original files remain in their original location. Otherwise, when -keep flag is not set, original files are removed once their compressed versions are successfully uploaded to the destination path.

Building from source (Go 1.13 or above is required):

$ git clone https://github.com/orlangure/gompress
$ cd gompress
$ go install .

$ gompress
Usage of gompress:
  -dst-bucket string
        target s3 bucket name
  -dst-prefix string
        new files will be prefixed with this value
  -dst-region string
        target region (default "us-east-1")
  -keep
        set to keep original files (remove by default)
  -src-bucket string
        source s3 bucket name
  -src-prefix string
        source file prefix
  -src-region string
        source region (default "us-east-1")