How to create a new bucket and add files to a specific folder using local stack (running in Docker)

4 min readOct 12, 2020

First copy the contents of this docker-compose.yml file listed in the localstack repository into a local directory (make sure the file name is not changed).

Run up the container with docker-compose up. This will get localstack running locally.

At the time of writing this post (Oct 2020), the S3 service is available on port 4566 (i.e. http://localhost:4566)

Install the aws / localstack wrapper

Install awscli-local from here as a wrapper around localstack. This will allow you to use awslocal in your command line, which is very close to the aws command usage.

Create a new bucket

awslocal s3api create-bucket --bucket business-time-nonprod --region eu-west-1# You don't need the region flag in here, but adding one doesn't hurt

Create a new folder inside your newly created bucket

The easiest way I found to create a new folder is to use the put-object api.

awslocal s3api put-object --bucket business-time-nonprod --key some/folder/name/here/
# Make sure you don't forget the trailing / after here. 
# Missing it will mean that aws will treat it as a file and not a folder.

UPDATE: You don’t need to create a folder like this ^ before adding a file. Everything in S3 is an object, and if you create a folder like this before creating a file, when you listObject on the bucket, you will see 2 items which might not be ideal to your use-case. You can add a file to a folder directly(without creating it first) using the following command:

awslocal s3 co helloworld.txt s3://some-bucket/prefix/

Add a new file to it

We’re going to copy over a file into the nested here folder you just created.

touch helloworld.txt # Create a file if you don't already have itawslocal s3 cp helloworld.txt s3://business-time-nonprod/some/folder/name/here/

Verify that your new bucket exists

awslocal s3api list-buckets --query "Buckets[].Name"

Verify that the file you just added exists

awslocal s3 ls s3://business-time-nonprod/some/folder/name/here --recursive --human-readable --summarize

Add tags to your file

awslocal s3api put-object-tagging \
    --bucket business-time-nonprod \
    --key helloworld.txt \
    --tagging '{"TagSet": [{ "Key": "version", "Value": "1.0.0" }]}'

Check the tags on a file

awslocal s3api get-object-tagging \
  --bucket business-time-nonprod \
  --key helloworld.txt

How to automatically create buckets and folders

If you’re using localstack in your project, and would like to create a certain bucket, or folders within that bucket when you start up localstack, you will need to modify the docker-compose.yml file I’ve listed above to include these values:

// docker-compose.ymlenvironment:
- SERVICES=s3
- DATA_DIR=/tmp/localstack/data...volumes:
- ./stubs/s3:/tmp/localstack
- /var/run/docker.sock:/var/run/docker.sock

With DATA_DIR=/tmp/localstack/data , we’re asking localstack to use tmp/localstack/data as the directory to store data inside the container. Then, we create a volume to sync all the data between the docker container and our local machine using ./stubs/s3:/tmp/localstack . If Docker doesn’t automatically create this /stubs/s3 folder for you, you might need to create it yourself.

Now, with the above changes to the docker-compose.yml file, run up the container using docker-compose up. With the localstack container up, create a new bucket called test:

awslocal s3api create-bucket --bucket test

Now, create a folder called /data inside the bucket.

awslocal s3api put-object --bucket test --key data/

You will notice you have a new file within the /stubs/s3/data folder called recorded_pi_calls.json which contains 2 new lines, which looks something like this:

{"a": "s3", "m": "PUT", "p": "/test", "d": "", "h": {"Remote-Addr": "172.17.0.1", "host": "localhost", "Accept-Encoding": "identity", "User-Agent": "aws-cli/2.0.54 Python/3.7.4 Darwin/19.5.0 exe/x86_64 command/s3api.create-bucket", "X-Amz-Date": "20201017T124953Z", "X-Amz-Content-Sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "Authorization": "AWS4-HMAC-SHA256 Credential=_not_needed_locally_/20201017/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=53ac9510e79cfd96ed389f1f649e20e748e511e7a2817f450ccdc7695c2df992", "Content-Length": "0", "X-Forwarded-For": "172.17.0.1, localhost:4566, 127.0.0.1, localhost:4566", "x-localstack-edge": "https://localhost:4566", "content-type": "binary/octet-stream", "Connection": "close"}, "rd": "PENyZWF0ZUJ1Y2tldFJlc3BvbnNlIHhtbG5zPSJodHRwOi8vczMuYW1hem9uYXdzLmNvbS9kb2MvMjAwNi0wMy0wMSI+PENyZWF0ZUJ1Y2tldFJlc3BvbnNlPjxCdWNrZXQ+dGVzdDwvQnVja2V0PjwvQ3JlYXRlQnVja2V0UmVzcG9uc2U+PC9DcmVhdGVCdWNrZXRSZXNwb25zZT4="}{"a": "s3", "m": "PUT", "p": "/test/data/", "d": "", "h": {"Remote-Addr": "172.17.0.1", "host": "localhost", "Accept-Encoding": "identity", "User-Agent": "aws-cli/2.0.54 Python/3.7.4 Darwin/19.5.0 exe/x86_64 command/s3api.put-object", "X-Amz-Date": "20201017T125025Z", "X-Amz-Content-Sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "Authorization": "AWS4-HMAC-SHA256 Credential=_not_needed_locally_/20201017/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7a53668d9719cde83f6ac0a100b3e1a14eca25d568bca1f63a20cb11695b2d7b", "Content-Length": "0", "X-Forwarded-For": "172.17.0.1, localhost:4566, 127.0.0.1, localhost:4566", "x-localstack-edge": "https://localhost:4566", "content-type": "binary/octet-stream", "Connection": "close"}, "rd": ""}

These are the lines that are responsible for creating the resources (buckets and folders that you need).

Important: Make sure you commit the recorded_api_calls.json file at this point to version control, so if others in your team (or you cloning this repo on another computer) will have access to these. In the same vein, make sure you’re not committing unnecessary changes to this file that do not reflect the state of the S3 buckets / folders you want when you spin up localstack.