Run multiple AWS CLI commands in Docker Compose

Recently I started working on modernizing one of the microservice dev setup. I wanted to use of Docker compose to run automatic tasks needed before starting project.

As the title suggests, I'm using AWS stack in my project and AWS CLI is the important part of the whole setup.

And for local AWS development, there is nothing better than localstack.

This it is what a docker compose file looks like using use localstack and AWS CLI for service initialization:

  localstack:
    container_name: '${LOCALSTACK_DOCKER_NAME-localstack_main}'
    image: localstack/localstack:latest
    ports:
      - '127.0.0.1:4566:4566' # LocalStack Gateway
      - '127.0.0.1:4510-4559:4510-4559' # external services port range
    environment:
      - DEBUG=${DEBUG-}
      - DOCKER_HOST=unix:///var/run/docker.sock
      - S3_DIR=/tmp/s3-buckets
      - PERSISTENCE=1
    volumes:
      - '${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'
      - './s3-buckets:/tmp/s3-buckets'
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:4566/_localstack/health']
      interval: 30s
      timeout: 10s
      retries: 5

  # Service to initialize AWS resources
  aws-init:
    image: amazon/aws-cli:latest
    depends_on:
      localstack:
        condition: service_healthy
    environment:
      - AWS_ACCESS_KEY_ID=local
      - AWS_SECRET_ACCESS_KEY=stack
      - AWS_DEFAULT_REGION=eu-west-1
      - AWS_ENDPOINT_URL=http://localstack:4566
    command: |
      sh -c '
        echo "Waiting for LocalStack to be ready..."
        sleep 10
        echo "Creating S3 bucket..."
        aws --endpoint-url=http://localstack:4566 s3api create-bucket \
          --bucket bucket-1 || true
        aws --endpoint-url=http://localstack:4566 s3api create-bucket \
          --bucket bucket-2 || true
        echo "Creating DynamoDB table..."
        aws --endpoint-url=http://localstack:4566 dynamodb create-table \
          --cli-input-json file:///tmp/ddb.skeleton.json \
          --region eu-west-1 || true
        echo "AWS resources initialized successfully!"
      '
...

Once we execute this docker compose file we come to realize that AWS init does not succeed. It waits for a command and then fails because of not receiving enough arguments.

The real problem for this happening is that AWS CLI Docker image is built in a way that the AWS CLI is THE entrypoint.

That means you could send the commands directly to the AWS CLI Docker image. And following is the right way to use the AWS CLI in docker compose (if you have just one command)

aws-init:
   image: amazon/aws-cli:latest
   depends_on:
     localstack:
       condition: service_healthy
   environment:
     - AWS_ACCESS_KEY_ID=local
     - AWS_SECRET_ACCESS_KEY=stack
     - AWS_DEFAULT_REGION=eu-west-1
     - AWS_ENDPOINT_URL=http://localstack:4566
   command: --endpoint-url=http://localstack:4566 s3api create-bucket \
         --bucket bucket-1
...

But if you want multiple commands to be executed, there is no direct way to do it.

If you want to execute that Docker image on the system, you could execute multiple commands directly. It's more convenient.

But when it comes to Docker compose, you cannot do that.

Solution

The solution for this problem comes with the fact that all the container images are built on some base system.

AWS CLI is built on top of Linux with Python already installed.

With Linux as a base system we could assume that there is some kind of shell already in place.

AWS CLI comes with sh shell already installed.

With AWS CLI docker image, we will manually change the entrypoint of the container. Then, we can send more than one commands to that entrypoint.

It would look like the following.

...
entrypoint:
     - /bin/sh
   command: |
     -c '
       echo "Waiting for LocalStack to be ready..."
       sleep 10
       echo "Creating S3 bucket..."
       # ...
       '
...

And with the above example changes we will change our compose file to the following.

aws-init:
   image: amazon/aws-cli:latest
   depends_on:
     localstack:
       condition: service_healthy
   environment:
     - AWS_ACCESS_KEY_ID=local
     - AWS_SECRET_ACCESS_KEY=stack
     - AWS_DEFAULT_REGION=eu-west-1
     - AWS_ENDPOINT_URL=http://localstack:4566
   entrypoint:
     - /bin/sh
   command: |
     -c '
       echo "Waiting for LocalStack to be ready..."
       sleep 10
       echo "Creating S3 bucket..."
       aws --endpoint-url=http://localstack:4566 s3api create-bucket \
         --bucket bucket-1 || true
       aws --endpoint-url=http://localstack:4566 s3api create-bucket \
         --bucket bucket-2 || true
       echo "Creating DynamoDB table..."
       aws --endpoint-url=http://localstack:4566 dynamodb create-table \
         --cli-input-json file:///tmp/ddb.skeleton.json \
         --region eu-west-1 || true
       echo "AWS resources initialized successfully!"
     '

Note again, we dont need to type th sh in the command anymore. The command is piped to the entrypoint, and hence it will not be needed.

Credits

• Photo by Sigmund on Unsplash