Victorio Berra

Posted on Jan 1, 2023

AWS Lightsail Container Services with Reverse Proxy

#aws #lightsail #containers #caddy

Including: SSL, custom domains and Route 53

Today I will show you how to leverage AWS Lightsail Container Services to create an environment where you have 2 public services behind a reverse proxy, and a database.

I host my domain in AWS Route 53. I use Lightsail Custom Domains to both assign custom domains to my container service, and to create a certificate.

Caddy? Logto?

The reverse proxy is Caddy

Caddy will proxy 2 apps:

An app called LogTo think self-hosted Auth0.
An app called whoami. This is a dead simple go app that spits back header and IP information.

In the real world, you would host an app instead of whoami that is protected with logto.

Lastly a Postgres container for this example. You should probably use the Lightsail Postgres database offering, at time of writing it is $10/mo.

Prerequisites

Grab the AWS CLI, configure it, and the lightsail cli plugin.
Install docker, I did this all on Windows 10 with WSL2.

I use a mix of the GUI and CLI. You could do it all in the GUI, or CLI.

I specify my AWS region as Ohio, because when I created my container service, it created it in Ohio (us-east-2). You might have to change your commands to reflect your region.

Buy a domain on Route53

Do not delete NS, or SOA records.

Create a container service

Head over to lightsail and create a container service, or use the CLI. Beware windows users, for the CLI I had to follow a guide here, and the CLI still wouldn't work for me. Ended up moving the lightsail exe to another bin folder for something like git to get it to work. I might have clobbered my environment vars with their setx command recommendation too...

A micro container service worked just fine for me. $10/mo at time of writing.

Create a cert, and some custom domains

For the cert and custom domains: Create example.com, www.example.com, logto.example.com and reserved.example.com. I created reserved in case I needed a new domain.

Create the DNS records, go to your lightsail instance and grab the Public domain (<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com) you will need the lightsail hosted zone id, list here: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-route-53-alias-record-for-container-service for example, for US East (Ohio) (us-east-2) it is Z10362273VJ548563IY84.

Create a file change-resource-record-sets.json this will be used by the AWS CLI to create a bunch of DNS records.

{
  "Comment": "Comment",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "example.com",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": "MYHOSTEDZONEID",
          "DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
          "EvaluateTargetHealth": true
        }
      }
    },
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "www.example.com",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": "MYHOSTEDZONEID",
          "DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
          "EvaluateTargetHealth": true
        }
      }
    },
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "logto.example.com",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": "MYHOSTEDZONEID",
          "DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
          "EvaluateTargetHealth": true
        }
      }
    },
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "reserved.example.com",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": "MYHOSTEDZONEID",
          "DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
          "EvaluateTargetHealth": true
        }
      }
    }
  ]
}

deploy with aws route53 change-resource-record-sets --hosted-zone-id MYHOSTEDZONEID --change-batch file://change-resource-record-sets.json

Don't forget to attach the cert to your instance!

Build and publish images to Lightsail Container Services Image Store

Create a Caddy, Logto, and Postgres docker file each in their own folders.

Caddy

Create a folder for /Caddy, create a file called Caddyfile, put the following in:

example.com:80 {
    reverse_proxy localhost:3001 {
        trusted_proxies private_ranges
    }
}

logto.example.com:80 {
    reverse_proxy localhost:3002 {
        trusted_proxies private_ranges
    }
}

Create the docker file /Caddy/Dockerfile

FROM caddy:latest

COPY Caddyfile /etc/caddy/Caddyfile

Logto

Create a docker file /Logto/Dockerfile

FROM ghcr.io/logto-io/logto:prerelease

ENTRYPOINT ["sh", "-c", "npm run cli db seed -- --swe && npm start"]

Notes: You need to call db seed --swe with LogTo so it will connect to Postgres, and create/seed the database. --swe means "skip when exists".

Postgres

Create a docker file /Postgres/Dockerfile

FROM postgres:14-alpine

Note: Yeah... I know. I had to do this. I could NOT just use registry.hub.docker.com/postgres:14-alpine, I had to create and push my own containers. Otherwise, Lightsail Container Services will refuse to start, the error log looks like this:

[1/Jan/2022:01:03:21] [deployment:8] Creating your deployment
[1/Jan/2022:01:04:31] [deployment:8] Started 1 new node
[1/Jan/2022:01:05:25] [deployment:8] Started 1 new node
[1/Jan/2022:01:06:34] [deployment:8] Started 1 new node
[1/Jan/2022:01:07:12] [deployment:8] Canceled

All the googling I did was unhelpful. I saw someone mention I should publish container images to my Lightsail container services image repo... and that worked.

Build

cd Caddy; docker build -t caddy-container .
cd Postgres; docker build -t postgres-container .
cd Logto; docker build -t logto-container .

Publish the images to Lightsail Container Registry

aws lightsail push-container-image --region us-east-2 --service-name myservice --label caddy-container --image caddy-container
aws lightsail push-container-image --region us-east-2 --service-name myservice --label postgres-container --image postgres-container
aws lightsail push-container-image --region us-east-2 --service-name myservice --label logto-container --image logto-container

Finally, create the container service

Create 2 JSON files. containers.json and public-endpoint.json

For containers.json:

NOTE: Environment variables are super important! Change them to whatever you need. Also image names!

{
  "whoami-3001": {
    "image": "registry.hub.docker.com/containous/whoami",
    "command": ["--port", "3001"]
  },
  "postgres": {
    "image": ":myservice.postgres-container.NUMBERHERE",
    "environment" : {
        "POSTGRES_USER": "postgresuser",
        "POSTGRES_PASSWORD": "yoursecretpassword1234"
    }
  },
  "logto": {
    "image": ":myservice.logto-container.NUMBERHERE",
    "environment" : {
        "TRUST_PROXY_HEADER": "1",
        "DB_URL": "postgres://postgresuser:yoursecretpassword1234@localhost:5432/logto",
        "ENDPOINT": "https://logto.example.com",
        "PORT": "3002",
        "NODE_ENV": "production"
    }
  },
  "caddy": {
    "image": ":myservice.caddy-container.NUMBERHERE",
    "ports": {
      "80": "HTTP"
    }
  }
}

for public-endpoint.json

{
  "containerName": "caddy",
  "containerPort": 80
}

Deploy!

aws lightsail create-container-service-deployment --region us-east-2 --service-name myservice --containers file://containers.json --public-endpoint file://public-endpoint.json

Questions/Pain-points

No way in UI to delete a certificate? What if I need to add a new subdomain to my cert? I saw the CLI has delete-certificate, I tried and got Distribution-related APIs are only available in the us-east-1 Region. Please set your Region configuration to us-east-1 to create, view, or edit distribution resources....
Installing the Amazon Lightsail container services plugin for Windows is difficult. Compare this to installing an Azure CLI extension. az extension add --name <extension-name>...
Postgres would not launch inside Lightsail when using public image (registry.hub.docker.com/postgres:14-alpine). I had to create my own blank docker file as shown above, and deploy it to the Lightsail container registry and it worked perfect. Still super confused by this one...
The public endpoint is doing SSL termination, and setting X-Forwarded-* headers in addition to allowing traffic from my custom domain. This means there is effectively a load balancer in front of my containers. But there does not seem to be docs explaining this. For example, Caddy is listening on port 80, and I expose HTTP port 80 as my public pot, but ALL traffic to my app comes in through SSL 443, and X-Forwarded-Proto is even https.
You can only have 4 custom domains pointed to your container service... if I hosted two more public apps in my container service, id be SOL.
What is a public port on a container exactly? My containers seem to be able to talk without me needing to use public ports. For example, logto can talk to Postgres without issue. Same with the "whoami" container. If I can set multiple public ports, then why can I only set a single public endpoint to one of these ports? What does public ports even do?

DNS notes

Route 53 is an easy powerful way to maintain your domain. Docs will advise you use Lightsail Container Services to host the domain, but unless you register your domain using Lightsail in the first place, you will need to wait 48 hours for the nameservers to update for Lightsail to have control.

I chose to keep my DNS on Route 53. But I did have issues with my Apex domain. I tried following the docs here https://Lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-Lightsail-route-53-alias-record-for-container-service

Somewhat odd they say:

"You can route traffic for a registered domain, such as example.com, to the applications running on a Lightsail container service. You do this by adding an alias record to the hosted zone of your domain that points to the default domain of your Lightsail container service. You can do this only by using the AWS Command Line Interface (AWS CLI). It cannot be done using the Route 53 console.".

They also say:

"To use the root of your domain with your Lightsail container service, you must specify an @ symbol in the subdomain space of your domain (for example, @.example.com)."

I tried this, and it did not work for me at all. In fact, simply creating an A record pointing to my Lightsail alias <service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com worked for me.

I created an issue on the AWS form (re:Post) and have not yet received a reply. https://repost.aws/questions/QU-y7P6lmzQZKj-QasRd8MyA/lightsail-container-services-with-an-apex-domain-in-route-53

DEV Community