Including: SSL, custom domains and Route 53
Today I will show you how to leverage AWS Lightsail Container Services to create an environment where you have 2 public services behind a reverse proxy, and a database.
I host my domain in AWS Route 53. I use Lightsail Custom Domains to both assign custom domains to my container service, and to create a certificate.
Caddy? Logto?
The reverse proxy is Caddy
Caddy will proxy 2 apps:
- An app called LogTo think self-hosted Auth0.
- An app called whoami. This is a dead simple go app that spits back header and IP information.
In the real world, you would host an app instead of whoami that is protected with logto.
Lastly a Postgres container for this example. You should probably use the Lightsail Postgres database offering, at time of writing it is $10/mo.
Prerequisites
- Grab the AWS CLI, configure it, and the lightsail cli plugin.
- Install docker, I did this all on Windows 10 with WSL2.
I use a mix of the GUI and CLI. You could do it all in the GUI, or CLI.
I specify my AWS region as Ohio, because when I created my container service, it created it in Ohio (us-east-2). You might have to change your commands to reflect your region.
Buy a domain on Route53
Do not delete NS, or SOA records.
Create a container service
Head over to lightsail and create a container service, or use the CLI. Beware windows users, for the CLI I had to follow a guide here, and the CLI still wouldn't work for me. Ended up moving the lightsail exe to another bin folder for something like git to get it to work. I might have clobbered my environment vars with their setx
command recommendation too...
A micro container service worked just fine for me. $10/mo at time of writing.
Create a cert, and some custom domains
For the cert and custom domains: Create example.com, www.example.com, logto.example.com and reserved.example.com. I created reserved in case I needed a new domain.
Create the DNS records, go to your lightsail instance and grab the Public domain (<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com
) you will need the lightsail hosted zone id, list here: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-route-53-alias-record-for-container-service for example, for US East (Ohio) (us-east-2)
it is Z10362273VJ548563IY84
.
Create a file change-resource-record-sets.json
this will be used by the AWS CLI to create a bunch of DNS records.
{
"Comment": "Comment",
"Changes": [
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "example.com",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "MYHOSTEDZONEID",
"DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
"EvaluateTargetHealth": true
}
}
},
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "www.example.com",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "MYHOSTEDZONEID",
"DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
"EvaluateTargetHealth": true
}
}
},
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "logto.example.com",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "MYHOSTEDZONEID",
"DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
"EvaluateTargetHealth": true
}
}
},
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "reserved.example.com",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "MYHOSTEDZONEID",
"DNSName": "<service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com",
"EvaluateTargetHealth": true
}
}
}
]
}
deploy with aws route53 change-resource-record-sets --hosted-zone-id MYHOSTEDZONEID --change-batch file://change-resource-record-sets.json
Don't forget to attach the cert to your instance!
Build and publish images to Lightsail Container Services Image Store
Create a Caddy, Logto, and Postgres docker file each in their own folders.
Caddy
Create a folder for /Caddy, create a file called Caddyfile
, put the following in:
example.com:80 {
reverse_proxy localhost:3001 {
trusted_proxies private_ranges
}
}
logto.example.com:80 {
reverse_proxy localhost:3002 {
trusted_proxies private_ranges
}
}
Create the docker file /Caddy/Dockerfile
FROM caddy:latest
COPY Caddyfile /etc/caddy/Caddyfile
Logto
Create a docker file /Logto/Dockerfile
FROM ghcr.io/logto-io/logto:prerelease
ENTRYPOINT ["sh", "-c", "npm run cli db seed -- --swe && npm start"]
Notes: You need to call db seed --swe
with LogTo so it will connect to Postgres, and create/seed the database. --swe
means "skip when exists".
Postgres
Create a docker file /Postgres/Dockerfile
FROM postgres:14-alpine
Note: Yeah... I know. I had to do this. I could NOT just use registry.hub.docker.com/postgres:14-alpine
, I had to create and push my own containers. Otherwise, Lightsail Container Services will refuse to start, the error log looks like this:
[1/Jan/2022:01:03:21] [deployment:8] Creating your deployment
[1/Jan/2022:01:04:31] [deployment:8] Started 1 new node
[1/Jan/2022:01:05:25] [deployment:8] Started 1 new node
[1/Jan/2022:01:06:34] [deployment:8] Started 1 new node
[1/Jan/2022:01:07:12] [deployment:8] Canceled
All the googling I did was unhelpful. I saw someone mention I should publish container images to my Lightsail container services image repo... and that worked.
Build
cd Caddy; docker build -t caddy-container .
cd Postgres; docker build -t postgres-container .
cd Logto; docker build -t logto-container .
Publish the images to Lightsail Container Registry
aws lightsail push-container-image --region us-east-2 --service-name myservice --label caddy-container --image caddy-container
aws lightsail push-container-image --region us-east-2 --service-name myservice --label postgres-container --image postgres-container
aws lightsail push-container-image --region us-east-2 --service-name myservice --label logto-container --image logto-container
Finally, create the container service
Create 2 JSON files. containers.json
and public-endpoint.json
For containers.json
:
NOTE: Environment variables are super important! Change them to whatever you need. Also image names!
{
"whoami-3001": {
"image": "registry.hub.docker.com/containous/whoami",
"command": ["--port", "3001"]
},
"postgres": {
"image": ":myservice.postgres-container.NUMBERHERE",
"environment" : {
"POSTGRES_USER": "postgresuser",
"POSTGRES_PASSWORD": "yoursecretpassword1234"
}
},
"logto": {
"image": ":myservice.logto-container.NUMBERHERE",
"environment" : {
"TRUST_PROXY_HEADER": "1",
"DB_URL": "postgres://postgresuser:yoursecretpassword1234@localhost:5432/logto",
"ENDPOINT": "https://logto.example.com",
"PORT": "3002",
"NODE_ENV": "production"
}
},
"caddy": {
"image": ":myservice.caddy-container.NUMBERHERE",
"ports": {
"80": "HTTP"
}
}
}
for public-endpoint.json
{
"containerName": "caddy",
"containerPort": 80
}
Deploy!
aws lightsail create-container-service-deployment --region us-east-2 --service-name myservice --containers file://containers.json --public-endpoint file://public-endpoint.json
Questions/Pain-points
- No way in UI to delete a certificate? What if I need to add a new subdomain to my cert? I saw the CLI has delete-certificate, I tried and got
Distribution-related APIs are only available in the us-east-1 Region. Please set your Region configuration to us-east-1 to create, view, or edit distribution resources.
... - Installing the Amazon Lightsail container services plugin for Windows is difficult. Compare this to installing an Azure CLI extension.
az extension add --name <extension-name>
... - Postgres would not launch inside Lightsail when using public image (
registry.hub.docker.com/postgres:14-alpine
). I had to create my own blank docker file as shown above, and deploy it to the Lightsail container registry and it worked perfect. Still super confused by this one... - The public endpoint is doing SSL termination, and setting
X-Forwarded-*
headers in addition to allowing traffic from my custom domain. This means there is effectively a load balancer in front of my containers. But there does not seem to be docs explaining this. For example, Caddy is listening on port 80, and I expose HTTP port 80 as my public pot, but ALL traffic to my app comes in through SSL 443, andX-Forwarded-Proto
is evenhttps
. - You can only have 4 custom domains pointed to your container service... if I hosted two more public apps in my container service, id be SOL.
- What is a public port on a container exactly? My containers seem to be able to talk without me needing to use public ports. For example, logto can talk to Postgres without issue. Same with the "whoami" container. If I can set multiple public ports, then why can I only set a single public endpoint to one of these ports? What does public ports even do?
DNS notes
Route 53 is an easy powerful way to maintain your domain. Docs will advise you use Lightsail Container Services to host the domain, but unless you register your domain using Lightsail in the first place, you will need to wait 48 hours for the nameservers to update for Lightsail to have control.
I chose to keep my DNS on Route 53. But I did have issues with my Apex domain. I tried following the docs here https://Lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-Lightsail-route-53-alias-record-for-container-service
Somewhat odd they say:
"You can route traffic for a registered domain, such as example.com, to the applications running on a Lightsail container service. You do this by adding an alias record to the hosted zone of your domain that points to the default domain of your Lightsail container service. You can do this only by using the AWS Command Line Interface (AWS CLI). It cannot be done using the Route 53 console.".
They also say:
"To use the root of your domain with your Lightsail container service, you must specify an @ symbol in the subdomain space of your domain (for example, @.example.com)."
I tried this, and it did not work for me at all. In fact, simply creating an A record pointing to my Lightsail alias <service-name>.<random-guid>.<aws-region-name>.cs.amazonLightsail.com
worked for me.
I created an issue on the AWS form (re:Post) and have not yet received a reply. https://repost.aws/questions/QU-y7P6lmzQZKj-QasRd8MyA/lightsail-container-services-with-an-apex-domain-in-route-53
Top comments (0)