Connect your Fly.io app to AWS RDS Postgres. Terraform complete walkthrough.

I built choremate.co with Phoenix and Elixir and the preferred PaaS for the stack nowadays is Fly.io, great CLI, and generous free tier. (Bonus, the creator of the Phoenix Framework works there.)

While their hosting and deployment experience is great, they do not offer a managed database option, so I had to look elsewhere: the good 'ol AWS RDS (Postgres).

I documented in this post all the necessary steps to use Terraform to create a Postgres instance on AWS and run an EC2 host with PGBouncer and Wireguard to connect your Fly.io app to AWS VPN through a tunnel.

This guide starts from this (repo)[https://github.com/fly-apps/rds-connector?tab=readme-ov-file] which at this point is outdated in some of their parts but still a very good starting point.

Overall we will be creating a Postgres instance not publicly accessible, an ec2 instance that will work as a bastion with PgBouncer functioning as a proxy and WireGuard installed so we'll be able to establish a tunnel between the Fly.io private network and our Aws private network.

Should look something like this

This guide will also make a couple of assumptions:

• you already have a web application running on Fly.io and an - you already have an active account on AWS
• you are working on a Mac Silicon machine (mostly impacts how you will install dependencies)

Required dependencies

brew install terraform
brew install awscli

Step 0.5 - AWS local Authentication

Not required but strongly suggested https://github.com/99designs/aws-vault allows you to manage your AWS credential on your local machine.

The instructions are pretty straightforward, but in case you need them, here is the link is where you create you can quickly create a root access key (I know you should use dedicated identity but I can't cover everything here)

Step 1 - Create SSH key-pair to use in our VPC

We'll reference those keys in our Terraform file so they will be used inside our VPC and will use it to ssh into our EC2 instance.

❯ ssh-keygen -b 4096 -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/Users/me/.ssh/id_rsa): aws_terraform
Enter passphrase (empty for no passphrase): #nothing here

❯ ls ~/.ssh
aws_terraform           aws_terraform.pub       config 

Step 2 - Generate the Wireguard configuration

Wireguard will be used to generate a VPN tunnel between the EC2 instance and the Fly.io VPN, enabling our app to send requests to our database without using the Internet.

fly wireguard create
flyctl wireguard create
? Select Organization:  [Use arrows to move, type to filter]
> ChoreMate (choremate)
❯
❯
❯  flyctl wireguard create choremate iad mypostgrespeer wg0.conf
? Select Organization: ChoreMate (choremate)
Creating WireGuard peer "mypostgrespeer" in region "iad" for organization choremate

!!!! WARNING: Output includes private key. Private keys cannot be recovered !!!!
!!!! after creating the peer; if you lose the key, you'll need to remove    !!!!
!!!! and re-add the peering connection.                                     !!!!
? Filename to store WireGuard configuration in, or 'stdout':

Will use this file in a moment.

Step 3 - Clone and customize the demo configuration

We'll keep everything in a single .tf file, but feel free to learn how to split a Terraform project into different files.

git clone git@github.com:xantrac/rds-fly-connector.git

the folder has the following structure

.
├── main.tf 
├── pgbouncer.ini.tmpl
├── terraform.tfvars
└── userlist.txt

• main.tf includes all the Terrform instructions to create our infrastructure
• pgbouncer.ini.tmpl is a template for pgbouncer configuration to run in our ec2 instance
• terraform.tfvars contains some variables that you will personalize
• userlist.txt is a list of postgres users that will be created with the database, defaults to postgres

3.1 Open the terraform.tfvars file and edit the existing placeholders:

aws_account = "cool-app"
vpc_name = "cool-app-vpc"
postgres_instance_name = "cool-app-prod"
database_name = "cool-app"

3.2 Copy your wg0.conf file to the root of the rds-fly-connector directory

cp ~/wg0.conf ~/rds-fly-connector #or wherever your files are located

Great, your Terraform file is ready.

Step 4 - Apply the terraform configuration

Will split this into two steps, first will plan the configuration.

From within the rds-fly-connector directory

aws-vault exec choremate -- terraform plan -out prod.tfstate

This command will analyze the configuration, fill in the templated variables, generate the final configuration object, and generate a .tfstate file which is the artifact of how our infrastructure will look at a given time.

Then will apply the .tfstate file to create our resources in AWS

aws-vault exec choremate -- terraform apply prod.tfstate

At this point, (likely 10 minutes later) Terraform is done applying the configuration. Let's verify the result.

• your database should be available at the following link https://us-east-1.console.aws.amazon.com/rds/home?region=us-east-1#databases:

• your ec2 bastion should be here https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Instances:instanceState=running

Then click on connect

and copy the public address

use the public address and the ssh key created in step 1 to connect to the bastion

❯  ssh -i ~/.ssh/aws_terraform  ubuntu@ec2-52-202-142-228.compute-1.amazonaws.com
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.2.0-1018-aws x86_64)

and attempt to connect to your database, you will need:

• username which is postgres unless you changed it
• password, can be found here https://us-east-1.console.aws.amazon.com/secretsmanager/listsecrets?region=us-east-1 (dig a little, should be easy)
• database address, on your database page https://us-east-1.console.aws.amazon.com/rds/home?region=us-east-1#databases: under Connectivity & security

putting al together we should have our postgres URL that looks like this

psql postgres://postgres:#{password}@#{database_url}:5432/#{database_name}

ubuntu@ip-172-16-101-225:~$ psql postgres://postgres:#########@############.us-east-1.rds.amazonaws.com:5432/chore_mate
psql (14.10 (Ubuntu 14.10-0ubuntu0.22.04.1), server 15.5)

If you see this everything is working, so far!

Let's check if WireGuard is working correctly when attempting to access the Postgres instance from within your Fly app.

When you create the WireGuard configuration in step 2 you also register the named peer within the Fly registry.

fly wireguard list

and the name you assigned to your configuration should be listed there.

Now because our ec2 bastion is running WireGuard and PgBouncer as a proxy our database should be reachable through that host at the #{peer_name}._peer.internal and we should be able to compose our postgres URL in this way.

postgres://postgres:#{password}@#{peer_name}._peer.internal:5432/#{db_name} 

fly ssh console

apt update
apt upgrade
apt install -y postgresql-client

 psql postgres://postgres:#{password}@#{peer_name}._peer.internal:5432/#{db_name}

psql (14.10 (Ubuntu 14.10-0ubuntu0.22.04.1), server 15.5)

If even this step worked you are done.

Step 5 - Add the DATABASE_URL env to your app

❯ fly secrets set DATABASE_URL=postgres://postgres:#{password}@#{peer_name}._peer.internal:5432/#{db_name} 

Enjoy your new AWS database!

P.S. Please let me know if you encounter any hiccups in the process so I can keep this guide updated.