Either if you have already decided to use a PostgreSQL remote state backend for Terraform, or stumbled on this guide out of curiosity, let me help you set everything up in a more secure manner π!
edit 2024-01-20: Added a tip on workspaces.
Table of Contents
- Why?
- Prerequisites
- What Versions of Software are Used?
- Set Up a Postgres Server
- Configure a Terraform Client
- What to Do After a Reboot?
- The Result
Why?
When you work with Terraform by yourself, don't store sensitive data in state, and have a couple of projects, it's OK to use the local state backend.
But as soon as the number of projects increases, you start working in a team, and suddenly realize there're things like passwords and API keys in your state file, uhmm... Things start to get tricky π« .
A number of questions arise:
- How do I protect sensitive data stored in state?
- How can I collaborate with my colleagues? The state must be somehow shared in real time.
That's where a remote state backend comes in handy. It can be encrypted, allows to team up on the same projects simultaneously by having a SSOT for state, and supports locking to prevent collisions.
But Why PostgreSQL?
It's quicker and easier to set up than other remote state backends. It doesn't require a ton of resources. The thing just works, yet highly flexible.
Prerequisites
What you need is a single server and a way to obtain a TLS certificate for it. For me, it's a VM on Proxmox and a self-signed certificate.
If you want to use self-signed certs for this, check out my old article on how to generate them via openssl π.
Play around and find the exact amount of resources suitable for your specific case. Maybe less cores and RAM, more storage, etc.
Ubuntu was chosen as one of the most popular server distros. The configuration can be adjusted for any OS.
OS | CPU | RAM | Disk |
---|---|---|---|
Ubuntu Server | 2 | 2 GB | 8 GB + 1 GB |
π NOTE: For this demo, 1 GB disk is dedicated for a Postgres database cluster. Be aware, in production such disk size may be inappropriately small.
What Versions of Software are Used?
- Ubuntu 22.04.3 LTS
- PostgreSQL 14.10
- cryptsetup 2.4.3
- Terraform v1.5.7 (on a separate host)
What is LUKS, btw π€?
Check out cryptsetup's README to learn more: https://gitlab.com/cryptsetup/cryptsetup/
Set Up a Postgres Server
Prepare Storage
Create a LUKS Device
First, we need to find out our disk's block device name. In my case, it's /dev/sdb.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 63.5M 1 loop /snap/core20/2015
loop1 7:1 0 111.9M 1 loop /snap/lxd/24322
loop2 7:2 0 40.8M 1 loop /snap/snapd/20092
sda 8:0 0 8G 0 disk
ββsda1 8:1 0 7.9G 0 part /
ββsda14 8:14 0 4M 0 part
ββsda15 8:15 0 106M 0 part /boot/efi
sdb 8:16 0 1G 0 disk
sr0 11:0 1 4M 0 rom
Now, create a new partition table and a partition on this block device.
I prefer to use GPT instead of MBR even for smaller disks most of the time.
# fdisk /dev/sdb
fdisk example
Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x20d6b116.
Command (m for help): g
Created a new GPT disklabel (GUID: 0B0CD5AB-337A-CF47-A2BF-F377A158D0FE).
Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-2097118, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2097118, default 2097118):
Created a new partition 1 of type 'Linux filesystem' and of size 1023 MiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
When the partition is created, LUKS can be initialized using cryptsetup.
Considering that block device naming in Linux is not persistent between reboots, we won't rely on device names. Instead, we'll use labels for both a LUKS partition and a filesystem on it later.
# cryptsetup luksFormat /dev/sdb1 --type luks2 --label tfstate-pg-luks
π LUKS version 2 is used because it supports labels. Use any label of your liking. Don't forget to adjust all the commands and configurations.
Configure crypttab and fstab
Opening an encrypted disk partition requires providing a password or a key file. We could automate this process, but as always, there's a trade-off between security and convenience.
/etc/crypttab and /etc/fstab β these two files can greatly simplify the experience of opening and mounting an encrypted disk manually after reboots.
crypttab
Open /etc/crypttab with your editor of choice and add the line below:
tfstate-pg-luks LABEL=tfstate-pg-luks none luks,noauto
π NOTE: π The
noauto
option prevents automatic opening and mapping of the partition.
fstab
Do the same thing with /etc/fstab.
π NOTE: Don't forget to change the filesystem name here if you don't use Btrfs π.
You can use a single mount point, and mount the device straight to /var/lib/postgres. It's all a matter of preference. I like my devices to be mounted under /media, and then bind-mounted to other places.
LABEL=tfstate-pg /media/tfstate-pg btrfs defaults,noauto 0 0
/media/tfstate-pg /var/lib/postgres none bind,noauto 0 0
Create a Filesystem
Choose whatever filesystem you're comfortable with. Don't forget to update fstab accordingly.
To create a filesystem on the encrypted disk partition, connect to it first.
# cryptdisks_start tfstate-pg-luks
Create the filesystem itself with the label specified in fstab:
# mkfs.btrfs --label tfstate-pg
Mount and Prepare the Filesystem
At this point, we're halfway there.
Create directories for the mount points specified in /etc/fstab, and mount the LUKS-encrypted partition.
# mkdir /media/tfstate-pg /var/lib/postgres
# mount /media/tfstate-pg/
# mount /var/lib/postgres/
To check if everything is appropriately mounted, issue findmnt.
$ findmnt
Example output
$ findmnt
***OUTPUT TRUNCATED***
ββ/var/lib/postgres /dev/mapper/tfstate-pg-luks btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/
ββ/media/tfstate-pg /dev/mapper/tfstate-pg-luks btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/
The directory where the database cluster will be located must be owned by the postgres user and group.
# chown -R postgres: /var/lib/postgres
Configure Postgres
Install
# apt install postgresql
Disable and Stop the Service
Since the database cluster will be located on the encrypted partition that must be manually connected after a reboot, the Postgres service should be disabled.
# systemctl disable --now postgresql
π‘ TIP: π The
--now
option stops the daemon.
Copy a TLS Certificate and Key
Encrypted storage is useless if the network traffic between the server and its remote clients is not encrypted.
Grab a TLS certificate and its key, and copy them to your server. You can place them wherever you want β filenames don't matter either, as long as the postgres user has access.
# mv deathroll-internal.pem /etc/ssl/certs/
# mv deathroll-internal-key.pem /etc/ssl/private/
# chown postgres: /etc/ssl/private/deathroll-internal-key.pem
File permissions
$ stat --printf '%n\nMode: %a (%A)\nOwner: %u (%U)\nGroup: %g (%G)\n' /etc/ssl/certs/deathroll-internal.pem
/etc/ssl/certs/deathroll-internal.pem
Mode: 644 (-rw-r--r--)
Owner: 1000 (t_chuchkanov)
Group: 1000 (t_chuchkanov)
$ sudo -u postgres stat --printf '%n\nMode: %a (%A)\nOwner: %u (%U)\nGroup: %g (%G)\n' /etc/ssl/private/deathroll-internal-key.pem
/etc/ssl/private/deathroll-internal-key.pem
Mode: 600 (-rw-------)
Owner: 113 (postgres)
Group: 122 (postgres)
Configure
Settings
Instead of editing the config file manually, it's easier and quicker to use pg_conftool.
# pg_conftool set ssl_cert_file /etc/ssl/certs/deathroll-internal.pem
# pg_conftool set ssl_key_file /etc/ssl/private/deathroll-internal-key.pem
# pg_conftool set listen_addresses localhost,tfstate-pg.deathroll.internal
To check if all the values are correct:
$ for P in listen_addresses ssl_cert_file ssl_key_file; do pg_conftool show $P; done
Access Control
Configure access control for your Postgres installation. Append the line below to the /etc/postgresql/14/main/pg_hba.conf file:
π NOTE: π Pay attention to the Postgres version. It may differ in your case.
hostssl all all 172.17.0.0/24 scram-sha-256
π NOTE: π Change the allowed network to whatever you need. It may even be
0.0.0.0/0
.
Start
# systemctl start postgresql.service
If your configuration is correct, the Postgres systemd unit must be in the running state, the Postgres cluster online, and the daemon listening on the port 5432. Congrats π₯³.
$ systemctl list-units postgres*.service
Example
$ systemctl list-units postgres*.service
UNIT LOAD ACTIVE SUB DESCRIPTION
postgresql.service loaded active exited PostgreSQL RDBMS
postgresql@14-main.service loaded active running PostgreSQL Cluster 14-main
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
$ pg_lsclusters
Example
$ pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
14 main 5432 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
# ss -tlpn | grep postgres
Example
# ss -tlpn | grep postgres
LISTEN 0 244 172.17.0.40:5432 0.0.0.0:* users:(("postgres",pid=19728,fd=7))
LISTEN 0 244 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=19728,fd=6))
LISTEN 0 244 [::1]:5432 [::]:* users:(("postgres",pid=19728,fd=5))
Almost there! The last few steps of Postgres configuration are left.
Configure a Database
Now we need to:
- Change the postgres user password
- Create a Postgres user for Terraform clients
- Create a database for Terraform state
- Grant some privileges on this database to the Terraform user
Let's roll!
$ sudo -u postgres psql
π NOTE: π Depending on your $PWD, psql may throw a warning about its inability to access this directory. Ignore it.
postgres=# ALTER USER postgres ENCRYPTED PASSWORD '******';
postgres=# CREATE USER terraform ENCRYPTED PASSWORD '******';
postgres=# CREATE DATABASE terraform_backend;
postgres=# GRANT CREATE,CONNECT,TEMPORARY ON DATABASE terraform_backend TO terraform;
Configure a Firewall
Don't forget to configure a firewall on your system.
# ufw allow 22,5432/tcp
# ufw enable
To check the status:
# ufw status
That's it! It's time to test the connection from another host and configure a Terraform client π.
Check Connection from Another Host
Install psql and issue the command below:
$ psql postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend
π‘ TIP: The connection string scheme is:
postgres://<user>@<host>/<database_name>
π‘ TIP: On Ubuntu, the package is called "postgresql-client".
You can quit out of the psql client prompt right after it establishes the connection, because there's nothing to do in it apart from testing connectivity.
Example
$ psql postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend
Password for user terraform:
psql (15.5 (Debian 15.5-0+deb12u1), server 14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.
terraform_backend=> \dn
List of schemas
Name | Owner
--------+----------
public | postgres
(1 row)
terraform_backend=>
Configure a Terraform Client
π‘TIP: When you need to work with multiple terraform projects, use workspaces, since all of the freshly initialized projects have only the default workspace. Hence, there will be remote state conflicts, if a single workspace is used for several independent states.
In a Terraform project's main file, create a backend
block inside of terraform
. Provide a string value for conn_str.
It is really that simple.
terraform {
backend "pg" {
conn_str = "postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend"
}
}
Before issuing any Terraform commands, export the Postgres password as an environment variable:
$ read -s PGPASSWORD
$ export PGPASSWORD
For a fresh project, terraform init
is sufficient. For an existing project, add the -reconfigure
flag to this command.
$ terraform init
Initializing the backend...
Successfully configured the backend "pg"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Finding telmate/proxmox versions matching "2.9.11"...
- Installing telmate/proxmox v2.9.11...
- Installed telmate/proxmox v2.9.11 (unauthenticated)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
β·
β Warning: Incomplete lock file information for providers
β
β Due to your customized provider installation methods, Terraform was forced to calculate lock file checksums locally for the following providers:
β - telmate/proxmox
β
β The current .terraform.lock.hcl file only includes checksums for linux_amd64, so Terraform running on another platform will fail to install these providers.
β
β To calculate additional checksums for another platform, run:
β terraform providers lock -platform=linux_amd64
β (where linux_amd64 is the platform to generate)
β΅
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
If you want to learn how to use Terraform providers in isolated environments, I have a yet another article for you π.
Using Terraform Providers in Isolation
deathroll γ» Aug 6 '23
#devops #tutorial #opensource #terraform
What to Do After a Reboot?
- Open and map the LUKS-encrypted partition
- Mount its filesystem
- Start the Postgres service
As simple as three to four commands!
The Result
As the result, we have a PostgreSQL server with a database cluster located on a LUKS-encrypted partition.
This Postgres installation is used for storing remote Terraform state, and the network traffic between it and its remote clients is encrypted.
Sounds great, innit π€?
Scroll down to the comments section to get some enhancement tips.
Top comments (1)
P.S. You can consider encrypting or disabling swap on the Posgres server for better security.