DEV Community

Cover image for Encrypted Remote Terraform State with Postgres and LUKS
deathroll
deathroll

Posted on • Edited on

Encrypted Remote Terraform State with Postgres and LUKS

#devops #security #postgres #terraform

Either if you have already decided to use a PostgreSQL remote state backend for Terraform, or stumbled on this guide out of curiosity, let me help you set everything up in a more secure manner 😁!

edit 2024-01-20: Added a tip on workspaces.

Table of Contents

Why?

When you work with Terraform by yourself, don't store sensitive data in state, and have a couple of projects, it's OK to use the local state backend.

But as soon as the number of projects increases, you start working in a team, and suddenly realize there're things like passwords and API keys in your state file, uhmm... Things start to get tricky 🫠.

A number of questions arise:

  • How do I protect sensitive data stored in state?
  • How can I collaborate with my colleagues? The state must be somehow shared in real time.

That's where a remote state backend comes in handy. It can be encrypted, allows to team up on the same projects simultaneously by having a SSOT for state, and supports locking to prevent collisions.

But Why PostgreSQL?

It's quicker and easier to set up than other remote state backends. It doesn't require a ton of resources. The thing just works, yet highly flexible.

Prerequisites

What you need is a single server and a way to obtain a TLS certificate for it. For me, it's a VM on Proxmox and a self-signed certificate.

If you want to use self-signed certs for this, check out my old article on how to generate them via openssl πŸ‘‡.

deathroll

Trusted self-signed TLS certificates for dummies (w/ thorough explanations included)

deathroll ・ Nov 25 '22

#security #https #ssl #beginners

Play around and find the exact amount of resources suitable for your specific case. Maybe less cores and RAM, more storage, etc.

Ubuntu was chosen as one of the most popular server distros. The configuration can be adjusted for any OS.

OS CPU RAM Disk
Ubuntu Server 2 2 GB 8 GB + 1 GB

πŸ“’ NOTE: For this demo, 1 GB disk is dedicated for a Postgres database cluster. Be aware, in production such disk size may be inappropriately small.

What Versions of Software are Used?

  • Ubuntu 22.04.3 LTS
  • PostgreSQL 14.10
  • cryptsetup 2.4.3
  • Terraform v1.5.7 (on a separate host)

What is LUKS, btw πŸ€”?
Check out cryptsetup's README to learn more: https://gitlab.com/cryptsetup/cryptsetup/

Set Up a Postgres Server

Prepare Storage

Create a LUKS Device

First, we need to find out our disk's block device name. In my case, it's /dev/sdb.



$ lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0  63.5M  1 loop /snap/core20/2015
loop1     7:1    0 111.9M  1 loop /snap/lxd/24322
loop2     7:2    0  40.8M  1 loop /snap/snapd/20092
sda       8:0    0     8G  0 disk 
β”œβ”€sda1    8:1    0   7.9G  0 part /
β”œβ”€sda14   8:14   0     4M  0 part 
└─sda15   8:15   0   106M  0 part /boot/efi
sdb       8:16   0     1G  0 disk 
sr0      11:0    1     4M  0 rom
Enter fullscreen mode Exit fullscreen mode

Now, create a new partition table and a partition on this block device.

I prefer to use GPT instead of MBR even for smaller disks most of the time.



# fdisk /dev/sdb
Enter fullscreen mode Exit fullscreen mode

fdisk example 


Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x20d6b116.

Command (m for help): g
Created a new GPT disklabel (GUID: 0B0CD5AB-337A-CF47-A2BF-F377A158D0FE).

Command (m for help): n
Partition number (1-128, default 1): 
First sector (2048-2097118, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2097118, default 2097118): 

Created a new partition 1 of type 'Linux filesystem' and of size 1023 MiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Enter fullscreen mode Exit fullscreen mode

When the partition is created, LUKS can be initialized using cryptsetup.

Considering that block device naming in Linux is not persistent between reboots, we won't rely on device names. Instead, we'll use labels for both a LUKS partition and a filesystem on it later.



# cryptsetup luksFormat /dev/sdb1 --type luks2 --label tfstate-pg-luks
Enter fullscreen mode Exit fullscreen mode

πŸ‘† LUKS version 2 is used because it supports labels. Use any label of your liking. Don't forget to adjust all the commands and configurations.

Configure crypttab and fstab

Opening an encrypted disk partition requires providing a password or a key file. We could automate this process, but as always, there's a trade-off between security and convenience.

/etc/crypttab and /etc/fstab β€” these two files can greatly simplify the experience of opening and mounting an encrypted disk manually after reboots.

crypttab

Open /etc/crypttab with your editor of choice and add the line below:



tfstate-pg-luks LABEL=tfstate-pg-luks   none    luks,noauto
Enter fullscreen mode Exit fullscreen mode

πŸ“’ NOTE: πŸ‘† The noauto option prevents automatic opening and mapping of the partition.

fstab

Do the same thing with /etc/fstab.

πŸ“’ NOTE: Don't forget to change the filesystem name here if you don't use Btrfs πŸ‘‡.

You can use a single mount point, and mount the device straight to /var/lib/postgres. It's all a matter of preference. I like my devices to be mounted under /media, and then bind-mounted to other places.



LABEL=tfstate-pg        /media/tfstate-pg       btrfs   defaults,noauto 0       0
/media/tfstate-pg       /var/lib/postgres       none    bind,noauto     0       0
Enter fullscreen mode Exit fullscreen mode

Create a Filesystem

Choose whatever filesystem you're comfortable with. Don't forget to update fstab accordingly.

To create a filesystem on the encrypted disk partition, connect to it first.



# cryptdisks_start tfstate-pg-luks
Enter fullscreen mode Exit fullscreen mode

Create the filesystem itself with the label specified in fstab:



# mkfs.btrfs --label tfstate-pg
Enter fullscreen mode Exit fullscreen mode

Mount and Prepare the Filesystem

At this point, we're halfway there.

Create directories for the mount points specified in /etc/fstab, and mount the LUKS-encrypted partition.



# mkdir /media/tfstate-pg /var/lib/postgres
# mount /media/tfstate-pg/
# mount /var/lib/postgres/
Enter fullscreen mode Exit fullscreen mode

To check if everything is appropriately mounted, issue findmnt.



$ findmnt
Enter fullscreen mode Exit fullscreen mode

Example output 


$ findmnt
***OUTPUT TRUNCATED***
β”œβ”€/var/lib/postgres                           /dev/mapper/tfstate-pg-luks btrfs       rw,relatime,space_cache=v2,subvolid=5,subvol=/
└─/media/tfstate-pg                           /dev/mapper/tfstate-pg-luks btrfs       rw,relatime,space_cache=v2,subvolid=5,subvol=/
Enter fullscreen mode Exit fullscreen mode

The directory where the database cluster will be located must be owned by the postgres user and group.



# chown -R postgres: /var/lib/postgres
Enter fullscreen mode Exit fullscreen mode

Configure Postgres

Install 



# apt install postgresql
Enter fullscreen mode Exit fullscreen mode

Disable and Stop the Service

Since the database cluster will be located on the encrypted partition that must be manually connected after a reboot, the Postgres service should be disabled.



# systemctl disable --now postgresql
Enter fullscreen mode Exit fullscreen mode

πŸ’‘ TIP: πŸ‘† The --now option stops the daemon.

Copy a TLS Certificate and Key

Encrypted storage is useless if the network traffic between the server and its remote clients is not encrypted.

Grab a TLS certificate and its key, and copy them to your server. You can place them wherever you want β€” filenames don't matter either, as long as the postgres user has access.



# mv deathroll-internal.pem /etc/ssl/certs/
# mv deathroll-internal-key.pem /etc/ssl/private/
# chown postgres: /etc/ssl/private/deathroll-internal-key.pem
Enter fullscreen mode Exit fullscreen mode

File permissions 


$ stat --printf '%n\nMode: %a (%A)\nOwner: %u (%U)\nGroup: %g (%G)\n' /etc/ssl/certs/deathroll-internal.pem
/etc/ssl/certs/deathroll-internal.pem
Mode: 644 (-rw-r--r--)
Owner: 1000 (t_chuchkanov)
Group: 1000 (t_chuchkanov)
$ sudo -u postgres stat --printf '%n\nMode: %a (%A)\nOwner: %u (%U)\nGroup: %g (%G)\n' /etc/ssl/private/deathroll-internal-key.pem
/etc/ssl/private/deathroll-internal-key.pem
Mode: 600 (-rw-------)
Owner: 113 (postgres)
Group: 122 (postgres)
Enter fullscreen mode Exit fullscreen mode

Configure

Settings

Instead of editing the config file manually, it's easier and quicker to use pg_conftool.



# pg_conftool set ssl_cert_file /etc/ssl/certs/deathroll-internal.pem
# pg_conftool set ssl_key_file /etc/ssl/private/deathroll-internal-key.pem
# pg_conftool set listen_addresses localhost,tfstate-pg.deathroll.internal
Enter fullscreen mode Exit fullscreen mode

To check if all the values are correct:



$ for P in listen_addresses ssl_cert_file ssl_key_file; do pg_conftool show $P; done
Enter fullscreen mode Exit fullscreen mode
Access Control

Configure access control for your Postgres installation. Append the line below to the /etc/postgresql/14/main/pg_hba.conf file:

πŸ“’ NOTE: πŸ‘† Pay attention to the Postgres version. It may differ in your case.



hostssl all             all             172.17.0.0/24            scram-sha-256
Enter fullscreen mode Exit fullscreen mode

πŸ“’ NOTE: πŸ‘† Change the allowed network to whatever you need. It may even be 0.0.0.0/0.

Start 



# systemctl start postgresql.service
Enter fullscreen mode Exit fullscreen mode

If your configuration is correct, the Postgres systemd unit must be in the running state, the Postgres cluster online, and the daemon listening on the port 5432. Congrats πŸ₯³.



$ systemctl list-units postgres*.service
Enter fullscreen mode Exit fullscreen mode

Example 


$ systemctl list-units postgres*.service
  UNIT                       LOAD   ACTIVE SUB     DESCRIPTION               
  postgresql.service         loaded active exited  PostgreSQL RDBMS
  postgresql@14-main.service loaded active running PostgreSQL Cluster 14-main

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
Enter fullscreen mode Exit fullscreen mode 



$ pg_lsclusters
Enter fullscreen mode Exit fullscreen mode

Example 


$ pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
14  main    5432 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
Enter fullscreen mode Exit fullscreen mode 



# ss -tlpn | grep postgres
Enter fullscreen mode Exit fullscreen mode

Example 


# ss -tlpn | grep postgres
LISTEN 0      244      172.17.0.40:5432      0.0.0.0:*    users:(("postgres",pid=19728,fd=7))       
LISTEN 0      244        127.0.0.1:5432      0.0.0.0:*    users:(("postgres",pid=19728,fd=6))       
LISTEN 0      244            [::1]:5432         [::]:*    users:(("postgres",pid=19728,fd=5))
Enter fullscreen mode Exit fullscreen mode

Almost there! The last few steps of Postgres configuration are left.

Configure a Database

Now we need to:

  1. Change the postgres user password
  2. Create a Postgres user for Terraform clients
  3. Create a database for Terraform state
  4. Grant some privileges on this database to the Terraform user

Let's roll! 



$ sudo -u postgres psql
Enter fullscreen mode Exit fullscreen mode

πŸ“’ NOTE: πŸ‘† Depending on your $PWD, psql may throw a warning about its inability to access this directory. Ignore it.



postgres=# ALTER USER postgres ENCRYPTED PASSWORD '******';
postgres=# CREATE USER terraform ENCRYPTED PASSWORD '******';
postgres=# CREATE DATABASE terraform_backend;
postgres=# GRANT CREATE,CONNECT,TEMPORARY ON DATABASE terraform_backend TO terraform;
Enter fullscreen mode Exit fullscreen mode

Example
Postgres shell example

Configure a Firewall

Don't forget to configure a firewall on your system.



# ufw allow 22,5432/tcp
# ufw enable
Enter fullscreen mode Exit fullscreen mode

To check the status:



# ufw status
Enter fullscreen mode Exit fullscreen mode

That's it! It's time to test the connection from another host and configure a Terraform client 😎.

Check Connection from Another Host

Install psql and issue the command below:



$ psql postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend
Enter fullscreen mode Exit fullscreen mode

πŸ’‘ TIP: The connection string scheme is: postgres://<user>@<host>/<database_name>

πŸ’‘ TIP: On Ubuntu, the package is called "postgresql-client".

You can quit out of the psql client prompt right after it establishes the connection, because there's nothing to do in it apart from testing connectivity.

Example 


$ psql postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend
Password for user terraform: 
psql (15.5 (Debian 15.5-0+deb12u1), server 14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.

terraform_backend=> \dn
  List of schemas
  Name  |  Owner   
--------+----------
 public | postgres
(1 row)

terraform_backend=>
Enter fullscreen mode Exit fullscreen mode

Configure a Terraform Client

πŸ’‘TIP: When you need to work with multiple terraform projects, use workspaces, since all of the freshly initialized projects have only the default workspace. Hence, there will be remote state conflicts, if a single workspace is used for several independent states.

In a Terraform project's main file, create a backend block inside of terraform. Provide a string value for conn_str.

It is really that simple.



terraform {
  backend "pg" {
    conn_str = "postgres://terraform@tfstate-pg.deathroll.internal/terraform_backend"
  }
}
Enter fullscreen mode Exit fullscreen mode

Before issuing any Terraform commands, export the Postgres password as an environment variable: 



$ read -s PGPASSWORD
$ export PGPASSWORD
Enter fullscreen mode Exit fullscreen mode

For a fresh project, terraform init is sufficient. For an existing project, add the -reconfigure flag to this command.



$ terraform init

Initializing the backend...

Successfully configured the backend "pg"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding telmate/proxmox versions matching "2.9.11"...
- Installing telmate/proxmox v2.9.11...
- Installed telmate/proxmox v2.9.11 (unauthenticated)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

β•·
β”‚ Warning: Incomplete lock file information for providers
β”‚ 
β”‚ Due to your customized provider installation methods, Terraform was forced to calculate lock file checksums locally for the following providers:
β”‚   - telmate/proxmox
β”‚ 
β”‚ The current .terraform.lock.hcl file only includes checksums for linux_amd64, so Terraform running on another platform will fail to install these providers.
β”‚ 
β”‚ To calculate additional checksums for another platform, run:
β”‚   terraform providers lock -platform=linux_amd64
β”‚ (where linux_amd64 is the platform to generate)
β•΅

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Enter fullscreen mode Exit fullscreen mode

If you want to learn how to use Terraform providers in isolated environments, I have a yet another article for you πŸ˜‰.

deathroll

Using Terraform Providers in Isolation

deathroll ・ Aug 6 '23

#devops #tutorial #opensource #terraform

What to Do After a Reboot?

  1. Open and map the LUKS-encrypted partition
  2. Mount its filesystem
  3. Start the Postgres service

As simple as three to four commands!

A set of commands to do after a reboot

The Result

As the result, we have a PostgreSQL server with a database cluster located on a LUKS-encrypted partition.

This Postgres installation is used for storing remote Terraform state, and the network traffic between it and its remote clients is encrypted.

Sounds great, innit 🀌?

A drawn diagram representing the resulting architecture

Scroll down to the comments section to get some enhancement tips.

Top comments (1)

Collapse
 
deathroll profile image
deathroll •

P.S. You can consider encrypting or disabling swap on the Posgres server for better security.

Read next

felipefreitag profile image

Create admin panels for Postgres in 3 short steps

Felipe Freitag Vargas -

egor_kaleynik_7dbe9393e86 profile image

DevOps Engineer Career Path Guide

Egor Kaleynik -

thenjdevopsguy profile image

Headlamp: Microsoft’s Open-Source k8s Manager

Michael Levan -

iammikeade profile image

Top 10 Active Directory (AD) Attack Methods

Adedeji Michael -