Introduction

When we are using Ansible, we often need passwords or tokens or any secrets that we don't want to store as plain text and definitely don't want to commit to a git repository. At least not as plain text. Personally, I don't like to commit secrets even if they are encrypted, unless it is required.

I used Ansible Vault for years, and I was suggested to replace it with GPG since we used that too anyway, but somehow I just didn't like it so much, so I kept using Ansible Vault. My problem with Ansible Vault was that when I encrypted a yaml file which contained secrets, the whole file was encrypted including parameter names. Of course there was a solution for that. Creating an unencrypted file without the values and another which was completely encrypted, so you could manage two files instead of one. Or you could encrypt individual variables manually.

Maybe about a year ago, I started to use Secrets Operations (SOPS) and it made everything easier. Not to mention that I can use SOPS without Ansible. In this post I will use SOPS to finally encrypt the sudo password (aka become pass) for the user on the remote server and I will use the Nix package manager to install the required to packages, sops and age.

If you want to be notified about new videos, you can subscribe to my YouTube channel: https://www.youtube.com/@akos.takacs

Table of contents

• Before you begin
  • Requirements
  • Download the already written code of the previous episode
  • Have an inventory file
  • Activate the Python virtual environment
• Why Nix again?
• Create a new config file
• Generate the age key pair
• Create a wrapper script for sops and detect the public key automatically
• Encrypt a file
• Edit an already encrypted file
• Use the secret in the inventory file
• Use a dedicated user for Ansible
• Run Ansible in a Nix shell
• Test if Ansible could read the secret
• Don't commit secrets to a git repository
• What more you should know
• Conclusion

Before you begin

Requirements

» Back to table of contents «

• The project requires Nix which we discussed in Install Ansible 8 on Ubuntu 20.04 LTS using Nix
• You will also need an Ubuntu remote server. I recommend an Ubuntu 22.04 virtual machine.

Download the already written code of the previous episode

» Back to table of contents «

If you started the tutorial with this episode, clone the project from GitHub:

git clone https://github.com/rimelek/homelab.git
cd homelab

If you cloned the project now, or you want to make sure you are using the exact same code I did, switch to the previous episode in a new branch

git checkout -b tutorial.episode.6b tutorial.episode.6

Have the inventory file

» Back to table of contents «

Copy the inventory template

cp inventory-example.yml inventory.yml

• Change ansible_host to the IP address of your Ubuntu server that you use for this tutorial,
• and change ansible_user to the username on the remote server that Ansible can use to log in.
• If you still don't have an SSH private key, read the Generate an SSH key part of Ansible playbook and SSH keys
• If you want to run the playbook called playbook-lxd-install.yml, you will need to configure a physical or virtual disk which I wrote about in The simplest way to install LXD using Ansible. If you don't have a usable physical disk, Look for truncate -s 50G <PATH>/lxd-default.img to create a virtual disk.

Activate the Python virtual environment

» Back to table of contents «

How you activate the virtual environment, depends on how you created it. In the episode of The first Ansible playbook describes the way to create and activate the virtual environment using the "venv" Python module and in the episode of The first Ansible role we created helper scripts as well, so if you haven't created it yet, you can create the environment by running

./create-nix-env.sh venv

Optionally start an ssh agent:

ssh-agent $SHELL

and activate the environment with

source homelab-env.sh

Why Nix again?

» Back to table of contents «

There are multiple ways which you can find in the git repository of sops to install it, but I will show you one that isn't there. The reason is that I want to support this project on Linux and macOS and I also want you to use the same version that I use. So if I follow the guide on GitHub, I need to either download a specific binary from GitHub or use different package managers on different platforms. Both would require automatically detecting the platform. Since downloading binaries would also mean that I would need to manually verify checksums and install dependencies and I like to use general solutions, I rather choose Nix. We already use Nix to create a Python virtual environment, so why not?

Create a new config file

» Back to table of contents «

SOPS supports multiple tools for encryption like PGP or age, and we will use age. The first step is to open a Nix shell with pre-installed age so we can use the age-keygen command to generate a keypair. Eventually, we need to run a command like this:

age-keygen -o age/private-key

SOPS will look for this file in specific folder which is different on macOS and on Linux, so we will override that, and regardless of the OS you are using, you can run the same commands, and I can use the same project on macOS and in a Linux virtual machine where I mount the project from the host. In order to use the same key in the terminal and in Ansible, we will need two environment variables:

• SOPS_AGE_KEY_FILE
• ANSIBLE_SOPS_AGE_KEYFILE

Since I will have multiple scripts using the same config, I will create a config file which will be a simple shell script. Let's save it in the project root and call it config.sh.

# HOMELAB variables
export HOMELAB_VAR_DIR="${HOMELAB_PROJECT_ROOT:-.}/var"

# SOPS variables
export SOPS_AGE_KEY_FILE="$HOMELAB_VAR_DIR/age/key"

# Ansible variables
export ANSIBLE_SOPS_AGE_KEYFILE="$SOPS_AGE_KEY_FILE"

We have a "HOMELAB variables" section for general variables, an "SOPS variables" section for variables that are used by the "sops" executable directly and an "Ansible variables" section for Ansible of course. We also use the HOMELAB_PROJECT_ROOT variable, which is just a simple dot by default, but can be set in each script. We do this only because this way we don't have to support sourcing the config file in different shells and use different ways to determine the project root.

Generate the age key pair

» Back to table of contents «

Now we need a script that runs age-keygen and stores the private key file based on the config file. Let's call it sops-keygen.sh

#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p age
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz

current_dir="$(cd "$(dirname "$0")" && pwd)"
export HOMELAB_PROJECT_ROOT="$current_dir"

source "$HOMELAB_PROJECT_ROOT/config.sh"

parent_dir="$(dirname "$SOPS_AGE_KEY_FILE")"

if [[ ! -e "$parent_dir" ]]; then
  mkdir -p "$parent_dir"
fi

if [[ ! -e "$SOPS_AGE_KEY_FILE" ]]; then
  age-keygen -o "$SOPS_AGE_KEY_FILE"
fi

echo "The private key file is at $SOPS_AGE_KEY_FILE"

It should be executable of course:

chmod +x sops-keygen.sh

And when you execute it,

./sops-keygen.sh

the script will generate the private key, save it and show the location. If you are wondering where the public key is, it is in the same file.

source config.sh
cat $SOPS_AGE_KEY_FILE

The output will be something like this:

# created: 2023-11-12T11:36:39+01:00
# public key: age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e
AGE-SECRET-KEY-1YCKA5MJ44V2YDA8RZ9MKV0YWTDLG8MXFSDEFT9AQ76GQ7005JFQQN0WFN4

Create a wrapper script for sops and detect the public key automatically

» Back to table of contents «

You can copy the public key manually when you need it, but sometimes you want to get it in a script like now. We need a wrapper script for the sops command. In this script we can get the public key and load it into a variable. We don't set this variable in the config file, because we need it only when we run sops. Even then, we would need it only when we want to encrypt a file, not when we decrypt it. I won't overcomplicate this script even more, so I will let the script read the public key every time. The filename will be sops.sh.

#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p sops
#! nix-shell -p age
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz

current_dir="$(cd "$(dirname "$0")" && pwd)"
export HOMELAB_PROJECT_ROOT="$current_dir"

source "$HOMELAB_PROJECT_ROOT/config.sh"

export SOPS_AGE_RECIPIENTS="$(age-keygen -y < $SOPS_AGE_KEY_FILE)"

sops "$@"

And of course we make it executable:

chmod +x ./sops.sh

You can test it with a simple command that shows its version number:

./sops.sh --version

Encrypt a file

» Back to table of contents «

What we need now is a file to encrypt. Let's create secrets.plain.yml in the project root.

become_pass:

And encrypt it:

./sops.sh --encrypt --output secrets.yml secrets.plain.yml

We will get a secrets.yml file like this:

become_pass: null
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNZjVEOTJJZWkwQytML1F6
            QVFJNHNmcVhHbThMZWI2SGJWalArd2c0bkVvCng3QjgvQms4SE9LT1o3Z21DN1Yw
            MVBWcFdnNVV3UGVKNFB4YS85UU9ScWsKLS0tIFNOTkdodkt6aEZMT1pvdGxxVjNm
            M29mL2JmMDI4N1FmbUMxVmpsRW1vMWcKRIxPc0dZv9JcEn2NyZ9OJ6QDerh9VIcw
            rvD0Tyvrbzoc32cxUZMEUGH+tCwFi5eQ212Fehw1jlLh/YYmYPYBEA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2023-11-12T20:21:57Z"
    mac: ENC[AES256_GCM,data:QsdpRH36FdY3Gq01U/4NvuiXt/OAkxQv+GURksOmlJCpEzyjKDXrqJOk6D/ZiuGrwPn48803MFSvojNtvdrLr0k8+sJ58/Kv9u1vf7/fxbZczvW5ohVhH1bfou6ZgqsJyMLu8yp3EbXukQ8hJe9N159HgqIdCkyycSFwOGko5O4=,iv:0Y4cP2mzq9wedRLxxYSBM4GAS/VvvbohKWGICO+IWw0=,tag:dfoEDyC0736bL+aXHEmAcA==,type:str]
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.8.1

The important part for us now is the first line where we can see the become_pass with a null value. Now I like to edit the values from a terminal and I can simply use

Edit an already encrypted file

» Back to table of contents «

./sops.sh secrets.yml

It will open the file in the default text editor like nano or vim and show you only the decrypted content without the rest of the YAML keys. My default editor currently is vim, but I'm not afraid to tell you that I usually prefer nano, although I don't care enough to change it. If I want, I can change it just for this specific command:

EDITOR=nano ./sops.sh secrets.yml

I will use the following new value in the encrypted file:

become_pass: HomeLab2023

After saving the file, the first line in the encrypted view will be like this:

become_pass: ENC[AES256_GCM,data:degiXqRIpLEVdTY=,iv:Loyf/XbEcTq6Z9enDq1ccCFzOYurL4kUa1Js9dpQzgo=,tag:xmhhExQREc6wkY373eW5+g==,type:str]

We can remove the plaintext file:

unlink secrets.plain.yml

Use the secret in the inventory file

» Back to table of contents «

We need to pass this file somehow to Ansible. We could do it multiple ways, but what I prefer is loading the content
in the inventory file. For that we need a lookup plugin called community.sops.sops which is the part ot the collection called community.sops. We can set a variable called sops and load all the secrets from yaml as an Ansible "dict".

all:
  vars:
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"

We also need to use the values somehow like this:

all:
  vars:
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
  hosts:
    ta-lxlt:
      ansible_become_pass: "{{ sops.become_pass }}"

Now as a reminder, this is my current full inventory.yml file:

all:
  vars:
    ansible_user: ansible-homelab
    config_lxd_zfs_pool_disks:
      - /dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6
    sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
  hosts:
    ta-lxlt:
      ansible_host: 192.168.4.58
      ansible_ssh_private_key_file: ~/.ssh/ansible
      ansible_become_pass: "{{ sops.become_pass }}"

Use a dedicated user for Ansible

» Back to table of contents «

You might have noticed that I changed the value of ansible_user from ta to ansible-homelab. I did it, so I don't need to change my user's password and I can still show you the secrets. It is also useful to have a dedicated user for Ansible when you have a CI/CD pipeline, and you don't want to share your password with others who can read the secrets. I created the user with the following command:

pass="HomeLab2023"
salt="sugar"
useradd \
  --shell /bin/bash \
  --create-home \
  --groups sudo \
  --password "$(openssl passwd -6 -salt "$salt" "$pass" )" \
  ansible-homelab

If you don't have openssl or just prefer to create a user manually, that's fine too, just make sure you add the user to the sudo group on Ubuntu, so the user can become root. After all, this is why we store the become pass (sudo password) in a secret.

Run Ansible in a Nix shell

» Back to table of contents «

Now there is one script that we already have, but needs to be changed. We used Nix to download sops, but that means Ansible needs to run in a Nix shell. We had this in our original run.sh in the project root:

ansible-playbook \
  -i inventory.yml \
  --ask-become-pass \
  "$@"

We don't want that script to force Ansible to ask for the become pass so that line must be removed, and we need the usual shebang lines for Nix and also our config parameters. The new run.sh will be this:

#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p sops
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz

source config.sh

ansible-playbook \
  -i inventory.yml \
  "$@"

Test if Ansible could read the secret

» Back to table of contents «

Now let's run the hello playbook which requires root privileges, but we need to change the original destination path of the hello-world.txt. Otherwise, Ansible would not need to copy again, and it could work even if the sudo password is incorrect.

./run.sh playbook-hello.yml -e hello_world_dest=/opt/hello-world-$(date +%s).txt

As you can see, we can override a role parameter from terminal. Unless this command failed, our secret worked.

Don't commit secrets to a git repository

» Back to table of contents «

We have one more job before we say goodbye. As I stated at the beginning of this post, I prefer not to commit these encrypted secrets either, so I add secrets.yml to gitignore. The other file I definitely don't want to commit is the private key so let's add the following two lines to gitignore:

/var
/secrets.yml

My full gitignore looks like this now:

/venv
/venv-linux
/inventory.yml
/var
/secrets.yml

# for macOS
/.DS_Store

What more you should know

» Back to table of contents «

In this tutorial I used a bit different way than what you can find in documentations. That made the commands easier, but also more complicated at the same time. sops supports passing the public key as an argument like sops --encrypt --age age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e, but I used the environment variable instead. Why? Because that way you can still have the option to override it from terminal, although this tutorial is not for production systems but for a private home lab, a playground, so you will probably not need to change it or add multiple public keys which means multiple recipients which would support multiple private keys to decrypt the secret file. You can find everything in the documentations and I recommend reading it and playing with sops and age, so you can discover more like how you can use age with a hardware key like YubiKey to decrypt files. For using YubiKey, you need a plugin called "age-plugin-yubikey".

Conclusion

» Back to table of contents «

We can now use secrets, but that is completely optional. If you want to pass test passwords like "abc" or "password" as plaintext on a machine where you have nothing to protect like a virtual machine which you just created for playing with Ansible or test some commands, that's fine. Then you don't need to use the lookup plugin in your inventory file.

I have to note again that this series is for creating a home lab, a local environment to develop and learn and not for a production environment. In a production environment you should always encrypt secrets. Don't forget that SOPS is just one tool so be prepared for other options too when you have to use a keyserver for example in a team.

The final source code of this episode can be found on GitHub:

https://github.com/rimelek/homelab/tree/tutorial.episode.7