Multiple Environments in Ansible

Many systems are deployed in a multienvironment context, for example: production, stage, and dev. These environments often share variables and artifacts. In Ansible, there are different methods to work in this context. For example, separate directory layout and soft links. However, it can end with a considerable amount of data and duplicate files between environments, exposing variables to all hosts or adding much more complexity to playbooks.

Demo multienv tests a stackable multienvironment directory layout for Ansible, using multienv Ansible role. The main goal is to have little file and data duplication in a multienvironment Ansible project while maintaining Ansible groups and host granularity. The environments are separated. However, they are based over the others in a hierarchical way. So each one only has the necessary files and artifacts.

The approach to achieve this is to use unionfs and a container (docker by default) for stacking the environments. There are other two alternative modes to run it: 2) podman or 3) binary unionfs install on the host. A variable is used for changing to the preferred method.

Approach

A separate directory layout is used as recommended by Ansible Best Practices. There could be: a. base environment: for the most common and shared variables and artifacts; b. dev environment; c. stage environment; and d. production. Then, two or more of them could be unified as one using unionfs. That directory is used as the current Ansible environment (inventory, vars, artifacts, etc). This way, one can modify each environment in its respective directory. And instruct Ansible to use the unified directory as its inventory parameter.

Example #1 - simple stacked environment

In the following example there is one environment for the most common variables and artifacts among environments. This is the base environment. Then, any other environment could be stacked over it by setting multienv_union. For example:

Dev

multienv_union:
  - base
  - dev

Production

multienv_union:
  - base
  - production

In the latter example, there are two separate directories: environment/base, and environment/production. The unified directory in that case would be union_environment/production, as a stacked environment: base+production

Example #2 - multiple stacked environments

More than two environments can be stacked. multienv_union could be set with base+dev+stage. In that case, base is mounted, then dev, and over it stage:
Stage

multienv_union:
  - base
  - dev
  - stage

Other options considered

• Ansible plugins: the logic could be added using Ansible plugins. It has to handled inventory, vars and other artifacts (files, templates, among others). Unionfs approach could handled those without addional plugins
• Overlayfs: It is another union filesystem included in the current linux kernel. However, I couldn't get it working for some type of modifications of the lower and upper layers using fuse-overlayfs. For example, while mounted, a file created in the base environment (lower) does not appear in the union directory. A file that exists in both environment and is deleted, breaks its path in the union folder. An operation like remount is needed to reflect those changes in the union mount because modifying the underlying directories is undefined.
• Rsync and Inotify. Using inotify to monitor modifications and rsync to sync into one environment. One consideration about this is how to handle deleted files in the destination. They are replace again every time the command runs.

Issues

• Centos: if using Centos and getting a message like 'is mounted on / but it is not a shared mount', you may need to make multienv_host_mountpoint a shared mount point with mount --make-rshared <multienv_host_mountpoint> . Replace <multienv_host_mountpoint> with the respective value