When we at Caredox decided to start writing services in Elixir, building a solution for safely storing and accessing secrets was one of the first action items we wanted to address. Our infrastructure has always been hosted on AWS so it made sense for us to continue integrating with their services to store our secrets. After considering the handful of encryption options that AWS provides, we decided to move forward with Parameter Store: a component of AWS Systems Manager (SSM).

Caredox uses Distillery to build releases for our Elixir application. Distillery has a concept called configuration providers to populate the runtime environment by parsing a designated file. Depending on how secrets are stored (JSON, TOML, YAML, etc.), each storage mechanism will require its own config provider. We didn't find a config provider that is designed to integrate with SSM, so we wrote our own which is available as a hex package. Our config provider affords us a relatively simple process for accessing secrets. Before launching our app, we request our secrets from AWS SSM and write them to a unidimensional JSON file. Our config provider reads this flat JSON file and generates a multidimensional data object that can be used by our application's runtime to access values, sensitive or otherwise. When updating or adding values in SSM, running docker containers have to be restarted to reflect the changes.

Specific Example

When adding functionality that requires a new config value, we'll first add that value to SSM. We follow the guideline that AWS suggests to define keys as slash (/) delimited strings, ie. /production/caredox/Redix/password. The first segment is the environment (production), the second segment is your project (caredox), the third segment is the name of the OTP application (Redix), and all remaining segments are the key you want stored in that OTP application (password).

aws ssm put-parameter --name '/production/caredox/Redix/password'\
 --type ‘SecureString’ --value 'YourSecretPassword'

We then deploy our app in a docker container. Our Dockerfile specifies an entrypoint to a bash bootscript which requests our secrets from SSM, writes them to a file, then starts our Elixir app.

aws --region us-east-1 ssm get-parameters-by-path --path "/production/YOUR_PROJECT/"\
--recursive --with-decryption --query "Parameters[]" > /etc/app_secrets.json
/path/to/your_project foreground

Now that our container has a file with our secrets in it, the config provider is ready to read that file.

# rel/confix.exs

environment :prod do
  set(config_providers: [{AwsSsmProvider, ["/etc/app_secrets.json"]}])
end

Shared Example

Sometimes we have multiple apps that depend on a single source of truth for a configuration. This might be required on a per environment basis (staging vs. production), or we might want a config to have a default value across the whole system, regardless of environment. To fetch the global secrets, the bootscript will require an additional call to SSM. We happen to write them to /etc/global_secrets.json

For values that are consistent across a specific environment, we use the magic string host_app to indicate to the SSM config provider that each app is in charge of managing how the config is set. For instance, a global SSM config by environment would look like this /staging/global/host_app/company_internal_password.

The config provider must be told what atom will substitute the string host_app in the SSM key, in this case the atom to be substituted in is :myApp. We do that by including :myApp as the second element in the list sent to the init function of the config provider.

# rel/confix.exs

environment :prod do
  set(config_providers: [{AwsSsmProvider, ["/etc/global_secrets.json", :myApp]}])
end

Conclusion

Our secrets now live securely in AWS SSM and are written to an ephemeral file in each docker container after it starts running. The container itself lives in a secure VPC so we have no loose ends.