Setup External Secrets with Hashicorp Vault on AWS EKS

external-secrets is one of the most efficient and secure ways manage Kubernetes Secrets. external-secrets integrates with many external secret stores like Hashicorp Vault, AWS Secret Manager, etc. and can be used to manage secret files and variables.

Key components

The main components of external secrets are as follows:

External Secrets Operator (ESO) is a collection of custom API resources - ExternalSecret, SecretStore, and ClusterSecretStore that provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you.

ExternalSecret - It is a declaration of what data has to be fetched from your external secret manager. It references a SecretStore, which knows how to access the data. You can do more things like set refresh interval, specify a blueprint for the resulting Kind=Secret, use inline templates to construct the desired config file containing your secret, set a target that will be created, and create and delete policies.

SecretStore - They are namespaced by design and cannot communicate with resources across namespaces. This is the file where you select your ESO controller, and cloud provider, along with the role and access IDs, and retry settings in case of connection failure.

The secrets are fetched from an external vault and made available in SecretStores, these are limited to that particular namespace. For use in multiple namespaces, use the ClusterSecretStore.

This guide will go through the process of setting up the External Secret Operator for your AWS EKS Kubernetes cluster.

To find out about other approaches to Kubernetes secrets management, check out this blog.

Pre-requisites

1. An existing Kubernetes cluster
2. Kubernetes v1.16.0 or later
3. Connected Apps or databases that require secrets to access
4. Hashicorp Vault as your external secret store
5. kubectl and helm CLI installed
6. Optional: an Argonaut account and (art cli)

Section I of the installation can be done using both Helm (1.a.) or Argonaut’s UI (1.b.). Choose one of the below section.

1.a. Install using Helm

To set this up, you will need your kubeconfig file. You can use a file created from your existing cluster or obtain this from Argonaut’s CLI.

1: Run art configure generate-aws-credentials

2: Run aws eks update-kubeconfig --name <clustername> --region <region>

Once this is ready, run the following commands from your Terminal.

For Mac and Linux:

export KUBECONFIG=/home/argo/.kube/clustername-kubeconfig.yaml

For Windows:

$env:KUBECONFIG = "C:\Users\argo\.kube\clustername-kubeconfig.yaml"

Confirm your current context by running the command:

kubectl config current-context

You will see an output with the name of your cluster argonaut-cluster. Once the context is set up, you can add external-secrets repo to Helm.

helm repo add external-secrets https://charts.external-secrets.io

You can run a helm update to make sure you’re on the latest version

helm repo update

You should see this output: "external-secrets" has been added to your repositories

You can now install the external-secrets repository

helm install external-secrets \
    external-secrets/external-secrets \
    -n tools \
    --create-namespace \
    --set installCRDs=true

You should now see an install successful message with the name and namespace specified.

1.b. Install with Argonaut

1. Open Argonaut, and navigate to your desired AWS EKS cluster.
2. Click on the Application + and the From Library option.
3. Choose Custom-Apps under configuration.
4. Ensure the selected environment is correct, then select the cluster you want to deploy the agent to.
5. Set the namespace you want to deploy the external secrets to be tools (or where you plan to use the secrets).
6. Set the release name of the application as external-secrets.
7. Set the chart name as external-secrets.
8. Set the chart repository as https://charts.external-secrets.io.
9. Leave the chart version blank. It will be automatically populated to the latest version.
10. Load the values.yaml file and make any changes (if required).
11. Then click Install. In just a minute, the external-secrets operator will be added to your cluster, and you will be able to see the outputs as follows.

Now your external-secrets operator is installed in the cluster as an Add-on application. You can view the logs, status, and update the configs by going clicking on external-secrets under Add-ons.

2. Connecting k8s CLI

External Secrets Operator (ESO) is now installed in your cluster. You can start to use it by connecting to your cluster from your Terminal and running the following kubectl commands.

To access your Kubernetes cluster, you will have to generate AWS credentials through art CLI and connect it to your EKS cluster. This is a quick two step process.

1: Run art configure generate-aws-credentials

2: Run aws eks update-kubeconfig --name <clustername> --region <region>

This connects and gives you access of your cluster in the specified region.

Note: The generated access tokens are valid for 12 hours. These inherit the same privileges granted to the Argonaut account for accessing the aws account and infra. To set up more apps such as Mirantis Lens and k9s, follow this docs page.

3. secret-store.yaml

The ClusterSecretStore is a file having the definition of how the external-secret operator can find the secrets from an external secret store. This includes two main things, an existing secret to provide authentication to the external secret store (AWS secret manager linked account) and the AWS project identifier of that particular secret that you wish to access. Make sure you have kubectl installed before proceeding further.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: "vault-backend"
spec:
  provider: # provider field contains the configuration to access the provider
    vault:
      server: "https://your-domain:8200"
      path: "kv" # Path is the mount path of the Vault KV backend endpoint
      version: "v1"
      auth:
        tokenSecretRef: # static token: https://www.vaultproject.io/docs/auth/token
          name: "vault-token"
          key: "token"

Note: A SecretStore file is also similar but is namespaced and maps to exactly one instance of an external API.

kubectl apply --filename secret-store.yaml

4. external-secret.yaml

This file tells the external-secret operator what specific data is to be fetched from your external secret store. It has three sections:

1. Secret Store reference - This references the previous resource ClusterSecretStore we created. It tells external-secrets on how to access the resources.
2. Target - This is the target secret that should be created. This defines the name and type of secret that is created in Kubernetes secret. For example PostgreSQL.
3. DataFrom - This defines the name of the secret as it is stored in your external AWS secret store.

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: "my-first-secret"
spec:
    secretStoreRef:
    name: vault-backend
    kind: SecretStore  # or ClusterSecretStore
  refreshInterval: "15s" # How often this secret is synchronized
  target: # Our target Kubernetes Secret
    name: my-first-aws-secret # If not present, then the secretKey field under data will be used
    creationPolicy: "Owner" # This will create the secret if it doesn't exist. Options are 'Owner', 'Merge', or 'None'
        deletionPolicy: "Retain"
  data:
    - secretKey: my-first-aws-secret-key
      remoteRef:
        key:message # This is the remote key in the secret provider (might change in meaning based on your provider)
        property:value # The property inside of the secret inside your secret provider

kubectl apply --namespace argonaut --filename external-secret.yaml

Once this is done, the secrets from your external secret store are brought securely to your cluster and are available to access as Kubernetes secrets. This can be checked by executing the following command:

kubectl --namespace argonaut get ExternalSecret my-first-secret

Output:

NAME              STORE           REFRESH INTERVAL   STATUS
my-first-secret   vault-backend   15s                SecretSynced

Note: The secrets are now in Kubernetes and available to anyone with access to the cluster.

Whenever you update a secret directly on your external secret store, it takes a few moments for it to reflect in your Kubernetes cluster. This refresh interval can be set as a part of your configuration.

5. Conclusion

You have now successfully set up external-secrets. This powerful open source tool allows you to manage secrets in a more secure way. You can also choose from various other secret manager tools like the AWS Secrets Manager or GCP Secrets Manager.

The external-secrets approach allows you to securely manage secrets and bring them to your cluster as needed. There are also other approaches to the same that we discuss in this blog. external-secrets can do so much more like generating secrets, CRDs, controller classes, and multi-tenancy. And many more exciting things in their roadmap.

Argonaut has a native secret management solution for scaling teams. Argonaut integrations with third party secret providers is coming soon Q1CY23.

With Argonaut’s modern deployment platform, you can get up and running on AWS or GCP in minutes, not weeks. Our intuitive UI and quick integrations with GitHub, GitLab, AWS, and GCP make managing your infra and applications easy - all in one place.