How to Manage Kubernetes Secrets with Terraform

In this article, we will look at Kubernetes (K8s) Secrets, explain what they are, and discuss why you might want to use Terraform to manage them. We will show how to set up Terraform with the Kubernetes provider so the secrets can be managed with examples.

What are Kubernetes Secrets?

Kubernetes Secrets are essentially a secure way to store sensitive information like passwords, API or Oauth tokens, SSH Keys, and certificates used within your Kubernetes applications. Instead of holding this information in a container or pod, putting this in a secret object provides a mechanism to abstract sensitive data away from your container images and configuration files, improving security. Secrets can be injected into pods as environment variables or mounted as files, making them accessible to applications without hardcoding sensitive data.

• Secrets are stored as separate objects from your application code and configuration files. This makes it easier to manage and update credentials without having to redeploy your application.
• Secrets are versatile and can be used for various purposes, such as setting environment variables for containers, providing credentials (such as SSH keys or passwords) to Pods, and allowing the kubelet to pull container images from private registries. The K8s control plane also uses Secrets for tasks like automating node registration.
• Secrets are stored in an unencrypted format (base64) within the Kubernetes cluster (in etc) by default, meaning that anyone with access to etcd can potentially decode them. However, to enhance security, they can be encrypted at rest.
• Access to Secrets is controlled through RBAC (role-based access control). This mitigates the risk of unauthorized access to sensitive credentials.
• Pods can access Secrets through environment variables or volume mounts, but only if they are explicitly authorized to do so.
• Regularly rotate Secrets where appropriate to minimize the impact of potential exposure.
• Never hardcode Secrets in your application code or configuration files.

Read more: How to create, use, and manage Kubernetes Secrets?

To create a Secret, you can use kubectl:

kubectl create secret generic my-secret --from-literal=username=myuser --from-literal=password=mypassword

Or YAML:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: bXl1c2Vy  # base64 encoded value of "myuser"
  password: bXlwYXNzd29yZA==  # base64 encoded value of "mypassword"

To mount a volume using a YAML config file:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    volumeMounts:
    - name: secret-volume
      mountPath: "/etc/secret"
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

Of the several types of Kubernetes Secrets available, the default type is Opaque.

Source

Why use Terraform for Kubernetes Secrets?

Now we understand what Secrets are and why they are used in Kubernetes, why would we want to bring Terraform into the equation? Well, here are a few good reasons:

Terraform allows you to define your Kubernetes Secrets alongside your other infrastructure resources in a declarative way, offering a centralized approach to managing Secrets, where you can create, update, and delete them within the same workflow. The Kubernetes provider in Terraform makes it easy for teams familiar with Terraform to manage Secrets using the kubernetes_secret resource, with all changes logged to provide a useful audit trail for security and compliance purposes.

Terraform itself offers features like secret rotation, which might be crucial for enhanced security. You might need to integrate it with external tools like Vault for such functionalities. Also, consider that Terraform stores secrets in plain text within its state file by default. This necessitates extra steps for secure storage and access control mechanisms.

If you don't already use Terraform for IaC management, Kubernetes offers built-in mechanisms for managing secrets. Dedicated secret management tools like HashiCorp Vault provide more robust security features that may be more appropriate for complex scenarios with stricter security requirements.

Read more: How to Deploy Kubernetes Resources with Terraform

💡 You might also like:

• Using ArgoCD & Terraform to Manage Kubernetes Cluster
• 20+ Best CI/CD Tools for DevOps
• How to Manage Okta with Terraform

Prerequisites: Setting up Terraform for Kubernetes

To follow the example below, you'll need access to an existing Kubernetes cluster. This could be a local cluster running on your machine (e.g., Minikube), a cluster managed by a cloud provider (e.g., AWS EKS, GCP GKE, Azure AKS), or a self-managed cluster on your own infrastructure.

If you don't already have a cluster, you can use kind (Kubernetes in Docker) to do so quickly:

1. Install kind using your package manager (e.g., Homebrew or Chocolatey).
2. Download the kind configuration file: kind-config.yaml.
3. Create a kind Kubernetes cluster with the following command:

kind create cluster --name terraform-learn --config kind-config.yaml

Once you have your cluster provisioned:

1. Download and install kubectl from the official Kubernetes website.
2. Verify the installation kubectl version --client
3. Ensure your kubeconfig file is configured correctly to access your Kubernetes cluster. By default, this file is located at ~/.kube/config.
4. Verify connectivity to the cluster kubectl get nodes

Next, get Terraform! Download and install Terraform using the appropriate version for your operating system from the official website and follow our Terraform installation guide if you need more assistance.

Configuring Terraform for Kubernetes Secrets management

With K8s up and running and Terraform installed, we can define the Terraform configuration files to use with Secrets management.

1. Define the Kubernetes provider

First, we define the kubernetes provider in the configuration. Your cluster setup will determine the method you use to authenticate Terraform with your cluster. 

For example, you can use a service account token, a Kubernetes API token, or a cloud provider-specific authentication method. The latest documentation will explain how to set this up for your specific scenario.

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
  }
}

provider "kubernetes" {
  host       = "https://<your_kubernetes_api_server>"
  # Configure authentication (token, service account, etc.) based on your cluster setup
}

Initialize Terraform, which tells it to download the necessary provider plugins using the terraform init command.

2. Configure the secrets.tf file

Create a new file and name it secret.tf --- here we will define a Kubernetes Secret using the kubernetes_secret resource. You can find all the available options for the resource on the Hashicorp docs pages.

Note that the values of Secrets are stored in base64-encoded format. This is not for security but rather a requirement of the Kubernetes API. The API expects the data to be base64-encoded so that it can handle arbitrary binary data as well as plain text.

secret.tf

resource "kubernetes_secret" "my_secret" {
  metadata {
    name      = "my-secret"
    namespace = "default"
  }
  data = {
    username = base64encode("my-username")
    password = base64encode("my-password")
  }
}

3. Pass the sensitive variables

In this example, the data block defines key-value pairs for secret data. To further enhance security, consider referencing the password from a variable marked as sensitive using the base64encode function to encode the data. The variables can be passed in using environment variables rather than specifying them directly in the configuration.

export TF_VAR_username=myuser
export TF_VAR_password=mypassword

variable "username" {
  sensitive = true
}

variable "password" {
  sensitive = true
}

resource "kubernetes_secret" "my_secret" {
  metadata {
    name      = "my-secret"
    namespace = "default"
  }
  data = {
    username = base64encode(var.username)
    password = base64encode(var.password)
  }
}

If your Secret is stored in Vault, you can use the lookup function within Terraform to retrieve the password from Vault during deployment:

resource "kubernetes_secret" "my_secret" {
metadata {
    name      = "my-secret"
    namespace = "default"
  }

  data = {
    username = "my-username"
    password = lookup(var.database_password, "path/to/password/in/vault")
  }
}

4. Run terraform plan and terraform apply

Once your configuration files look good, run terraform plan to see the changes and terraform apply to apply them.

5. Verify the Secrets created

Lastly, you need to verify the Secret existing in your cluster:

kubectl get secret my-secret -o yaml

You will see the encoded values in the output, which can be decoded using a base64 decoder.

echo 'encoded_value' | base64 --decode

Kubernetes and Terraform with Spacelift

Spacelift supports both Terraform and Kubernetes and enables users to create stacks based on them. Leveraging Spacelift, you can build CI/CD pipelines to combine them and get the best out of each tool. This allows you to use a single tool to manage your Terraform and Kubernetes resources lifecycle, enabling your teams to collaborate easily and adding some necessary security controls to your workflows.

For example, you could deploy Kubernetes clusters with Terraform stacks and then deploy your containerized applications to your clusters on separate Kubernetes stacks. This approach allows you to easily integrate drift detection into your Kubernetes stacks and enables your teams to manage all your stacks from a single place. 

This article shows why using Kubernetes and Terraform with Spacelift makes sense. The code is available here.

If you want to learn more about Spacelift, create a free account or book a demo with one of our engineers.

Key points

Be careful not to expose your Secrets when using Terraform to manage them. To do this, implement strict RBAC controls and encryption at rest, and always protect your Terraform state file, which can contain sensitive information!

By using Terraform to manage your K8s secrets, you can bring your K8s infrastructure under IaC control, streamlining your workflows and improving overall security and efficiency.

Written by Jack Roper