Tutorial: How to use Kubernetes Secrets for storing sensitive config data

Hello and welcome 👋👋 We continue the "Kubernetes in a Nutshell" journey! In one of the previous blogs, we saw how to configure Kubernetes apps using the ConfigMap object. In this post, we will explore Kubernetes Secrets and how they can be used to store sensitive configuration data which needs to be handled securely e.g. database credentials, API keys etc.

As usual, this is going to be example driven and you will learn about:

• How to create Secrets (CLI, yaml etc.), and
• Various ways of using them in your apps (env variables, volumes, etc.)

The code (and YAML!) is available on GitHub

Happy to get your feedback via Twitter or just drop a comment 🙏🏻

This blog has been divided into two logical sections:

1. Ways to create Secrets
2. Techniques to use Secrets in your applications

Pre-requisites:

To go through the examples in this post, all you need is a minikube cluster and kubectl CLI tool to access the cluster.

Install minikube as a single-node Kubernetes cluster in a virtual machine on your computer. On a Mac, you can simply:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 \
  && chmod +x minikube

sudo mv minikube /usr/local/bin

Install kubectl to interact with the minikube cluster. On a Mac, you can:

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Creating Secrets

Let’s look at techniques using which you can create a Secret

Using a manifest file

Use the data section

It’s possible to create a Secret along with the configuration data stored as key-value pairs in the data section of the definition.

apiVersion: v1
kind: Secret
metadata:
  name: service-apikey
data:
  apikey: Zm9vYmFy

The Secret contains key-value data representing the sensitive info, with apikey being the key and value is a base64 encoded string

To create this Secret in Kubernetes:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-data.yaml

To keep things simple, the YAML file is being referenced directly from the GitHub repo, but you can also download the file to your local machine and use it in the same way.

To confirm that the Secret has been created:

kubectl get secret/service-apikey -o yaml

You will get a (YAML) response similar to:

apiVersion: v1
data:
  apikey: Zm9vYmFy
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"apikey":"Zm9vYmFy"},"kind":"Secret","metadata":{"annotations":{},"name":"service-apikey","namespace":"default"}}
  creationTimestamp: "2019-12-17T11:11:27Z"
  name: service-apikey
  namespace: default
  resourceVersion: "113009"
  selfLink: /api/v1/namespaces/default/secrets/service-apikey
  uid: 671b547c-3316-4916-b6dc-be2b551b974e
type: Opaque

Fetching the Secret details using kubectl get does not disclose its contents

Notice that apikey: Zm9vYmFy was what we had provided in the YAML manifest. You can check the plain text form by decoding it:

echo 'Zm9vYmFy' | base64 --decode

//foobar

Use the stringData section

The data attribute used in the above example is used to save base64 encoded information. If you want to store plaintext data securely, you can use stringData section. Here is an example:

apiVersion: v1
kind: Secret
metadata:
  name: plaintext-secret
stringData:
  foo: bar
  mac: cheese

The values for foo and mac are being passed as plain text. Create this Secret and confirm:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-plaintext.yaml

kubectl get secret/plaintext-secret -o yaml

Here is the data portion of the YAML response. The actual data is stored in base64 endcoded format

data:
  foo: YmFy

If you decode the data, you can confirm that it matches the original plain text input (bar) we had provided

echo 'YmFy' | base64 --decode

//bar

Note that data section does not accept plain text attribute. Trying to do so will result in error similar to this: illegal base64 data at input byte 8

File contents

You can provide the contents of an entire file as input to the stringData section as well! Here is what this might look like:

apiVersion: v1
kind: Secret
metadata:
  name: secret-in-a-file
stringData:
  app-config.yaml: |-
    hello: world
    john: doe

Create this Secret:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/secret-file.yaml

This resulting Secret will contain a key named app-config.yaml and its contents (value) will be the base64 encoding of the provided data

As usual, you confirm this in Kubernetes as well as decode the contents

kubectl get secret/secret-in-a-file -o yaml
echo '<"data" content in yaml response> | base64 --decode

Note: when you use this technique, your application is responsible for parsing out the data which represents the Secret configuration. In this case, it happens to be newline-separated key-value pairs, but it could be anything else

Using kubectl

You can use the kubectl create secret command to create Secret objects

Using --from-literal

You can use plain text data to create Secret using the CLI (this will be stored in base64 encoded format in Kubernetes)

kubectl create secret generic redis-credentials --from-literal=user=poweruser --from-literal=password='f0ob@r'

Using --from-file

kubectl create secret generic topsecret --from-file=api_keys.txt

This will create a Secret (topsecret) with

• a key with the same name of the file i.e. api_keys.txt in this case
• and, value as the contents of the file

From files in a directory

You can simply point to a directory and all the files within will be used to create the Secret

kubectl create secret generic topsecrets --from-file=/home/credentials/

You will end up with

• multiple keys which will the same as the individual file name
• the value will be the contents of the respective file

Using Secrets

For Secrets to be useful, we need to ensure that they are available to our applications i.e Pods. Let's explore the ways in which we can do this

Environment variables

You can consume the Secret data as environment variables in a Pod (just like ConfigMap). Here is an example:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  containers:
    - name: nginx
      image: nginx
      env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: service-apikey
              key: apikey

We use the key apikey from Secret service-apikey and make sure its value is available as an environment variable API_KEY inside the Pod.

Create the Pod (assuming you have the Secret created from the example before) and confirm

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-env.yaml
kubectl get pods -w

Wait for the Pod to transition to Running state. Then, confirm that the environment variable has been injected into the Pod

kubectl exec pod1 -- env | grep API_KEY

You should get this response - API_KEY=foobar

Instead of referring to individual entries in a Secret, you can use envFrom to conveniently use all entries as environment variables in a Pod. This is how you might use this:

apiVersion: v1
kind: Pod
metadata:
  name: pod2
spec:
  containers:
    - name: nginx
      image: nginx
      envFrom:
        - secretRef:
            name: plaintext-secret

We are referring to the plaintext-secret Secret using envFrom.secretRef. To create this Pod

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-envFrom.yaml
kubectl get pods -w

Wait for the Pod to transition to Running state and then confirm the presence of environment variables

kubectl exec pod2 -- env | grep foo
//foo=bar

kubectl exec pod2 -- env | grep mac
//mac=cheese

This confirms that both foo and mac were added as environment variables into the Pod along with their decoded values i.e. bar and cheese respectively

Volumes

You can mount Secrets as Volume within a Pod. For e.g.

apiVersion: v1
kind: Pod
metadata:
  name: pod3
spec:
  containers:
    - name: nginx
      image: nginx
      volumeMounts:
        - name: apikey-config-volume
          mountPath: /secret
          readOnly: true
  volumes:
    - name: apikey-config-volume
      secret:
        secretName: service-apikey

The apikey-config-volume volume refers to the service-apikey Secret. To create this Pod:

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-volume.yaml
kubectl get pods -w

Wait for the Pod to transition to Running state. Then execute the following command:

kubectl exec pod3 -- cat /secret/apikey
//foobar

This confirms that the key apikey in secret service-apikey was mounted as a file (with name apikey) in the /secret directory (as specific in Pod). The content of the file is nothing but the secret value i.e. foobar in this case

Using imagePullSecrets

There is a way to use Secrets such that your application Pod can use it to authenticate and pull Docker images from private Docker registries.

There are actually three types of Secrets

• generic - used to store key-value pairs, as we have seen in the examples so far
• tls - store public.private key pair info as Secrets
• docker-registry - credentials for authenticating to a Docker registry.

The way this technique is used is very simple:

• Use docker-registry Secret type to store private Docker registry credentials in Kubernetes
• And then, imagePullSecrets (in a Pod) to reference the Secret containing the Docker registry credentials

An example always helps:

apiVersion: v1
kind: Pod
metadata:
  name: pod4
spec:
  containers:
    - name: privateapp
      image: abhirockzz/test-private-repo:latest
      command: ["/bin/sh"]
      args: ["-c", "while true; do date; sleep 5;done"]
  imagePullSecrets:
    - name: docker-repo-secret

See how imagePullSecrets.name refers to a Secret called docker-repo-secret. Let's create it

But, before that please ensure that you have a private Docker registry - I used DockerHub, but you can choose any other

Start by creating a Secret (with the name docker-repo-secret) which contain your Docker credentials using kubectl create secret docker-registry command

kubectl create secret docker-registry docker-repo-secret --docker-server=DOCKER_REG_SERVER --docker-username=DOCKER_REG_USERNAME --docker-password=DOCKER_REG_PASSWORD --docker-email=DOCKER_REG_EMAIL

For Docker Hub e.g.

kubectl create secret docker-registry docker-repo-secret --docker-server=https://index.docker.io/v1/ --docker-username=foobarbaz --docker-password=t0ps3cr3t --docker-email=foobarbaz@gmail.com

kubectl get secret/docker-repo-secret -o yaml

https://index.docker.io/v1/ is the Docker Hub registry server

To test things out, we will use a busybox image and tag it

docker pull busybox

docker tag busybox [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]
e.g. 
docker tag busybox abhirockzz/test-private-repo:latest

... and push it

docker push [DOCKER_REG]/[DOCKER_PRIVATE_REPO]:[IMAGE_TAG]
e.g. 
docker push abhirockzz/test-private-repo:latest

Once the private repo is ready, you can create the Pod which will pull the image from the private repo using the registry credentials supplied to it via the Secret

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kubernetes-in-a-nutshell/master/secrets/pod-secret-docker.yaml
kubectl get pods -w

Wait for the Pod to move to Running status.

If you see an ErrImagePull error, it indicates that there might be problem authenticating to the Docker registry. To get details use: kubectl describe pod/pod4

To confirm that the pod is working fine: kubectl logs -f pod4

Since the busybox image does not really do anything by itself, we execute: while true; do date; sleep 5;done (as provided in the Pod spec). As a result, you should see the logs (printed every 5 secs)

Tue Dec 17 14:17:34 UTC 2019
Tue Dec 17 14:17:39 UTC 2019
Tue Dec 17 14:17:44 UTC 2019
Tue Dec 17 14:18:49 UTC 2019

All good! What that means is that the Pod was able to pull down your image from a private Docker repo using the Docker credentials which was injected into the Pod using imagePullSecrets which itself referenced a Secret

Good to know

Here is a (non-exhaustive) list of things which you should bear in mind when using Secrets:

• Secret has to be created before any Pod that wants to use it.
• Secrets are applicable within a namespace i.e. they can only be used by Pods in the same namespace
• The Pod will not start if there is a reference to a non existent key in a Secret (using secretKeyRef)
• 1MiB is the size limit for individual Secrets

That's it for this edition of the "Kubernetes in a Nutshell" series. Stay tuned for more!

If you are interested in learning Kubernetes and Containers using Azure, simply create a free account and get going! A good starting point is to use the quickstarts, tutorials and code samples in the documentation to familiarize yourself with the service. I also highly recommend checking out the 50 days Kubernetes Learning Path. Advanced users might want to refer to Kubernetes best practices or watch some of the videos for demos, top features and technical sessions.

I really hope you enjoyed and learned something from this article 🙌 Please like and follow if you did!