Shhhh... Kubernetes Secrets Are Not Really Secret!

Original post written by Deepu K Sasidharan for Auth0 Blog.

Learn how to setup secure secrets on Kubernetes using Sealed Secrets, External Secrets Operator, and Secrets Store CSI driver.

Kubernetes has become an inevitable part of the modern software infrastructure. Hence managing sensitive data on Kubernetes is also an essential aspect of modern software engineering so that you can put the security back into DevSecOps. Kubernetes offers a way to store sensitive data using the Secret object. While it's better than nothing, it is not really a secret, as it is just base64 encoded strings that anyone with access to the cluster or the code can decode.

Caution:
Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access, such as the ability to create a Deployment.
— Kubernetes docs

The problem of reading secrets from the cluster can be fixed using proper RBAC configuration and by securing the API server, check out How to Secure Your Kubernetes Clusters With Best Practices to learn more about RBAC and cluster API security. Securing secrets on the source code is the bigger problem. Everyone who has access to the repositories containing those secret definitions can also decode them. This makes it quite tricky to manage Kubernetes secrets in Git, like every other resource.

Let's see how to setup more secure secrets using the;

• Sealed Secrets,
• External Secrets Operator,
• Secrets Store CSI driver.

You would need a Kubernetes cluster to run the samples. I used k3d to create a local cluster. You can also use kind or minikube for this purpose.

Sealed Secrets

Sealed Secrets is an open-source Kubernetes controller and a client-side CLI tool from Bitnami that aims to solve the "storing secrets in Git" part of the problem, using asymmetric crypto encryption. Sealed Secrets with an RBAC configuration preventing non-admins from reading secrets is an excellent solution for the entire problem.

It works as below;

1. Encrypt the secret on the developer machine using a public key and the kubeseal CLI. This encodes the encrypted secret into a Kubernetes Custom Resource Definition (CRD)
2. Deploy the CRD to the target cluster
3. The Sealed Secret controller decrypts the secret using a private key on the target cluster to produce a standard Kubernetes secret.

The private key is only available to the Sealed Secrets controller on the cluster, and the public key is available to the developers. This way, only the cluster can decrypt the secrets, and the developers can only encrypt them.

Read more...