Argo CD and Sealed Secrets is a perfect match

TL;DR

Sealed Secrets can works perfectly well with Argo CD. It is one of the very best secret managment approachs that you should consider.

The same idea can apply to both Helm, Kustomize or other GitOps tools.

Introduction

There is no golden standard about secret management in Argo CD. Instead, Argo CD provides you a list of solutions here and you have to think it yourself.

Although I opt-in for Sealed Secrets, I also want to show you how my decision making process guides me to such a conclusion.

To start with, we need to think of two questions.

Where to store your secret?

Although there are many solutions available, basically there are just two ways to store secrets, i.e. store in secret management platforms or git repositories.

Secret management platform approach

I don't prefer to use secret management platforms.

No matter what platform you use, it won't be K8S native because the platform lives outside of your cluster. You always needs to install some plugins in your Kubernetes clusters to read the secrets from your platform .

If you opt-in for this approach, you have to manage the platforms, plugins and integrating everything together. It introduces too much complexity and we know that complexity means more errors, bugs, human mistakes...

Git repository approach

You can upload encrypted secrets together with your Kubernetes manifests to your repository. Then Argo CD will decrypt the secret when it syncs your application. Everything is bundled together which is good for management.

Sealed Secrets is one of the solution based on this approach.

How to ingest your secret?

Now we are moving forward to git repository approach. Let us review what solutions are available.

Helm Secrets

To most people who use Helm, Helm Secrets is a popular choice. Users can encrypt values files and then Argo CD will decrypt these files accordingly.

However it is also too complicated to me.

First of all, to integrate Helm Secrets with ArgoCD, you have to do certain modifications, which you can see their tutoral. Again these complexities always leads to many issues.

Moreover, there is no way to simply avoid someone committs unencrypted values files as these files are normal YAML files.

Lastly, we know that ArgoCD allow us to specify multiple values files. So now you got a set of values files and encrypted values files. How to tell there is no unintended value overriding issue happen? Ideally, we want to avoid decrypting secrets files but now we may have to do so.

Sealed Secrets

Sealed Secrets is a general usage secret management solution. Even if you don't use Argo CD, you can also use it to encrypt secrets.

It is simple to use

1. When you install Sealed Secrets to your cluster, it creates a controller that manages a RSA certificate internally.
2. Then you use kubeseal utility, a cli tool provided by Sealed Secrets, to encrypt Secret resources to SealedSecret resources.
3. Whenever you apply SealedSecret resources, the controller will decrypt it as Secret resources.

# Sample usage
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin > mysecret.yaml
kubeseal --format yaml -f mysecret.yaml > mysealedsecret.yaml
kubectl create -f mysealedsecret.yaml

Let discuss the benefits of using Sealed Secrets.

Firstly, everything is K8S native. When you are doing development work, you can simply reference Secret resources. When you are ready to commit your code, you just need to use kubeseal to encrypt it by using an one-off command.

Secondly, committing Secret resources can be blocked by setting pre-commit hook.

Thirdly, it can avoid value overriding issue. Secrets remains to be secrets.

You can bring you own key. I find it is quite useful because it allows you to share the same secret to multiple clusters.

How to use Sealed Secrets with Argo CD?

If you are familiar with Argo CD, you know it only allows you to set some values or values files. Actually, this restriction can be easily bypass by creating a proxy chart.

Inside the Chart.yaml, you specify the target chart in the dependency section. Then you put all the SealedSecret resources to the templates folder.

# Example proxy chart
├── Chart.yaml
├── templates
│   └── sealed-secret.yaml
└── values.yaml

In Argo CD, values files need to be stored in the same repository with the Helm charts. It is violating the best practice, as explained by Argo CD documentation itself.

So ideally, you will setup at least two repositories, which makes using proxy charts a natural move.

Base charts repository
All your target charts store in here and your proxy charts will will use these charts as sub-charts.

Proxy charts repository
This is where Argo CD sync with your proxy charts, which is representing the desired application state.

Using proxy chart is a very useful technique. Sometimes if you want to customise the chart but directly modify the target chart is not an ideal choice (maybe you are referencing a public chart), then you can use this technique to tailor-made your chart.

There are many more tricks you can do with this pattern, let say in some cases you want to share the same secret to multiple deployments, you can do something like this.

# Maybe you want to reuse the same database credential for deployments in all regions?
├── Chart.yaml
├── templates
│   └── sealed-secret.yaml
├── region-a-values.yaml
└── region-b-values.yaml

Exercise

I have setup a minimal setup for you to experience my approach.

You can checkout my repository https://github.com/timtsoitt/argocd-proxy-charts and follow the instructions.

Conclusion

While there is no one size fit all shoes solution, keep things as simple is always a best practice. Rather than introducing many dependencies or complexities, we see a great value of using Sealed Secret with Argo CD.

If you have any ideas or questions, free feel to comment here. Also please do spend a little time to like, bookmark and share it. Thank you.