Describing Kubernetes deployments having best practices applied is quite verbose and maintenance intensive. Doing that for multiple applications in various stages by different teams results in lots of yam(l)mer. Helm charts provide a convenient solution. For the best deployment experience, we're maintaining and using a generic application chart.
Resources over Resources
Operating applications on Kubernetes beyond basic tutorials requires a few resources:
- the Deployment itself
- a ServiceAccount most of the time
- one to many ConfigMaps as well as Secrets
- Service and Ingress if someone shall interact with your deployment
- often a [Horizontal|Vertical]PodAutoscaler referring the deployment
As we're adopting GitOps practices for quite some time with great success - you may have heard that 😇 - the manifests of all those resources have to be literally written into a Git repository.
Mentioning GitOps... for automating rollouts there are even more (custom) resources required in order to instruct our GitOps controllers (Flux & Friends):
- the ImageRepository targeting the container registry
- a ImagePolicy declaring version patterns to automate
- and eventually a Canary as no one wants to observe rollouts manually
Observation - another great topic: To announce applications to the Prometheus monitoring stack, a ServiceMonitor resource comes on top.
Well... over time you'll probably add a PodDisruptionBudget or want to have a service mesh in place, adding furthermore resources, for instance DestinationRules and VirtualServices in case of Istio.
Wait - with Istio in place, the Ingress resource gets replaced by a VirtualService.
As we're constantly evolving, our added service mesh may get superseded by the Kubernetes Gateway API, introducing Gateway or HttpRoutes in favor of the Istio CRDs.
Guess you got the point. Writing all those manifest isn't even the hardest part. They need to be maintained constantly. Kubernetes APIs come and go, with some version lifecycle in between.
Scaling pains
Of course, some team has written all those resource manifests for the first application that was deployed. And they quickly put them into a Helm chart when it was about replication to multiple stages. The Helm chart could also be reused for further applications of that team, because - honestly - all applications ("microservices") are looking very similar.
To ensure, that changes on templated manifests don't break everything immediately, the team linted their Helm chart using different tools, hinting on upcoming Kubernetes API changes, or recommending best practices.
That's all nice and works out well. If there weren't other teams doing exactly the same but different. After a while, there were countless charts around, that all served the same purpose, but varied only slightly.
Like everywhere else, reuse is practiced by duplication, but we weren't able to copy solutions, because of the minor differences in the description of our application declarations.
Combine and generalize
So we did what was the obvious solution: We took the best from all those Helm charts and brewed a generic application chart.
This chart was even our first open sourced component. It describes a typical application workload, incorporating the requirements of many, many product teams - and guess what: they're all nearly the same.
Our application chart is tested against the latest seven Kubernetes versions (1.23 to 1.29 at the time of writing) in lots of variations using the Helm chart-testing project. Each permutation is also validated using a specialized Kubernetes manifest linter for best practices and security hardening.
Every time, someone requires something not yet possible, the chart is improved for the benefit of all. In the meanwhile, there's quite an impressive feature list, which you can easily study by reading its values.yaml - being also the documentation of the chart. Some highlights are:
- Init container and sidecar configuration without hassle
- Google Kubernetes Engine (GKE) support on workload identity and ingress config
- Automated canary deployment support using Flagger, implemented using:
- Service meshes (Istio or Linkerd) as well as the Gateway API
- Various configuration, secret and volume mount options
- Job execution before rollout for preparation tasks like database schema updates
All in all, the following [custom] resource manifests are managed by the chart (at the time of writing):
- flagger.app Canary
- image.toolkit.fluxcd.io ImagePolicy
- image.toolkit.fluxcd.io ImageRepository
- cloud.google.com BackendConfig
- networking.istio.io DestinationRule
- networking.istio.io VirtualService
- ConfigMap
- apps Deployment
- autoscaling HorizontalPodAutoscaler
- gateway.networking.k8s.io HTTPRoute
- networking.k8s.io NetworkPolicy
- policy PodDisruptionBudge
-
batch Job (as
pre-install,pre-upgrade
Helm hook) - Secret
- ServiceAccount
- Service
- monitoring.coreos.com ServiceMonitor
Needless to mention, that the highest API version available is also respected, so full compatibility to all supported Kubernetes releases is ensured.
Sample time
As stated earlier, this chart, which we're using for a wide spectrum of different types of application on different Kubernetes versions, is available for everyone.
The chart can be installed using the Helm cli
helm repo add mms-tech https://helm-charts.mms.tech
or Flux
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: mms-tech
namespace: default
spec:
interval: 120m
url: https://helm-charts.mms.tech
To give you an impression on its usage, this is how our (well-known and beloved) GitHub app Technolinator is configured:
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
name: technolinator
namespace: app
spec:
chart:
spec:
chart: application
version: "~1"
sourceRef:
kind: HelmRepository
name: chart-repo
namespace: default
interval: 60m
interval: 10m
values:
image:
repository: ghcr.io/mediamarktsaturn/technolinator
tag: 1.48.11 # {"$imagepolicy": "app:technolinator:tag"}
tagSemverRange: "~1"
secretEnvFrom:
- technolinator-config
resources:
requests:
cpu: "1"
memory: 10Gi
limits:
cpu: "6"
memory: 35Gi
container:
port: 8080
livenessProbe:
path: /q/health/live
readinessProbe:
path: /q/health/ready
monitoring:
serviceMonitor: true
metricsPath: /q/metrics
podSecurityContext:
runAsUser: 201
runAsGroup: 101
fsGroup: 101
fsGroupChangePolicy: Always
volumeMounts:
- name: data
pvcName: technolinator-data
mountPath: /data
configuration:
APP_ANALYSIS_TIMEOUT: 120M
APP_PROCESS_LOGLEVEL: DEBUG
APP_PULL_REQUESTS_CONCURRENCY_LIMIT: 10
Again: All features are explorable in the values.yaml.
We will be happy, if this chart could also help others. And we're welcome contributions of any kinds for further improvements and use cases that we have not yet considered.
The end
There's a very divisive discussion about such generic (monster/mega) charts. Sure, unique applications, provided by their projects have their own charts, and it doesn't make any sense to squeeze them into a generic one.
But in our opinion it's different for internal business applications of a company. Using our generic chart is a matter of hardening, reuse and easy knowledge sharing on application deployment configuration. The first one demanding a certain setup crafts it once, for everyone's benefit.
Especially when it comes to repetitive complex configurations like sidecars used for integration with databases or database schema update jobs, there's no need to reinvent the wheel over and over.
Reach out (for instance using GitHub issues or discussions), if you'd like to discuss with us.
get to know us 👉 https://mms.tech 👈
Top comments (4)
Hi Florian,
That's an interesting concept! If you only create one Helm chart, all these components will be installed in the same namespace right? What if multiple applications depend on the same base components?
Some manifests (e.g. the ServiceMonitor, or the ImagePolicy) have configurable namespaces, but yes, generally everthing goes into the same namespace.
What do you consider a base component?
Operators usually prefer to get installed in a specific
operator-system
namespace this and if multiple applications require a specific version of the that operator always the helm chart needs to be touched. I'd rather prefer to have multiple smaller packages that depend on each other.A base component could be the cert-manager, kube-prometheus stack or any other operator.
Think, I got it.
Our application helm chart is most of the time used for our own applications, not 3rd parties (but in rare cases).
So usually we of course use provided charts like from the prometheus-community project.
In terms of version compatibility with for instance ServiceMonitor, or other external CRDs, we use
.Capabilities.APIVersions.Has
checks, so that the appropriate API version is addressed on each environment.