Originally published at https://developer-friendly.blog on May 6, 2024.
Kubernetes is a great orchestration tool for managing your applications and all its dependencies. However, it comes with an extensible architecture and with an unopinionated approach to many of the day-to-day operational tasks.
One of these tasks is the management of TLS certificates. This includes issuing as well as renewing certificates from a trusted Certificate Authority. This CA may be a public internet-facing application or an internal service that needs encrypted communication between parties.
In this post, we will introduce the industry de-facto tool of choice for managing certificates in Kubernetes: cert-manager. We will walk you through the installation of the operator, configuring the issuer(s), and receiving a TLS certificate as a Kubernetes Secret for the Ingress or Gateway of your application.
Finally, we will create the Gateway CRD and expose an application securely over HTTPS to the internet.
If that gets you excited, hop on and let's get started!
Introduction
If you have deployed any reverse proxy in the pre-Kubernetes era, you might have, at some point or another, bumped into the issuance and renewal of TLS certificates. The trivial approach, back in the days as well as even today, was to use certbot
1. This command-line utility abstracts you away from the complexity of the underlying CA APIs and deals with the certificate issuance and renewal for you.
Certbot is created by the Electronic Frontier Foundation (EFF) and is a great tool for managing certificates on a single server. However, when you're working at scale with many applications and services, you will benefit from the automation and integration that cert-manager2 provides.
cert-manager is a Kubernetes-native tool that extends the Kubernetes API with custom resources for managing certificates. It is built on top of the Operator Pattern3, and is a graduated project of the CNCF4.
With cert-manager, you can fetch and renew your TLS certificates behind automation, passing them along to the Ingress5 or Gateway6 of your platform to host your applications securely over HTTPS without losing the comfort of hosting your applications in a Kubernetes cluster.
With that introduction, let's kick off the installation of cert-manager.
Huge Thanks to You π€
If you're reading this, I would like to thank you for the time you spend on this blog πΉ. Whether this is your first time, or you've been here
before and have liked the content and its quality, I truly appreciate thetime you spend here.As a token of appreciation, and to celebrate with you, I would like to
share the achievements of this blog over the course of ~11 weeks since its launch (the initial commit on Feb 13, 20247).
- 10 posts published π
- 14k+ words written so far (40k+ including codes) π
- 2.5k+ views since the launch π
- 160+ clicks coming from search engines π
Here are the corresponding screenshots:
I don't run ads on this blog (yet!? π€) and my monetization plan, as of the moment, is nothing! I may switch gear at some point; financial independence and doing this full-time makes me happy honestly βΊοΈ.
But, for now, I'm just enjoying writing in Markdown format and seeing how Material for Mkdocs8 renders rich content from it.If you are interested in supporting this effort, the GitHub Sponsors program, as well as the PayPal donation link are available at the bottom of all the pages in our website.
Greatly appreciate you being here and hope you keep coming back.
Pre-requisites
Before we start, make sure you have the following set up:
- A Kubernetes cluster. We have a couple of guides in our archive if you need help setting up a cluster:
- OpenTofu v1.79
- Although not required, we will use FluxCD as a GitOps approach for our deployments. You can either follow along and use the Helm CLI instead, or follow our earlier guide for introduction to FluxCD.
- Optionally, External Secrets Operator installed. We will use it in this guide to store the credentials for the DNS01 challenge.
- We have covered the installation of ESO in our last week's guide if you're interested to learn more: External Secrets Operator: Fetching AWS SSM Parameters into Azure AKS
Step 0: Installation
cert-manager comes with a first-class support for Helm chart installation.
This makes the installation rather straightforward.
As mentioned earlier, we will install the Helm chart using FluxCD CRDs.
# cert-manager/namespace.yml
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
# cert-manager/repository.yml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: cert-manager
spec:
interval: 60m
url: https://charts.jetstack.io
# cert-manager/release.yml
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
name: cert-manager
spec:
chart:
spec:
chart: cert-manager
sourceRef:
kind: HelmRepository
name: cert-manager
version: v1.14.x
interval: 30m
maxHistory: 10
releaseName: cert-manager
targetNamespace: cert-manager
timeout: 2m
valuesFrom:
- kind: ConfigMap
name: cert-manager-config
Although not required, it is hugely beneficial to store the Helm values as it is in your VCS. This makes your future upgrades and code reviews easier.
helm repo add jetstack https://charts.jetstack.io
helm repo update jetstack
helm show values jetstack/cert-manager \
--version v1.14.x > cert-manager/values.yml
# cert-manager/values.yml
# NOTE: truncated for brevity ...
# In a production setup, the whole file will be stored in VCS as is!
installCRDs: true
Additionally, we will use Kubernetes Kustomize10:
# cert-manager/kustomizeconfig.yml
nameReference:
- kind: ConfigMap
version: v1
fieldSpecs:
- path: spec/valuesFrom/name
kind: HelmRelease
# cert-manager/kustomization.yml
configurations:
- kustomizeconfig.yml
configMapGenerator:
- files:
- values.yaml=./values.yml
name: cert-manager-config
resources:
- namespace.yml
- repository.yml
- release.yml
namespace: cert-manager
Notice the namespace we are instructing Kustomization to place the resources in. The FluCD Kustomization CRD will be created in the flux-system
namespace, while the Helm release itself is placed in the cert-manager
namespace.
Ultimately, to create this stack, we will create a FluxCD Kustomization resource11:
# cert-manager/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cert-manager
namespace: flux-system
spec:
force: false
interval: 10m0s
path: ./cert-manager
prune: true
sourceRef:
kind: GitRepository
name: flux-system
wait: true
You may either advantage from the recursive reconciliation of FluxCD, add it to your root Kustomization or apply the resources manually from your command line.
kubectl apply -f cert-manager/kustomize.yml
Build Kustomization
A good practice is to build your Kustomization locally and optionally apply
them as a dry-run to debug any potential typo or misconfiguration.kustomize build ./cert-manager
And the output:
apiVersion: v1 kind: Namespace metadata: name: cert-manager --- apiVersion: v1 data: values.yaml: | # NOTE: truncated for brevity ... # In a production setup, the whole file will be stored in VCS as is! installCRDs: true kind: ConfigMap metadata: name: cert-manager-config-8b8tf9hfb4 namespace: cert-manager --- apiVersion: helm.toolkit.fluxcd.io/v2beta2 kind: HelmRelease metadata: name: cert-manager namespace: cert-manager spec: chart: spec: chart: cert-manager sourceRef: kind: HelmRepository name: cert-manager version: v1.14.x interval: 30m maxHistory: 10 releaseName: cert-manager targetNamespace: cert-manager timeout: 2m valuesFrom: - kind: ConfigMap name: cert-manager-config-8b8tf9hfb4 --- apiVersion: source.toolkit.fluxcd.io/v1beta2 kind: HelmRepository metadata: name: cert-manager namespace: cert-manager spec: interval: 60m url: https://charts.jetstack.io
Step 1.0: Issuer 101
In general, you can fetch your TLS certificate in two ways: either by verifying your domain using the HTTP01 challenge or the DNS01 challenge. Each have their own pros and cons, but both are just to make sure that you own the domain you're requesting the certificate for. Imagine a world where you could request a certificate for google.com
without owning it! π±
The HTTP01 challenge requires you to expose a specific path on your web server and asking the CA to send a GET request to that endpoint, expecting a specific file to be present in the response.
This is not always possible, especially if you're running a private service.
On a personal note, the HTTP01 feels like a complete hack to me. π
As such, in this guide, we'll use the DNS01 challenge. This challenge will create a specific DNS record in your nameserver. You don't specifically
have to manually do it yourself, as that is the whole point of automation that cert-manager will bring to the table.
For the DNS01 challenge, there are a couple of nameserver providers natively supported by cert-manager. You can find the list of supported providers on their website12.
For the purpose of this guide, we will provide examples for two different nameserver providers: AWS Route53 and Cloudflare.
AWS services are the indudstry standard for many companies, and Route53 is one of the most popular DNS services (fame where it's due).
Cloudflare, on the other hand, is handling a significant portion of the internet's traffic and is known for its networking capabilities across the
globe.
If you have other needs, you won't find it too difficult to find support for your nameserver provider in the cert-manager documentation.
Step 1.1: AWS Route53 Issuer
The developer-friendly.blog domain is hosted in Cloudflare and to demonstrate the AWS Route53 issuer, we will make it so that a subdomain will be resolved
by a Route53 Hosted Zone. That way, we can instruct the cert-manager controller to talk to the Route53 API for record creation and domain verfication.
# hosted-zone/variables.tf
variable "root_domain" {
type = string
default = "developer-friendly.blog"
}
variable "subdomain" {
type = string
default = "aws"
}
variable "cloudflare_api_token" {
type = string
nullable = false
sensitive = true
}
# hosted-zone/versions.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.47"
}
cloudflare = {
source = "cloudflare/cloudflare"
version = "~> 4.30"
}
}
}
provider "cloudflare" {
api_token = var.cloudflare_api_token
}
# hosted-zone/main.tf
data "cloudflare_zone" "this" {
name = var.root_domain
}
resource "aws_route53_zone" "this" {
name = format("%s.%s", var.subdomain, var.root_domain)
}
resource "cloudflare_record" "this" {
for_each = toset(aws_route53_zone.this.name_servers)
zone_id = data.cloudflare_zone.this.id
name = var.subdomain
type = "NS"
value = each.value
ttl = 1
depends_on = [
aws_route53_zone.this
]
}
# hosted-zone/outputs.tf
output "hosted_zone_id" {
value = aws_route53_zone.this.zone_id
}
output "name_servers" {
value = aws_route53_zone.this.name_servers
}
To apply this stack we'll use OpenTofu.
We could've either separated the stacks to create the Route53 zone beforehand, or we will go ahead and target our resources separately from command line as
you see below.
export TF_VAR_cloudflare_api_token="PLACEHOLDER"
export AWS_PROFILE="PLACEHOLDER"
tofu plan -out tfplan -target=aws_route53_zone.this
tofu apply tfplan
# And now the rest of the resources
tofu plan -out tfplan
tofu apply tfplan
Why Applying Two Times?
The values in a TF
for_each
must be known at the time of planning, AKA, static values13.
And since that is not the case withaws_route53_zone.this.name_servers
, we have to make sure to create the Hosted Zone first before passing its output to another resource.
We should have our AWS Route53 Hosted Zone created as you see in the screenshot
below.
Now that we have our Route53 zone created, we can proceed with the cert-manager configuration.
AWS IAM Role
We now need an IAM Role with enough permissions to create the DNS records to satisfy the DNS01 challenge14.
Make sure you have a good understanding of the
OpenID Connect, the technique we're employing in the trust relationship of the AWS IAM Role.
# route53-iam-role/variables.tf
variable "role_name" {
type = string
default = "cert-manager"
}
variable "hosted_zone_id" {
type = string
description = "The Hosted Zone ID that the role will have access to. Defaults to `*`."
default = "*"
}
variable "oidc_issuer_url" {
type = string
description = "The OIDC issuer URL of the cert-manager Kubernetes Service Account token."
nullable = false
}
variable "access_token_audience" {
type = string
default = "sts.amazonaws.com"
}
variable "service_account_name" {
type = string
default = "cert-manager"
description = "The name of the service account."
}
variable "service_account_namespace" {
type = string
default = "cert-manager"
description = "The namespace of the service account."
}
# route53-iam-role/versions.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.47"
}
tls = {
source = "hashicorp/tls"
version = "~> 4.0"
}
}
}
# route53-iam-role/main.tf
data "aws_iam_policy_document" "iam_policy" {
statement {
actions = [
"route53:GetChange",
]
resources = [
"arn:aws:route53:::change/${var.hosted_zone_id}",
]
}
statement {
actions = [
"route53:ChangeResourceRecordSets",
"route53:ListResourceRecordSets",
]
resources = [
"arn:aws:route53:::hostedzone/${var.hosted_zone_id}",
]
}
statement {
actions = [
"route53:ListHostedZonesByName",
]
resources = [
"*",
]
}
}
data "aws_iam_policy_document" "assume_role_policy" {
statement {
actions = [
"sts:AssumeRoleWithWebIdentity"
]
effect = "Allow"
principals {
type = "Federated"
identifiers = [
aws_iam_openid_connect_provider.this.arn
]
}
condition {
test = "StringEquals"
variable = "${aws_iam_openid_connect_provider.this.url}:aud"
values = [
var.access_token_audience
]
}
condition {
test = "StringEquals"
variable = "${aws_iam_openid_connect_provider.this.url}:sub"
values = [
"system:serviceaccount:${var.service_account_namespace}:${var.service_account_name}",
]
}
}
}
data "tls_certificate" "this" {
url = var.oidc_issuer_url
}
resource "aws_iam_openid_connect_provider" "this" {
url = var.oidc_issuer_url
client_id_list = [
var.access_token_audience
]
thumbprint_list = [
data.tls_certificate.this.certificates[0].sha1_fingerprint
]
}
resource "aws_iam_role" "this" {
name = var.role_name
inline_policy {
name = "${var.role_name}-route53"
policy = data.aws_iam_policy_document.iam_policy.json
}
assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
}
# route53-iam-role/outputs.tf
output "iam_role_arn" {
value = aws_iam_role.this.arn
}
output "service_account_name" {
value = var.service_account_name
}
output "service_account_namespace" {
value = var.service_account_namespace
}
output "access_token_audience" {
value = var.access_token_audience
}
tofu plan -out tfplan -var=oidc_issuer_url="KUBERNETES_OIDC_ISSUER_URL"
tofu apply tfplan
If you don't know what OpenID Connect is and what we're doing here, you might want to check out our ealier guides on the following topics:
- Establishing a trust relationship between bare-metal Kubernetes cluster and AWS IAM
- Same concept of trust relationship, this time between Azure AKS and AWS IAM
The gist of both articles is that we are providing a means for the two services to talk to each other securely and without storing long-lived credentials.
In essence, one service will issue the tokens (Kubernetes cluster), and the other will trust the tokens of the said service (AWS IAM).
Kubernetes Service Account
Now that we have our IAM role set up, we can pass that information to the cert-manager Deployment. This way the cert-manager will assume that role with the Web Identity Token flow15 (there are five flows in
total).
We will also create a ClusterIssuer CRD to be responsible for fetching the TLS certificates from the trusted CA.
# route53-issuer/variables.tf
variable "role_arn" {
type = string
default = null
}
variable "kubeconfig_path" {
type = string
default = "~/.kube/config"
}
variable "kubeconfig_context" {
type = string
default = "k3d-k3s-default"
}
variable "field_manager" {
type = string
default = "flux-client-side-apply"
}
variable "access_token_audience" {
type = string
default = "sts.amazonaws.com"
}
variable "chart_url" {
type = string
default = "https://charts.jetstack.io"
}
variable "chart_name" {
type = string
default = "cert-manager"
}
variable "release_name" {
type = string
default = "cert-manager"
}
variable "release_namespace" {
type = string
default = "cert-manager"
}
variable "release_version" {
type = string
default = "v1.14.x"
}
# route53-issuer/versions.tf
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.29"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.13"
}
}
}
provider "kubernetes" {
config_path = var.kubeconfig_path
config_context = var.kubeconfig_context
}
provider "helm" {
kubernetes {
config_path = var.kubeconfig_path
config_context = var.kubeconfig_context
}
}
# route53-issuer/values.yml.tftpl
extraEnv:
- name: AWS_ROLE_ARN
value: ${sa_role_arn}
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/aws/token
volumeMounts:
- name: token
mountPath: /var/run/secrets/aws
readOnly: true
volumes:
- name: token
projected:
sources:
- serviceAccountToken:
audience: ${sa_audience}
expirationSeconds: 3600
path: token
securityContext:
fsGroup: 1001
# route53-issuer/main.tf
data "terraform_remote_state" "iam_role" {
count = var.role_arn != null ? 0 : 1
backend = "local"
config = {
path = "../route53-iam-role/terraform.tfstate"
}
}
data "terraform_remote_state" "hosted_zone" {
backend = "local"
config = {
path = "../hosted-zone/terraform.tfstate"
}
}
locals {
sa_audience = coalesce(var.access_token_audience, data.terraform_remote_state.iam_role[0].outputs.access_token_audience)
sa_role_arn = coalesce(var.role_arn, data.terraform_remote_state.iam_role[0].outputs.iam_role_arn)
}
resource "helm_release" "cert_manager" {
name = var.release_name
repository = var.chart_url
chart = var.chart_name
version = var.release_version
namespace = var.release_namespace
reuse_values = true
values = [
templatefile("${path.module}/values.yml.tftpl", {
sa_audience = local.sa_audience,
sa_role_arn = local.sa_role_arn
})
]
}
resource "kubernetes_manifest" "cluster_issuer" {
manifest = yamldecode(<<-EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: route53-issuer
spec:
acme:
email: admin@developer-friendly.blog
enableDurationFeature: true
privateKeySecretRef:
name: route53-issuer
server: https://acme-v02.api.letsencrypt.org/directory
solvers:
- dns01:
route53:
hostedZoneID: ${data.terraform_remote_state.hosted_zone.outputs.hosted_zone_id}
region: eu-central-1
EOF
)
}
# route53-issuer/outputs.tf
output "cluster_issuer_name" {
value = kubernetes_manifest.cluster_issuer.manifest.metadata.name
}
tofu plan -out tfplan -var=kubeconfig_context="KUBECONFIG_CONTEXT"
tofu apply tfplan
If you're wondering why we're changing the configuration of the cert-manager Deployment with a new Helm upgrade, you will find an exhaustive discussion and my comment on the relevant GitHub issue16.
The gist of that conversation is that the cert-manager Deployment won't take into account the eks.amazonaws.com/role-arn
annotation on its Service Account, as you'd see the External Secrets Operator would. It won't even consider using the ClusterIssuer.spec.acme.solvers[*].dns01.route53.role
field for some reason! π«
That's why we're manually passing that information down to its AWS Go SDK17 using the official environment variables18.
This stack allows the cert-manager controller to talk to AWS Route53.
Notice that we didn't pass any credentials, nor did we have to create any IAM User for this communication to work. It's all the power of OpenID Connect and allows us to establish a trust relationship and never have to worry about any credentials in the client service. β
Is There a Simpler Way?
Sure there is. If you don't fancy OpenID Connect, there is always the option to pass the credentials around in your environment. That leaves you with the burden of having to rotate them every now and then, but if you're cool with that, there's nothing stopping you from going down that path. You also have the possibility of automating such rotation using less than 10 lines of code in any programming language of course.
All that said, I have to say that I consider this to be an implementation bug16; where cert-manager does not provide you with a clean interface to easily pass around IAM Role ARN. The cert-manager controller SHOULD be able to assume the role it is given with the web identity flow!
Regardless of such shortage, in this section, I'll provide you a simpler way around this.
Bear in mind that I do not recommend this approach, and wouldn't use it in my own environments either. π€·
The idea is to use our previously deployed ESO and pass the AWS IAM User credentials to the cert-manager controller (easy peasy, no drama!).
# iam-user/variables.tf
variable "user_name" {
type = string
default = "cert-manager"
}
variable "hosted_zone_id" {
type = string
description = "The Hosted Zone ID that the role will have access to. Defaults to `*`."
default = "*"
}
# iam-user/versions.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.47"
}
}
}
# iam-user/main.tf
data "aws_iam_policy_document" "iam_policy" {
statement {
actions = [
"route53:GetChange",
]
resources = [
"arn:aws:route53:::change/${var.hosted_zone_id}",
]
}
statement {
actions = [
"route53:ChangeResourceRecordSets",
"route53:ListResourceRecordSets",
]
resources = [
"arn:aws:route53:::hostedzone/${var.hosted_zone_id}",
]
}
statement {
actions = [
"route53:ListHostedZonesByName",
]
resources = [
"*",
]
}
}
resource "aws_iam_user" "this" {
name = var.user_name
}
resource "aws_iam_access_key" "this" {
user = aws_iam_user.this.name
}
resource "aws_ssm_parameter" "access_key" {
for_each = {
"/cert-manager/access-key" = aws_iam_access_key.this.id
"/cert-manager/secret-key" = aws_iam_access_key.this.secret
}
name = each.key
type = "SecureString"
value = each.value
}
# iam-user/outputs.tf
output "iam_user_arn" {
value = aws_iam_user.this.arn
}
output "iam_access_key_id" {
value = aws_iam_access_key.this.id
sensitive = true
}
output "iam_access_key_secret" {
value = aws_iam_access_key.this.secret
sensitive = true
}
And now let's create the corresponding ClusterIssuer, passing the credentials like a normal human being!
# route53-issuer-creds/variables.tf
variable "kubeconfig_path" {
type = string
default = "~/.kube/config"
}
variable "kubeconfig_context" {
type = string
default = "k3d-k3s-default"
}
variable "field_manager" {
type = string
default = "flux-client-side-apply"
}
# route53-issuer-creds/versions.tf
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.29"
}
}
}
provider "kubernetes" {
config_path = var.kubeconfig_path
config_context = var.kubeconfig_context
}
# route53-issuer-creds/main.tf
data "terraform_remote_state" "hosted_zone" {
backend = "local"
config = {
path = "../hosted-zone/terraform.tfstate"
}
}
resource "kubernetes_manifest" "external_secret" {
manifest = yamldecode(<<-EOF
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: route53-issuer-aws-creds
namespace: cert-manager
spec:
data:
- remoteRef:
key: /cert-manager/access-key
secretKey: awsAccessKeyID
- remoteRef:
key: /cert-manager/secret-key
secretKey: awsSecretAccessKey
refreshInterval: 5m
secretStoreRef:
kind: ClusterSecretStore
name: aws-parameter-store
target:
immutable: false
EOF
)
}
resource "kubernetes_manifest" "cluster_issuer" {
manifest = yamldecode(<<-EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: route53-issuer
spec:
acme:
email: admin@developer-friendly.blog
enableDurationFeature: true
privateKeySecretRef:
name: route53-issuer
server: https://acme-v02.api.letsencrypt.org/directory
solvers:
- dns01:
route53:
hostedZoneID: ${data.terraform_remote_state.hosted_zone.outputs.hosted_zone_id}
region: eu-central-1
accessKeyIDSecretRef:
key: awsAccessKeyID
name: route53-issuer-aws-creds
secretAccessKeySecretRef:
key: awsSecretAccessKey
name: route53-issuer-aws-creds
EOF
)
}
# route53-issuer-creds/outputs.tf
output "external_secret_name" {
value = kubernetes_manifest.external_secret.manifest.metadata.name
}
output "external_secret_namespace" {
value = kubernetes_manifest.external_secret.manifest.metadata.namespace
}
output "cluster_issuer_name" {
value = kubernetes_manifest.cluster_issuer.manifest.metadata.name
}
We're now done with the AWS issuer. Let's switch gear for a bit to create the Cloudflare issuer before finally creating a TLS certificate for our desired domain(s).
Step 1.2: Cloudflare Issuer
Since Cloudflare does not have native support for OIDC, we will have to pass an API token to the cert-manager controller to be able to manage the DNS records on our behalf.
That's where the External Secrets Operator comes into play, again. I invite you to take a look at our last week's guide if you haven't done so already.
We will use the ExternalSecret CRD to fetch an API token from the AWS SSM Parameter Store and pass it down to our Kubernetes cluster as a Secret resource.
Notice the highlighted lines.
# cloudflare-issuer/externalsecret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: cloudflare-issuer-api-token
spec:
data:
- remoteRef:
key: /cloudflare/api-token
secretKey: cloudflareApiToken
refreshInterval: 5m
secretStoreRef:
kind: ClusterSecretStore
name: aws-parameter-store
target:
immutable: false
# cloudflare-issuer/clusterissuer.yml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: cloudflare-issuer
spec:
acme:
email: meysam@licenseware.io
enableDurationFeature: true
privateKeySecretRef:
name: cloudflare-issuer
server: https://acme-v02.api.letsencrypt.org/directory
solvers:
- dns01:
cloudflare:
apiTokenSecretRef:
key: cloudflareApiToken
name: cloudflare-issuer-api-token
email: admin@developer-friendly.blog
# cloudflare-issuer/kustomization.yml
resources:
- externalsecret.yml
- clusterissuer.yml
namespace: cert-manager
# cloudflare-issuer/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cloudflare-issuer
namespace: flux-system
spec:
interval: 5m
path: ./cloudflare-issuer
prune: true
sourceRef:
kind: GitRepository
name: flux-system
wait: true
kubectl apply -f cloudflare-issuer/kustomize.yml
That's all the issuers we aimed to create for today. One for AWS Route53 and another for Cloudflare.
We are now equipped with enough access in our Kubernetes cluster to just create the TLS certificate and never have to worry about how to verify their ownership.
With that promise, let's wrap this up with the easiest part! π
Step 2: TLS Certificate
You should have noticed by now that the root developer-friendly.blog will be resolved by Cloudflare as our initial nameserver. We also created a subdomain and a Hosted Zone in AWS Route53 to resolve the aws.
subdomain using Route53 as its nameserver.
We can now fetch a TLS certificate for each of them using our newly created ClusterIssuer resource. The rest is the responsibility of the cert-manager to verify the ownership within the cluster through the DNS01 challenge and using the access we've provided it.
# tls-certificates/aws-subdomain.yml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: aws-developer-friendly-blog
spec:
dnsNames:
- '*.aws.developer-friendly.blog'
issuerRef:
kind: ClusterIssuer
name: route53-issuer
privateKey:
rotationPolicy: Always
revisionHistoryLimit: 5
secretName: aws-developer-friendly-blog-tls
# tls-certificates/cloudflare-root.yml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: developer-friendly-blog
spec:
dnsNames:
- '*.developer-friendly.blog'
issuerRef:
kind: ClusterIssuer
name: cloudflare-issuer
privateKey:
rotationPolicy: Always
revisionHistoryLimit: 5
secretName: developer-friendly-blog-tls
# tls-certificates/kustomization.yml
resources:
- cloudflare-root.yml
- aws-subdomain.yml
namespace: cert-manager
# tls-certificates/kustomize.yml
resources:
- cloudflare-root.yml
- aws-subdomain.yml
namespace: cert-manager
kubectl apply -f tls-certificates/kustomize.yml
It'll take less than a minute to have the certificates issued and stored as Kubernetes Secrets in the same namespace as the cert-manager Deployment.
If you would like the certificates in a different namespace, you're better off creating Issuer instead of ClusterIssuer.
The final result will have a Secret with two keys: tls.crt
and tls.key
. This will look similar to what you see below.
---
- apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: aws-developer-friendly-blog
namespace: cert-manager
spec:
dnsNames:
- "*.aws.developer-friendly.blog"
issuerRef:
kind: ClusterIssuer
name: route53-issuer
privateKey:
rotationPolicy: Always
revisionHistoryLimit: 5
secretName: aws-developer-friendly-blog-tls
status:
conditions:
- lastTransitionTime: "2024-05-04T05:44:12Z"
message: Certificate is up to date and has not expired
observedGeneration: 1
reason: Ready
status: "True"
type: Ready
notAfter: "2024-07-30T04:44:12Z"
notBefore: "2024-05-04T04:44:12Z"
renewalTime: "2024-06-29T04:44:12Z"
---
apiVersion: v1
data:
tls.crt: ...truncated...
tls.key: ...truncated...
kind: Secret
metadata:
annotations:
cert-manager.io/alt-names: "*.aws.developer-friendly.blog"
cert-manager.io/certificate-name: aws-developer-friendly-blog
cert-manager.io/common-name: "*.aws.developer-friendly.blog"
cert-manager.io/ip-sans: ""
cert-manager.io/issuer-group: ""
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: route53-issuer
cert-manager.io/uri-sans: ""
labels:
controller.cert-manager.io/fao: "true"
name: aws-developer-friendly-blog-tls
namespace: cert-manager
type: kubernetes.io/tls
Step 3: Use the TLS Certificates in Gateway
At this point, we have the required ingredients to host an application within cluster and exposing it securely through HTTPS into the world.
That's exactly what we aim for at this step. But, first, let's create a Gateway CRD that will be the entrypoint to our cluster. The Gateway can be thought of as the sibling of Ingress resource, yet more handsome, more successful, more educated and more charming19.
The key point to keep in mind is that the Gateway API doesn't come with the implementation. Infact, it is unopinionated about the implementation and you can use any networking solution that fits your needs and has support for it.
In our case, and based on the personal preference and tendency of the author π, we'll use Cilium as the networking solution, both as the CNI, as well as the implementation for our Gateway API.
We have covered the Cilium installation before, but, for the sake of completeness, here's the way to do it20.
# cilium/playbook.yml
- name: Bootstrap the Kubernetes cluster
hosts: localhost
gather_facts: false
become: true
environment:
KUBECONFIG: ~/.kube/config
vars:
helm_version: v3.14.4
kube_context: k3d-k3s-default
tasks:
- name: Install Kubernetes library
ansible.builtin.pip:
name: kubernetes<30
state: present
- name: Install helm binary
ansible.builtin.shell:
cmd: "{{ lookup('ansible.builtin.url', 'https://git.io/get_helm.sh', split_lines=false) }}"
creates: /usr/local/bin/helm
environment:
DESIRED_VERSION: "{{ helm_version }}"
- name: Install Kubernetes gateway CRDs
kubernetes.core.k8s:
src: "{{ item }}"
state: present
context: "{{ kube_context }}"
loop:
- https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
- https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml
- name: Install cilium
block:
- name: Add cilium helm repository
kubernetes.core.helm_repository:
name: cilium
repo_url: https://helm.cilium.io
- name: Install cilium helm release
kubernetes.core.helm:
name: cilium
chart_ref: cilium/cilium
namespace: kube-system
state: present
chart_version: 1.15.x
kube_context: "{{ kube_context }}"
values:
gatewayAPI:
enabled: true
kubeProxyReplacement: true
encryption:
enabled: true
type: wireguard
operator:
replicas: 1
And now, let's create the Gateway CRD.
# gateway/gateway.yml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: developer-friendly-blog
spec:
gatewayClassName: cilium
listeners:
- allowedRoutes:
namespaces:
from: All
name: http
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: All
name: https
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ""
kind: Secret
name: developer-friendly-blog-tls
namespace: cert-manager
- group: ""
kind: Secret
name: aws-developer-friendly-blog-tls
namespace: cert-manager
mode: Terminate
Notice that we did not create the gatewayClassName
. It comes as battery-included with Cilium. You can find the GatewayClass
as soon as Cilium installation completes with the following command:
kubectl get gatewayclass
GatewayClass is to Gateway as IngressClass is to Ingress.
Also note that we are passing the TLS certificates to this Gateway we have created earlier. That way, the gateway will terminate and offload the SSL/TLS encryption and your upstream service will receive plaintext traffic.
However, if you have set up your mTLS the way we did with Wireguard encryption (or any other mTLS solution for that matter), node-to-node and/or pod-to-pod communications will also be encrypted.
# gateway/http-to-https-redirect.yml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: https-redirect
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: developer-friendly-blog
namespace: cert-manager
sectionName: http
rules:
- filters:
- requestRedirect:
scheme: https
statusCode: 301
type: RequestRedirect
matches:
- path:
type: PathPrefix
value: /
Though not required, the above HTTP to HTTPS redirect allows you to avoid accepting any plaintext HTTP traffic on your domain.
# gateway/kustomization.yml
resources:
- gateway.yml
- http-to-https-redirect.yml
namespace: cert-manager
# gateway/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: gateway
namespace: flux-system
spec:
interval: 5m
path: ./gateway
prune: true
sourceRef:
kind: GitRepository
name: flux-system
wait: true
kubectl apply -f gateway/kustomize.yml
Step 4: HTTPS Application
That's all the things we aimed to do today. At this point, we can create our HTTPS-only application and expose it securely to the wild internet!
# app/deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo-server
spec:
replicas: 1
selector:
matchLabels:
app: echo-server
template:
metadata:
labels:
app: echo-server
spec:
containers:
- envFrom:
- configMapRef:
name: echo-server
image: ealen/echo-server
name: echo-server
ports:
- containerPort: 80
name: http
securityContext:
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
readOnlyRootFilesystem: true
securityContext:
runAsGroup: 1000
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
# app/service.yml
apiVersion: v1
kind: Service
metadata:
name: echo-server
spec:
ports:
- name: http
port: 80
targetPort: http
type: ClusterIP
# app/httproute.yml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: echo-server
spec:
hostnames:
- echo.developer-friendly.blog
- echo.aws.developer-friendly.blog
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: developer-friendly-blog
namespace: cert-manager
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: echo-server
port: 80
weight: 1
filters:
- responseHeaderModifier:
set:
- name: Strict-Transport-Security
value: max-age=31536000; includeSubDomains; preload
type: ResponseHeaderModifier
matches:
- path:
type: PathPrefix
value: /
# app/configs.env
PORT=80
LOGS__IGNORE__PING=false
ENABLE__HOST=true
ENABLE__HTTP=true
ENABLE__REQUEST=true
ENABLE__COOKIES=true
ENABLE__HEADER=true
ENABLE__ENVIRONMENT=false
ENABLE__FILE=false
# app/kustomization.yml
resources:
- deployment.yml
- service.yml
- httproute.yml
images:
- name: ealen/echo-server
newTag: 0.9.2
configMapGenerator:
- name: echo-server
envs:
- configs.env
replacements:
- source:
kind: Deployment
name: echo-server
fieldPath: spec.template.metadata.labels
targets:
- select:
kind: Service
name: echo-server
fieldPaths:
- spec.selector
options:
create: true
- source:
kind: ConfigMap
name: echo-server
fieldPath: data.PORT
targets:
- select:
kind: Deployment
name: echo-server
fieldPaths:
- spec.template.spec.containers.[name=echo-server].ports.[name=http].containerPort
namespace: default
# app/kustomize.yml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: app
namespace: flux-system
spec:
interval: 5m
path: ./app
prune: true
sourceRef:
kind: GitRepository
name: flux-system
wait: true
kubectl apply -f app/kustomize.yml
That's everything we had to say for today. We can now easily access our application as follows:
curl -v https://echo.developer-friendly.blog -sSo /dev/null
or...
curl -v https://aws.echo.developer-friendly.blog -sSo /dev/null
...truncated...
* expire date: Jul 30 04:44:12 2024 GMT
...truncated...
Both will show that the TLS certificate is present. signed by a trusted CA, is valid and matches the domain we're trying to access. π
You shall see the same expiry date on your certificate if accessing as follows:
kubectl get certificate \
-n cert-manager \
-o jsonpath='{.items[*].status.notAfter}'
2024-07-30T04:44:12Z
As you can see, the information we get from the publicly available certificate as well as the one we get internally from our Kubernetes cluster are the same down to the second. πͺ
Conclusion
These days, I am never spinning up a Kubernetes cluster without having cert-manager installed on it as its day 1 operation task. It's such a life-saver tool to have in your toolbox and you can rest assured that the TLS certificates in your cluster are always up-to-date and valid.
If you ever had to worry about the expiry date of your certificates before, those days are behind you and you can benefit a lot by employing the cert-manager operator in your Kubernetes cluster. Use it to its full potential and you shall be served greatly.
Hope you enjoyed reading this material.
Until next time π«‘, ciao π€ and happy hacking! π¦
π§ π³
Top comments (0)