Kubernetes TLS, Demystified

This is the anniversary 10th article in this series.

🛡️ It is more than obvious a secured connection to any exposed service running in Kubernetes cluster is important.

The supposition for this article is that you wish to set up TLS (Transport Layer Security) realm for your ingress resource and that you already have a functioning ingress controller established in your cluster.

The SSL replacement technology is called Transport Layer Security (TLS) today. An enhanced version of SSL is TLS.

Similar to how SSL operates, it uses encryption to safeguard the transmission of data and information. Although SSL is still commonly utilized in the industry, the two names are frequently used interchangeably.

Getting a certificate: what paths can be taken?

A TLS/SSL certificate is the fundamental prerequisite for ingress TLS. These certificates are available to you in the following ways.

• Path one. Self-signed certificates: Our own Certificate Authority (root CA) created and signed the TLS certificate. It is a well-known, stunt choice for testing scenarios where you can "collaborate" on the root CA so that browsers will accept the certificate.

🖼️ Pic source: Bizagi

• Path two. Get an SSL certificate: for production use cases, you must purchase an SSL certificate from a reputable certificate authority that operating systems and browsers trust. But you must bear in mind that a so-called wildcard certificate suitable to protect all subdomains in a domain can cost $300+/year from major commercial issuers.

• Path three. Use a Let's Encrypt certificate: Let's Encrypt is a reputable certificate authority that issues free TLS certificates.

A few words about Let's Encrypt

It is a non-profit organization founded by enthusiasts in the field of struggle for privacy and security in 2014.

The challenge–response protocol used to automate enrolling with the certificate authority is called Automated Certificate Management Environment (ACME). It can query either Web servers or DNS servers controlled by the domain covered by the certificate to be issued.

🖼️ Pic source: Let's Encrypt

If you are interested in the implementation of the protocol, read what Adrian Colyer from SpringSource writes about them.

Each SSL certificate has an expiration date. So, before the certificate expires, you need to rotate it. For instance, Let's Ecrypt certificates have a three-month expiration date (and here they tell why).

This way, below in this article series, the author will dwell on the third path further in detail. Why? Well, since this path is interesting for its relative self-sufficiency and [relative] independence from commercial / state-owned certificate issuers. In general, the motto of this path is: "If made something with your hands, you know how it works - you're more adapted to survival!"

Of course, Let's Encrypt approach does not always fit needs, but for academic purposes and for startups it works.

But let's look at the situation "in manual mode" - just try to associate a certificate with a protected application. So,

Chicken or egg?

The ingress controller, not the ingress resource, is in charge of SSL. In other words, the ingress controller accesses the TLS certificates you provide to the ingress resource as a Kubernetes secret and incorporates them into its configuration.

Setup TLS/SSL certificates for ingress

Let's examine the procedures for setting up TLS for ingress. We'll start by launching a test application on the cluster. This application will be used to test our TLS-secured ingress.

Establish the new namespace, trial.

kubectl create namespace trial

Keep this as hello-app.yaml. The Deployment and Service objects are present.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-app
  namespace: trial
spec:
  selector:
    matchLabels:
      app: hello
  replicas: 1
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: "rbalashevich/hello-app:2.0"
---
apiVersion: v1
kind: Service
metadata:
  name: hello-service
  namespace: trial
  labels:
    app: hello
spec:
  type: ClusterIP
  selector:
    app: hello
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

Deploy the application with a command:

kubectl apply -f hello-app.yaml -n trial

Create a Kubernetes TLS Secret

It is necessary to make the SSL certificate a Kubernetes secret. It will subsequently be directed to the tls block for Ingress resources.

The server.crt (CA trust chain) and server.key (private key) SSL files are assumed to be available from a Certificate Authority, your company, or self-signed, as a last resort.

⚠️ A private key is created by you (the certificate owner) when you request your certificate with a Certificate Signing Request (CSR). Saying other words, you receive a private key when generate a CSR. You submit the CSR code to the certificate authority and keep private key in a safe place.

As for the big three public cloud providers, they have instructions for exporting certificates: AWS CM, GCP CAS, Azure KV.

It is necessary to make the SSL certificate a Kubernetes secret. It will subsequently be directed to the tls block for Ingress resources.

And yes, keep the private key (server.key)!

Let's use the server.crt and server.key files to construct a Kubernetes secret of tls type (SSL certificates). In the trial namespace, where the hello-app deployment is located, we are creating the secret.

Run the kubectl command listed below from the directory where your server is located. Supply the absolute path to the files or the .crt and .key files. The name hello-app-tls is made up.

kubectl create secret tls hello-app-tls \
    --namespace trial \
    --key server.key \
    --cert server.crt

The comparable YAML file, where you must include the contents of the .crt and .key files, is provided below.

apiVersion: v1
kind: Secret
metadata:
  name: hello-app-tls
  namespace: trial
type: kubernetes.io/tls
data:
  server.crt: |
       <crt contents here>
  server.key: |
       <private key contents here>

A Kubernetes ingress is a set of rules that can be configured to give services externally reachable URLs. Based on this understanding, to turn on secure connection, we should add tls block to Ingress object. So, in the trial namespace, we create the sample ingress TLS-capable resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-app-ingress
  namespace: trial
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - app.hosting.cloudprovider.com
    secretName: hello-app-tls
  rules:
  - host: "app.hosting.cloudprovider.com"
    http:
      paths:
        - pathType: Prefix
          path: "/"
          backend:
            service:
              name: hello-service
              port:
                number: 80

⚠️ Replace app.hosting.cloudprovider.com to your actual hostname. The host(s) should be the same in both the rules and tls blocks in the Ingress manifest. In other words, they must match.

In case of using NGINX ingress, you can add the supported annotation by the ingress controller you are using if you want a strict SSL. For instance, you can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" in the Nginx ingress controller to permit SSL traffic up until the application.

The way to make sure

Let's check with curl https://app.hosting.cloudprovider.com -kv, is the connection to the app secure now:

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=app.hosting.cloudprovider.com
*  start date: Oct 6 15:35:07 2022 GMT
*  expire date: Oct 6 15:35:07 2023 GMT
*  issuer: CN=Go Daddy Secure Certificate Authority - G2,
              OU=http://certs.godaddy.com/repository/,
              O="GoDaddy.com, Inc.",L=Scottsdale,ST=Arizona,C=US
*  SSL certificate verify ok.

🔒 If the certificate is valid, then the browser will not swear and there will be no frightening warnings either. Voilà, the connection to our app is secure!

Okay, we've covered the situations for the first and second paths. The next step is pathfinding the third path involving Let's Encrypt certificate.

Estne vita vere brevis?

Sed vita est cum dignitate vivendum. As the author noted above, the life of a certificate from a let's encrypt is short. You have to pay for insolence. Accordingly, some solution is required that would automate the re-issuance of short-lived certificates, right? And such a solution exists, it is cert-manager! It streamlines the process of getting, renewing, and using certificates by adding certificates and certificate issuers as resource types in Kubernetes clusters.

It can generate certificates from a number of supported sources, including Let's Encrypt.

Furthermore, it will check that certificates are current and valid and make an attempt to renew them for a specified period before they expire.

Install cert-manager on Kubernetes

According to the official cert-manager documentation, you can install it by using kubectl or by the provided helm chart.

# Create a dedicated Kubernetes namespace for cert-manager
kubectl create namespace cert-manager

# Add official cert-manager repository to helm CLI
helm repo add jetstack https://charts.jetstack.io

# Update Helm repository cache (think of apt update)
helm repo update

# Install cert-manager on Kubernetes
## cert-manager relies on several Custom Resource Definitions (CRDs)
helm install certmgr jetstack/cert-manager \
    --set installCRDs=true \
    --version v1.9.1 \
    --namespace cert-manager

The Issuer is responsible for issuing certificates. It is the signing authority and based on its configuration. The issuer knows how certificate requests are handled.

Cert-manager also creates several objects using different specifications such as CertificateRequest.

A Certificate resource is a readable representation of a certificate request. Certificate resources are linked to an Issuer who is responsible for requesting and renewing the certificate.

To determine if a certificate needs to be re-issued, cert-manager looks at the the spec of Certificate resource and latest CertificateRequests as well as the data in Secret containing the certificate.

Let's Encrypt: staging or production server?

An Issuer is a custom resource (CRD) which tells cert-manager how to sign a Certificate. Following this howto (section 7) the Issuer will be configured to connect to the Let's Encrypt staging server, which allows us to test everything without using up your Let's Encrypt certificate quota for the domain name.

After debugging, you can safely issue a certificate by using LE's production server.

A video describing cert-manager YAML syntax and recommended by the author of this article is 📽️ Anton Putra's good one.

Conclusion

SSL certificate acquisition was made simple by Let's Encrypt's reputation as a reliable certificate authority. Together with cert-manager tool, ops can quickly and easily assure correct transport encryption and interoperability with already-existing parts like NGINX Ingress. In addition to the example mentioned above, cert-manager can help with trickier situations like those involving wildcard SSL certificates.

If you're interested in using letsencrypt outside of a kubernetes cluster, take a look at Caddy, a 43k ⭐ open source web server, and also at Certbot, a 29k ⭐ ACME client which is open source, too.

Ever tried using wireshark to monitor web traffic? Follow Aaron Phillips from Comparitech to learn how.

Safe connections to you!