Kubernetes has become an unavoidable part of a software infrastructure these days. If you are an enterprise or medium/large company, chances are you are already running Kubernetes clusters for your workloads. If you are a DevOps engineer, there is a good chance you are maintaining either an on-prem Kubernetes cluster or a PaaS like Amazon EKS, Microsoft AKS, or GKE. But regardless of how you run your Kubernetes clusters, you need to make sure that they are secure.
The Kubernetes API server has multiple layers of security.
- Transport security: All API communication is done via TLS (transport layer security) using valid certificates.
- Authentication: All API requests are authenticated with one of the several authentication mechanisms supported by Kubernetes.
- Authorization: All authenticated requests are authorized using one or more of the supported authorization models.
- Admission control: All authorized requests, except read/get requests, are validated by admission control modules.
Kubernetes comes with many security options out of the box, but to bulletproof your infrastructure, you need to consider many more security best practices. Today we will look into some of the vital security best practices for Kubernetes.
Kubernetes supports multiple authorization models for its API server. These are ABAC (Attributed-Based Access Control), RBAC (Role-Based Access Control), Node authorization and the Webhook mode. Out of all these, RBAC is the more secure and most widely used and is ideal for enterprises and medium to large organizations. With RBAC, you can define role-based access control that closely resembles your organization's business rules. RBAC also works great with OIDC authentication.
Most Kubernetes distributions have RBAC enabled by default. You can check this by running the command
kubectl cluster-info dump | grep authorization-mode, which should have
authorization-mode=RBAC in the output. If not, you can enable it using the
--authorization-mode flag for the API server when creating or patching the cluster. For example, setting
--authorization-mode=RBAC,Node will enable both RBAC and Node authorization on the cluster.
Once RBAC is enabled, you can create roles (
ClusterRole) and bindings (
ClusterRoleBinding) to control access to your resources. Here is an example of a role and role binding that lets users view pods and services.
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: fancy-namespace name: pod-service-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods", "services] verbs: ["get", "watch", "list"]
The below binds the above role to a specific group of users.
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: read-pods-services namespace: fancy-namespace roleRef: kind: Role #this must be Role or ClusterRole name: pod-service-reader # this must match the name of the Role or ClusterRole you wish to bind to apiGroup: rbac.authorization.k8s.io subjects: # subject can be individual users or a group of users. Group is defined in the external authentication service, in this case, an OIDC server - kind: Group name: k8s-restricted-users
Kubernetes supports many authentication mechanisms. Some of the most common are:
- Client certificates
- Basic authentication
- Tokens (Service account tokens, Bearer tokens, and so on)
- OpenID Connect (OIDC)
Out of all these authentication mechanisms, OIDC is the most secure and scalable solution. It is ideal for clusters accessed by large teams as it provides a single sign-on solution for all users and makes it easy to onboard and offboard users. It is also way more secure than other mechanisms as you don't have to store any sensitive information on a user's computer, like client secrets and passwords. You can also use features like MFA and Yubikey if supported by your OIDC provider.
OIDC combined with RBAC becomes necessary as more and more people start accessing the cluster. It becomes essential to create groups and roles and provide limited access to specific groups. You can read more about this in my previous post How to Secure Your Kubernetes Cluster with OpenID Connect and RBAC.
This one should be a no-brainer. Kubernetes has a Secret resource that can be used to store sensitive data. This is a great way to store passwords, keys, and other sensitive data. Secrets can be used for storing string data, docker config, certificates, tokens, files, and so on. Secrets can be mounted as data volumes or exposed as environment variables to be used in containers. Secrets can be plain text or encoded, but please don't be that person who uses plain text secrets.
Secrets are flexible and native to Kubernetes, so there is no reason for you not to use them. Also, make sure to implement proper RBAC for secrets so that not everyone has access to them.
Like any other software, Kubernetes also has bugs and issues. And from time to time, there might be a high severity bug that calls for a CVE. Hence, it's an excellent idea to keep the Kubernetes version up to date on the server and the CLI client. You can check the Kubernetes security and disclosure information website to see if there are known security vulnerabilities for your Kubernetes version. If you are using a managed PaaS, it should be pretty easy to upgrade, and for on-prem installations, there are tools like kOps, kubeadm, and so on, that makes it easy to upgrade clusters.
When someone has access to a Kubernetes cluster, they can access the k8s API server and SSH into the cluster nodes themselves. To limit node access, cluster access should be limited as much as possible. Disable SSH access for non-admin users. Secure the API server using OIDC and RBAC, as we saw earlier, so that only authenticated users with sufficient roles can access the API.
Securing the container images that run on the cluster is as important as securing the cluster itself. A malicious image running on a cluster could wreak havoc. Follow these best practices for container image security.
- Do not run containers as root as this would give the container unlimited access to the host. Always run the containers using a non-root user.
- Enable container image scanning in your CI/CD phase to catch known vulnerabilities using tools like clair or Anchore.
- Use minimal up-to-date official base images and remove all unwanted dependencies, packages, and debugging tools from the image as it will make it more secure and lightweight.
- Prevent loading unwanted kernel modules in the containers. These can be restricted using rules in
/etc/modprobe.d/kubernetes-blacklist.confof the node or by uninstalling the unwanted modules from the node.
- Use official verified images for popular software. Use a trusted registry for non-official images and always verify the image publisher
- Use Docker Bench for Security to audit your container images
- Use Pod security policies to limit a container's access to the host further
You can read more about it in our "Container Security: A Developer Guide".
Generally, pods within the same cluster will be able to communicate with each other, and if you have multiple clusters in the same network, there may be traffic between them as well. Do not leave this all open, as it could lead to a compromised cluster when another in the network is affected. Use Kubernetes network policies to control traffic between pods and clusters and allow only necessary traffic.
Do not run all your workloads in a single namespace. Isolating workloads in different namespaces based on business needs is more secure and easier to manage with RBAC. This way, you can fine-tune RBAC even further to let users access only what they need to see. You can also use Kubernetes network policies to isolate traffic between namespaces where applicable.
As with securing APIs and the cluster itself, it is also essential to set resource limits on how much CPU, memory, and persistent disk space is used by namespaces and resources. This secures your cluster from denial of service attacks when a particular container uses up all the resources. Resources quotas and limit ranges can be used to set limits at the namespace level, and Requests and limits can be used to set resource limits at container level as well.
Finally, it is also extremely important to monitor and audit your clusters. Enable audit logging for the cluster and use monitoring tools to keep an eye on the networking traffic to, from and within a cluster. Monitoring can be done using open-source tools, like Prometheus, Grafana, or with proprietary tools.
Furthermore, keep these infrastructure best practices also in mind when securing your Kubernetes cluster.
- Ensure that all communication is done via TLS.
- Protect etcd with TLS, Firewall, and Encryption and restrict access to it using strong credentials.
- Set up IAM access policies in a supported environment like a PaaS.
- Secure the Kubernetes Control Plane.
- Rotate infrastructure credentials frequently.
- Restrict cloud metadata API access when running in a PaaS like AWS, Azure, or GCP.
If you want to learn more about Kubernetes, OIDC, or using OIDC with Kubernetes, and security in general, check out these additional resources.
- How to Secure Your Kubernetes Cluster with OpenID Connect and RBAC
- Securing a Cluster
- OpenID Connect Tokens
- RBAC vs. ABAC: Definitions & When to Use
- OAuth 2.0 and OpenID Connect Overview
- Secure Access to AWS EKS Clusters for Admins
- Managing Multiple Okta Instances with Terraform Cloud
If you liked this tutorial, chances are you'll enjoy the others we publish. Please follow @oktadev on Twitter and subscribe to our YouTube channel to get notified when we publish new developer tutorials.