In September 2021, CISA and NSA released new joint, technical cybersecurity guidance detailing the complexities of securing Kubernetes and providing both guidance and templates for risk mitigation and overall defense hardening across the technology.¹ Around the same time, I completed my Certificate of Cloud Security Knowledge through CSA.
So, in light of both events -- I wanted to review this guidance and share what I believe to be the most relevant aspects for practitioners today.
For anyone interested in learning this technology, I do recommend checking out CSA’s in-person and online CSSK training library.
Introduction
The joint guidance is honestly an exciting and substantive report for the continued expansion of Kubernetes across both private and public/critical infrastructure.
But before we dive into the contents of this report — you may be quietly asking yourself: what is Kubernetes, why does CISA care about hardening guidance for it, or what are their recommendations? Fear not!
In this writing, I’ll review each of these three topics and propose important next steps for putting this guidance into action. CISA takes the time to outline and specify these components with considerable detail in appendices and diagrams — some of which I’ve reproduced here to aid in the review.
Fundamentally, this topic revolves around cloud-based, containerized computing; Kubernetes is the name of the open-source container orchestration system behind it all. Today, it represents 90% of all container management in use by industry today.²
CISA’s report on hardening Kubernetes includes very familiar topics to anyone familiar with general cybersecurity best practices. One of the major advantages of containerized computing is the ability for greater flexibility and automated scaling and deployment of software applications in the cloud.³ So while Kubernetes may feel mysterious or too hard to understand unless you are an extremely experienced engineer — that is very far from the truth. This is a topic that will only increase in both use and security parlance as more enterprises move to cloud-based computing, away from more traditional on-premise servers, and adopt new Dev(Sec)Ops deployments and microservices.
General Architectural Background:
Pods are the smallest deployable Kubernetes unit and consist of one or more containers. One or more pods roll up into a cluster, which is commonly hosted using a cloud service provider (CSP). Each cluster consists of a number of control planes and one or more physical or virtual machines called worker nodes. The relationships between these entities are illustrated in the two figures below, taken from the report.
Understanding these basic components of the technology is vital to grasping the hardening recommendations within the report. To their credit, CISA seems to have understood that and dedicated time and space within the report to outline the architecture, even specifying important features of the control plane in a given cluster. For the purposes of this review, it is sufficient to associate the control plane as the decision-maker within a cluster.
Kubernetes can be a valuable target for data and/or compute power theft. While data theft is traditionally the primary motivation, cyber actors seeking computational power (often for cryptocurrency mining) are also drawn to Kubernetes to harness the underlying infrastructure. In addition to resource theft, cyber actors may also target Kubernetes to cause a denial of service.
-- National Security Agency, Cybersecurity Infrastructure & Security Agency, August 2021
3 Common Vulnerability Sources:
The threat model reviewed in the report focuses on three primary categories: supply chain risks, malicious threat actors, and insider threats. While these are certainly not unique to Kubernetes, the guidance goes into considerable detail into aspects of each category while taking an unvarnished view into why clusters are targeted for attack.
These attack opportunities include data theft, computational power theft, and denial of service. Similar to the vulnerabilities mentioned above, it’s worth noting these attack surfaces are not exclusive to Kubernetes and are thematic to general cybersecurity defense challenges. The motivation for stealing compute power from a virtual machine (VM) or worker node within a cluster, is highlighted as a current trend for directing computing resources towards cryptocurrency mining.
Supply Chain Risk
Supply chain risk is the risk that an adversary may subvert any element that makes up a system, including product components, services, or personnel that help supply the end product.⁴
A malicious container or application from a third party could provide cyber actors with a foothold in the cluster, most specifically the control plane.
Malicious Threat Actors
Kubernetes architecture exposes several APIs that cyber actors could potentially leverage for remote exploitation.⁵ The report highlights the control plan, worker nodes, and any containerized application within a targeted cluster
Cyber actors frequently take advantage of exposed control plane components lacking appropriate access controls.
Insider Threat
- The report acknowledges the fact that within a given organization there are usually few employees with elevated access privileges and given the fact that Kubernetes does not currently support two-person integrity controls — the risk for escalated privileges and specialized knowledge is high.⁶
While specifically not mentioned in the report outright, the implication of a fourth vulnerability — simple, human error is omnipresent. CISA acknowledges specifically that “Kubernetes clusters can be complex to secure and are often abused in compromises that exploit their misconfigurations.” To this end, the report effectively includes multiple boldfaced references to detailed appendices with example configurations, template code, and YAML files, and detailed “call-outs” for when the default setting for a given Kubernetes feature needs to be modified in accordance with a mitigation strategy. This “hands-on” inclusion of examples carries with it a positive consequence within the broader cybersecurity community. Armed with specific mitigation strategies, this report is not, yet another administrative framework or rulebook, but reads like a practical and useful guide for practitioners.
Hardening Mitigations:
In its guidance, the report lists 18 individual mitigations, grouping them into four major categories. A fifth general, catch-all category is included for additional best practices.
These 4 categories include: Kubernetes Pod security, network separation and hardening, authentication and authorization, and log auditing.
The report also lists a detailed appendix with substantive examples and references for each of the hardening strategies. Included below is a summary of primary themes and mitigations across the major categories.
1. Kubernetes Pod Security:
The guidance puts this category first, and with good reason. The pod is most commonly the first execution environment for the attacker upon exploiting a container. “By default, many container services run as the privileged root user, and applications execute inside the container as root despite not requiring privileged execution.
Preventing root execution by using non-root containers or a rootless container engine limits the impact of a container compromise.”⁷ The report goes on to define non-root configuration details for implementation in a Dockerfile, and appreciatively warns to avoid the use of rootless container engines as a hardening technique in any production environment until “more stable” versions compatible with Kubernetes are released by vendors.⁸
Reassuringly, common mitigation themes seen elsewhere in cybersecurity are also outlined and detailed with their applicability for pod security. Most notably including: using principles least privileged access, removing or hiding service tokens from containerized applications, and implementing pod security policies for a given cluster.
2. Network Separation and Hardening:
Separation controls are certainly not unique to Kubernetes, and general best practices for segmenting and limiting potential lateral movement or escalation within a cluster are effectively reviewed and summarized in the guidance. “Resource separation and encryption can be an effective way to limit a cyber actor’s movement and escalation within a cluster.”⁹
Additionally, namespaces — a Kubernetes architectural element with the ability to partition application elements within a given cluster — are specifically mentioned as a hardening mitigation. Namespaces do assign a label to a scope, which can be used to specify authorization rules via RBAC and networking policies.¹⁰ The hardening guidance does call out to the reader an important detail in that no network policies are applied to pods or namespaces by default.
This default — if unchanged — results in unrestricted ingress and egress traffic within the pod network. This represents a major risk, and the guidance provides a detailed resource with an example network policy for administrators.¹¹ In addition to network isolation, policies can limit storage and compute resources to provide better control over Pods at the namespace level.
The importance of hardening the control plane is also described in detail within this section. Because of the sensitive, high-consequence functions of the control plane, it deserves special attention and particular hardening actions irrespective of almost all use cases.
CISA emphasizes six actions, specific to the control plane while separately including a table listing all associated TCP ports and services.¹²
Setup TLS encryption
Setup strong authentication methods
Disable access to the internet and unnecessary, or untrusted networks
a. Kubernetes is often deployed on VMs, and therefore administrators must carefully consider access to the attack surface of these environments, paying particular attention to any VM where a worker node is running that could be compromised and escalated within the pod.Use RBAC policies to restrict access to the control plane
Secure the
etcddatastore with authentication and RBAC policies
a. Theetcdbackend database stores state information and cluster Secrets.
b. It is a critical control plane component, and gaining write access toetcdcould give a cyber actor root access to the entire cluster.Protect
kubeconfigfiles from unauthorized modifications
a. These files contain sensitive information about clusters, users, namespaces, and authentication mechanisms.
3. Authentication and Authorization:
Authentication and authorization conduct the primary heavy lifting for protecting access to all Kubernetes cluster resources. The report specifies that by default, Kubernetes does not include any user authentication as part of its package; however, multiple methods exist for adding authentication to a cluster.¹³
An interesting default is called out by the report pertaining to anonymous requests in Kubernetes 1.6 and newer. Anonymous requests are requests that are rejected by other configured authentication methods and are not tied to any individual user or Pod.
These requests are enabled by default, and the guidance explicitly calls out disabling this setting by passing —
-anonymous-auth=falsetothe API server.¹⁴ “Leaving anonymous requests enabled could allow a cyber actor to access cluster resources without authentication.”
4. Log Auditing:
Not surprisingly, the report specifies that multiple frameworks and existing guidance dictate the need (and requirement) for log auditing in cybersecurity best practices. Nonetheless, the guidance in this report maintains a practical approach to this challenge — acknowledging the needed data for audit exists, while also clearly stating that “as is” Kubernetes does not natively provide in-depth monitoring or alerting services.
Kubernetes auditing capabilities are disabled by default, so if no audit policy has been written, nothing is logged … logging Kubernetes events alone is not enough to provide a full picture of the actions occurring on the system. Logging should also be performed at the host level, application level, and on the cloud if applicable.¹⁵
-- National Security Agency, Cybersecurity Infrastructure & Security Agency, August 2021
The report gives 4 primary guidance points when implementing this work:
- Establish Pod baselines at creation to enable anomalous activity identification.
- Perform logging at the host level, application level, and on the cloud if applicable.
- Integrate existing network security tools for aggregate scans, monitoring, alerts, and analysis.
- Set up local log storage to prevent loss in case of a communication failure.
The report goes in with additional specifics from the Kubernetes API on what named events should be logged. Going even further, they provide example guidance for a baseline audit policy in the report’s appendix, shown above.
Industry Next Steps
In summary, this technical report and associated guidance represent a meaningful and substantive resource for practitioners in both the public and private sector today. Cumbersome referencing and lengthy technical jargon is a welcome absence from this report and contributes significantly to its effectiveness. It is my belief this kind of guidance from CISA directly supports and extends their mission to build a more secure and resilient infrastructure for the future. And it is my hope that more relevant, approachable, and timely guidance such as this will continue to follow, for the benefit of all of us. ✌️
Illustration thanks to work by Absurd.design, all rights reserved.
Footnotes
[1] National Security Agency Central Security Service, Press Report, Fort Meade MD: “NSA, CISA release Kubernetes Hardening Guidance” Aug 3, 2021, https://www.nsa.gov/News-Features/Feature-Stories/Article-View/Article/2716980/nsa-cisa-release-kubernetes-hardening-guidance/
[2] While other container orchestration solutions existed, in 2015 “Kubernetes” formerly Borg was released by Google as an open-source project after removing company specific components they had been using for their use business for approximately ten years prior. Google’s product ended up “winning” and is the default standard today.
“How Did Kubernetes Become so Important?” Pure Storage (blog) July 29, 2021. https://blog.purestorage.com/products/the-road-toward-modern-apps-kubernetes/
[3] For those looking to learn more, or are new to Kubernetes container concepts: “Learn Kubernetes Basics” (epub) https://kubernetes.io/docs/tutorials/kubernetes-basics/ contains the authoritative documention, diagrams, and interactive tutorials for understanding and using the technology.
[4] “NSA, CISA release Kubernetes Hardening Guidance” Aug 3, 2021. pg 5
[6] The system of storage and handling designed to prohibit individual access to certain communication security keying material by requiring the presence of at least two authorized persons, each capable of detecting incorrect or unauthorized security procedures with respect to the task being performed. NIST Glossary: CNSSI 4009–2015 from NSA/CSS Manual Number 3–16 (COMSEC) https://csrc.nist.gov/glossary/term/two_person_integrity
[7] “NSA, CISA release Kubernetes Hardening Guidance” Aug 3, 2021. pg 7.
[10] RBAC: role based access control. Additional details on using RBAC within Kubernetes can be found in their documentation “Using RBAC Authorization” https://kubernetes.io/docs/reference/access-authn-authz/rbac/, Ibid pg 13.
[12] Additional details added from the report for Kubernetes specific architectural elements. Ibid. pg 15.
[13] Kubernetes uses client certificates, bearer tokens, an authenticating proxy, or HTTP basic auth to authenticate API requests through authentication plugins. The full documentation can be found here: https://kubernetes.io/docs/reference/access-authn-authz/authentication/. Ibid. pg 19.
Top comments (0)