Secure Kubernetes with Kubescape

Kubernetes keeps growing. According to a recent survey, its adoption among developers increased by a staggering 67% in 2021. Enterprises are migrating to Kubernetes to enjoy the flexibility and scalability of cloud-native applications.

There’s no doubt of Kubernetes’ value to enterprises, but it comes at a cost: due to its complexity, it’s remarkably easy to leave a cluster vulnerable to attacks. Enterprises know this; in its state of Kubernetes security report, Red Hat stated that 59% of respondents considered container security a threat.

Installing automated security checks in continuous integration and delivery (CI/CD), the only place where all code must pass before entering production, is the best way to start defending against threats, and Kubescape can help you do just that.

In this article, we'll learn how to run Kubescape in our CI/CD pipelines to detect threats before they are deployed.

Enter Kubescape

Kubescape is an open-source security checking tool for Kubernetes. Developed by ARMO, this tool can scan Kubernetes clusters, inspect containers, and detect unsafe deployments.

Even after years of experience running Kubernetes, enterprises are still learning the ins and out of securing a cluster. In 2019, a cryptocurrency miner somehow got into JW Player's cluster and began mining using their resources. Developers found that a container with elevated permissions was the cause. In this case, the elevated container permissions would have been detected prior to deployment had the JW Player's team used Kubescape in their deployment process.

Kubescape implements more than 70 controls based on NSA, CISA, and Microsoft’s security guidelines. These controls are grouped into four categories, called frameworks:

• NSA: follows the NSA Kubernetes Hardening Guidelines. This is a top-down framework that focuses on application security.
• MITRE: follows Microsoft’s MITRE Framework. MITRE is primarily concerned with protecting Kubernetes infrastructure.
• ArmoBest: the ARMO Framework is maintained by the same folks behind Kubescape. The framework attempts to cover the blind spots between the MITRE and NSA frameworks.
• DevOpsBest: a minimal framework covering the most essential points for infrastructure and application security.

By default, Kubescape will run all checks and print out a risk level for each framework.

Sanity controls to secure Kubernetes

We can approach Kubernetes security from two perspectives. On the most fundamental level, there is a cluster that must be properly hardened against attacks — this is especially important for self-managed and on-premises installations. On the application level, we must check that deployments follow best practices and do not leave openings to be exploited.

Secure Kubernetes with Kubescape

The first set of controls we’ll see involve securing deployment manifests, which Kubernetes uses to describe the desired state of a set of resources. By analyzing this manifest, Kubescape can detect problems such as:

• Unsecured SSH services running inside a container.
• Sudo-commands in the container start command.
• Running containers as root or with excessive capabilities
• Running pods without a parent ReplicaSet.
• Pods taking all the available CPU and memory.

You can see all the controls and available frameworks in the ArmoSec docs. Each control is associated with a risk level ranging from low to critical. At the end of the scanning process, Kubescape calculates a risk score from 0% (very secure) to 100% (not secure).

Scanning Kubernetes manifests

Let’s see Kubescape in action. For this part of the tutorial, any valid Kubernetes manifest will do. As an example, I’ll use the deployment manifest from this repository:

• semaphore-demo-nodejs-k8s

The basic command for scanning a Kubernetes manifest file is shown below:

$ kubescape scan deployment.yml

Controls: 31 (Failed: 14, Excluded: 0, Skipped: 0)

+----------+----------------------------------------+------------------+--------------------+---------------+--------------+
| SEVERITY | CONTROL NAME | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE |
+----------+----------------------------------------+------------------+--------------------+---------------+--------------+
| High | Resources CPU limit and request | 1 | 0 | 1 | 100% |
| High | Resources memory limit and request | 1 | 0 | 1 | 100% |
| Medium | Allow privilege escalation | 1 | 0 | 1 | 100% |
| Medium | CVE-2022-0492-cgroups-container-escape | 1 | 0 | 1 | 100% |
| Medium | Configured liveness probe | 1 | 0 | 1 | 100% |
| Medium | Images from allowed registry | 1 | 0 | 1 | 100% |
| Medium | Ingress and Egress blocked | 1 | 0 | 1 | 100% |
| Medium | Linux hardening | 1 | 0 | 1 | 100% |
| Medium | Non-root containers | 1 | 0 | 1 | 100% |
| Low | Configured readiness probe | 1 | 0 | 1 | 100% |
| Low | Immutable container filesystem | 1 | 0 | 1 | 100% |
| Low | K8s common labels usage | 1 | 0 | 1 | 100% |
| Low | Label usage for resources | 1 | 0 | 1 | 100% |
| Low | Resource policies | 1 | 0 | 1 | 100% |
+----------+----------------------------------------+------------------+--------------------+---------------+--------------+
| | RESOURCE SUMMARY | 1 | 0 | 1 | 40.27% |
+----------+----------------------------------------+------------------+--------------------+---------------+--------------+
FRAMEWORKS: MITRE (risk: 0.00), AllControls (risk: 40.27), ArmoBest (risk: 36.73), DevOpsBest (risk: 63.16), NSA (risk: 36.99)

Without selecting a specific framework, Kubescape evaluates all the controls. You can limit the scan to a specific framework by adding framework. For example:

$ kubescape scan framework DevOpsBest deployment.yml

You can also run individual controls by listing their ID codes

$ kubescape scan control C-0076,C-0004 deployment.yml

As you can see, Kubescape outputs a table with the controls, their state, and a final risk score.

Sanity checking Kubernetes clusters

Since Kubernetes’ default configuration causes about 47% of security issues, most self-managed clusters are at risk. Kubescape can examine the running cluster configuration, detect potential problems and offer suggestions to fix them. Here are a few of the cluster-level checks the tool can perform:

• Identify unsecured worker nodes.
• Check for exposed and unsecured interfaces, e.g. an administrative dashboard.
• Detect if the cluster is affected by any known CVEs.
• Detect missing network policies.
• Find Kubelet clients running without TLS authentication.

To scan a cluster, use the following command:

$ kubescape scan

Controls: 57 (Failed: 35, Excluded: 0, Skipped: 5)

+----------+---------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| SEVERITY | CONTROL NAME | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE |
+----------+---------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| Critical | Data Destruction | 17 | 0 | 67 | 25% |
| Critical | Disable anonymous access to Kubelet service | 0 | 0 | 0 | skipped* |
| Critical | Enforce Kubelet client TLS authentication | 0 | 0 | 0 | skipped* |
| High | Cluster-admin binding | 1 | 0 | 67 | 1% |
| High | List Kubernetes secrets | 10 | 0 | 67 | 15% |
| High | Privileged container | 1 | 0 | 6 | 14% |
| High | Resources CPU limit and request | 6 | 0 | 6 | 100% |
| High | Resources memory limit and request | 5 | 0 | 6 | 69% |
| High | Workloads with Critical vulnerabilities exposed to external traffic | 0 | 0 | 0 | skipped** |
| High | Workloads with RCE vulnerabilities exposed to external traffic | 0 | 0 | 0 | skipped** |
| High | Workloads with excessive amount of vulnerabilities | 0 | 0 | 0 | skipped** |
| High | Writable hostPath mount | 3 | 0 | 6 | 42% |
| Medium | Access container service account | 2 | 0 | 2 | 100% |
| Medium | Allow privilege escalation | 5 | 0 | 6 | 69% |
| Medium | Allowed hostPath | 3 | 0 | 6 | 42% |
| Medium | Automatic mapping of service account | 4 | 0 | 8 | 57% |
| Medium | CVE-2022-0492-cgroups-container-escape | 1 | 0 | 6 | 31% |
| Medium | Cluster internal networking | 5 | 0 | 5 | 100% |
| Medium | Configured liveness probe | 1 | 0 | 6 | 14% |
| Medium | CoreDNS poisoning | 3 | 0 | 67 | 4% |
| Medium | Delete Kubernetes events | 3 | 0 | 67 | 4% |
| Medium | Exec into container | 1 | 0 | 67 | 1% |
| Medium | HostNetwork access | 5 | 0 | 6 | 69% |
| Medium | HostPath mount | 4 | 0 | 6 | 56% |
| Medium | Ingress and Egress blocked | 6 | 0 | 6 | 100% |
| Medium | Linux hardening | 1 | 0 | 6 | 14% |
| Medium | Mount service principal | 4 | 0 | 6 | 56% |
| Medium | Namespace without service accounts | 4 | 0 | 7 | 57% |
| Medium | Network mapping | 5 | 0 | 5 | 100% |
| Medium | No impersonation | 1 | 0 | 67 | 1% |
| Medium | Non-root containers | 6 | 0 | 6 | 100% |
| Medium | Portforwarding privileges | 1 | 0 | 67 | 1% |
| Low | Audit logs enabled | 1 | 0 | 1 | 100% |
| Low | Configured readiness probe | 4 | 0 | 6 | 56% |
| Low | Immutable container filesystem | 5 | 0 | 6 | 69% |
| Low | K8s common labels usage | 6 | 0 | 6 | 100% |
| Low | Label usage for resources | 2 | 0 | 6 | 44% |
| Low | PSP enabled | 1 | 0 | 1 | 100% |
| Low | Resource policies | 6 | 0 | 6 | 100% |
| Low | Secret/ETCD encryption enabled | 1 | 0 | 1 | 100% |
+----------+---------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| | RESOURCE SUMMARY | 37 | 0 | 90 | 16.62% |
+----------+---------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
FRAMEWORKS: MITRE (risk: 12.58), AllControls (risk: 16.62), ArmoBest (risk: 13.44), DevOpsBest (risk: 40.27), NSA (risk: 17.92)

Add --enable-host-scan to install a diagnostic container on every node. This gives more detailed results, as shown below:

$ kubescape scan framework DevOpsBest --enable-host-scan

Controls: 11 (Failed: 6, Excluded: 0, Skipped: 0)

+----------+------------------------------------+------------------+--------------------+---------------+--------------+
| SEVERITY | CONTROL NAME | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE |
+----------+------------------------------------+------------------+--------------------+---------------+--------------+
| High | Resources CPU limit and request | 6 | 0 | 6 | 100% |
| High | Resources memory limit and request | 5 | 0 | 6 | 69% |
| Medium | Configured liveness probe | 1 | 0 | 6 | 14% |
| Low | Configured readiness probe | 4 | 0 | 6 | 56% |
| Low | K8s common labels usage | 6 | 0 | 6 | 100% |
| Low | Label usage for resources | 2 | 0 | 6 | 44% |
+----------+------------------------------------+------------------+--------------------+---------------+--------------+
| | RESOURCE SUMMARY | 6 | 0 | 6 | 40.27% |
+----------+------------------------------------+------------------+--------------------+---------------+--------------+
FRAMEWORK DevOpsBest

Automate security audits with CI/CD

So now that we have secured our Kubernetes environment, how do we ensure that it stays secure? We can embed sanity checks in the CI/CD pipeline in order to prevent unsafe deployments from reaching our systems.

For this, we’re going to use Semaphore as a CI/CD solution. If you’re unfamiliar with Semaphore, check the guided tour for the basics. You can follow all the steps in this tutorial with a free account.

A starter pipeline. We’ll add a few tests prior to deployment.

To start, fork the repository and switch to the fork-and-run branch. Then:

1. Add the project to Semaphore.
2. Edit the pipeline and click on Deploy to show the continuous deployment pipeline.
3. Click + Add Block.
4. Open the Prologue and enter the Kubescape installation command: curl -s https://raw.githubusercontent.com/armosec/kubescape/master/install.sh | /bin/bash
5. In the job, add the following commands. The checkout command clones the repository in the CI machine so we can access the manifest file (deployment.yml):

 checkout
 export MAX_RISK=40
 kubescape scan -t "$MAX_RISK" deployment.yml --format junit -o report.xml

1. Arrange the job dependencies in the pipeline, so that the new job runs first.

Configuring test reports

In this section, we will configure test reports, which is a feature that can analyze Kubescape results across all CI/CD runs and show them in a unified report.

1. Open the Epilogue section of the continuous deployment pipeline and enter the following line to process the report generated by Kubescape.

 [[ -f report.xml ]] && test-results publish report.xml

1. Click on Add After Jobs and enter the following command. This will gather reports from all jobs and update the dashboard:

 test-results gen-pipeline-report

1. Click on Run the workflow to test your modifications.

The pipeline should start immediately. Wait until all the jobs in the CI pipeline are done before clicking the Promote button. If the risk level exceeds the set threshold (40% in our example, but feel free to adjust), the pipeline will fail, blocking deployment to protect the cluster.

Continuous deployment pipeline with manifest sanity check installed.

Once finished, you can check out the failures found by Kubescape in the test results tab.

Test reports give you insights into manifest security problems.

Cluster pre-flight checks

Securing deployments is only half of the solution. The other half consists of continuously sanity-checking the Kubernetes cluster itself. There are two complementary ways in which Kubescape can achieve this:

• Installing Kubescape on the cluster and letting it run in the background. You can check the results in the Armo Cloud portal.
• Scanning the cluster in the CI/CD pipeline before every deployment.

So, let’s update the CI/CD pipeline to run a pre-flight check to ensure that it’s safe to deploy. Before doing that, we’ll need to upload the Kubeconfig file to Semaphore so it can connect to the cluster. That means we have to create a secret with the file in it. We’ll call this secret kubeconfig.

Finish the setup by editing the pipeline and adding a pipeline scan job to the block we created before. The command to run the check is shown below:

checkout
export MAX_RISK=40
kubescape scan --exclude-namespaces kube-system,kube-public -t $MAX_RISK --format junit -o report.xml

Before saving, open the secrets section and enable the kubeconfig entry.

Add the cluster scanning job and the kubeconfig secret to the block.

That’s it. Run the pipeline once more to check that everything is working.

The final CI/CD pipeline with all checks in place.

Conclusion

Kubernetes is a popular platform to run applications at scale, but that also makes it a prime target for attacks. Security is never something that can be tacked on as an afterthought. It is a continuous practice that should be treated as an integral component of every part of the software development cycle. And part of that practice is leveraging tools like Kubescape to secure your systems.