As a developer, you build new features, fix bugs, and ship code. That's what the original job description included. Software developers bear additional responsibilities today, such as expertise in deploying or securing software. You could make a convincing case that security is a production-only concern. As long as you're working on your features, security is not essential (yet). But what if security issues arise in production? These kinds of tickets return to engineering and interrupt everything you're doing. While these situations can happen anytime, you can play an active role in preventing them in the first place—the result: Fewer high-priority tickets and interruptions.
In this blog post, let's explore tools you can use to be proactive about a class of security incidents.
Over the last few years, companies like GitHub invested heavily in making software more secure. Dependabot automatically scans your dependency lists (i.e.,
package.json) and submits pull requests if you're using a vulnerable library version. Since it's such a low-effort workflow, developers can quickly patch vulnerable packages as part of their regular workflow.
In this blog post, we want to shed some light on another aspect of software security: Production runtime. If you're building Docker Images as part of your CI/CD workflow, this blog post is for you. No matter what specific build strategy you use for your images, you likely install additional packages to ensure your code runs without issues.
Have you ever considered the security of the Docker images you run in production? If we use the current Ubuntu LTS image, it has
106 packages installed:
docker run --rm ubuntu:jammy dpkg -l | wc -l 106
Not using Ubuntu? Alpine only has 15 preinstalled packages:
docker run --rm alpine:latest apk list -i | wc -l 15
If a Docker image has 15 or over 100 installed packages, any of these packages could potentially be vulnerable to an attack. If you think, "Ok I'll patch once I see a report in the news" (like Heartbleed or Log4Shell), you're "missing out" on most patch opportunities. Most (severe) security vulnerabilities don't get a front-page listing on Hacker News.
There's no need to go through the CVE database manually. Instead, we can work towards a workflow that resembles GitHub's Dependabot. We can scan Docker Images as we build them to find out if there are any vulnerable packages and patch them. While we will never achieve 100% security,
we're getting a step closer with a low overhead process.
Introducing KubeClarity. KubeClarity is an open-source project to help you ship more secure software. While KubeClarity covers many different use cases, let's focus on image scanning for now.
Check out the KubeClarity README to find installation instructions for your specific platform. In this
tutorial, we mainly use the CLI but feel free to install the Dashboard for visualization.
To illustrate image scanning, we use this repository,
containing an example Rust workload with a vulnerable OpenSSL version. Run the following command:
kubeclarity-cli scan ghcr.io/schultyy/rust-workload:0.0.3 --input-type image -o table NAME INSTALLED FIXED-IN VULNERABILITY SEVERITY SCANNERS perl-base 5.28.1-6+deb10u1 CVE-2023-31484 HIGH grype libgcc1 1:8.3.0-6 CVE-2018-12886 HIGH grype libsystemd0 241-7~deb10u9 CVE-2019-3844 HIGH grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u4 CVE-2023-0215 HIGH grype ncurses-base 6.1+20181013-2+deb10u3 CVE-2023-29491 HIGH grype libudev1 241-7~deb10u9 CVE-2019-3844 HIGH grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u5 CVE-2023-0464 HIGH grype libstdc++6 8.3.0-6 CVE-2019-15847 HIGH grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u5 CVE-2023-2650 HIGH grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u4 CVE-2023-0286 HIGH grype libc-bin 2.28-10+deb10u2 CVE-2020-1751 HIGH grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u4 CVE-2022-4450 HIGH grype libss2 1.44.5-1+deb10u3 CVE-2022-1304 HIGH grype libsystemd0 241-7~deb10u9 CVE-2021-3997 MEDIUM grype libsystemd0 241-7~deb10u9 CVE-2022-3821 MEDIUM grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u5 CVE-2023-0466 MEDIUM grype libudev1 241-7~deb10u9 CVE-2022-3821 MEDIUM grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u4 CVE-2022-2097 MEDIUM grype libudev1 241-7~deb10u9 CVE-2021-3997 MEDIUM grype libsystemd0 241-7~deb10u9 CVE-2022-4415 MEDIUM grype libudev1 241-7~deb10u9 CVE-2022-4415 MEDIUM grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u4 CVE-2022-4304 MEDIUM grype openssl 1.1.1n-0+deb10u3 1.1.1n-0+deb10u5 CVE-2023-0465 MEDIUM grype libpcre3 2:8.39-12 CVE-2020-14155 MEDIUM grype libgcrypt20 1.8.4-5+deb10u1 CVE-2019-13627 MEDIUM grype login 1:4.5-1.1 CVE-2023-29383 LOW grype bsdutils 1:2.33.1-0.1 CVE-2021-37600 LOW grype
(Output truncated for brevity)
kubeclarity scan command outputs a list of packages for which CVEs have been filed.
openssl comes up several times, with a version that fixes the issue (See
Dockerfile and move to line 19. Line 19 right now installs a specific
RUN apt-get update && apt-get install -y openssl=1.1.1n-0+deb10u3 && rm -rf /var/lib/apt/lists/*
Update this line to install the latest version according to the
kubeclarity-cli scan result:
RUN apt-get update && apt-get install -y openssl=1.1.1n-0+deb10u5 && rm -rf /var/lib/apt/lists/*
It indicated that issues would subsequently be fixed in
1.1.1n-0+deb10u5. Therefore, we'll go with the latest version available.
Let's build a new image:
docker build -t ghcr.io/schultyy/rust-workload:0.0.4 .
To verify the issue has been fixed, we rerun KubeClarity, this time with the
LOCAL_IMAGE_SCAN=true variable set. We want to scan our local image before pushing, verifying that OpenSSL does not have any outstanding vulnerabilities:
LOCAL_IMAGE_SCAN=true kubeclarity-cli scan ghcr.io/schultyy/rust-workload:0.0.4 --input-type image -o table | grep openssl openssl 1.1.1n-0+deb10u5 CVE-2010-0928 NEGLIGIBLE grype openssl 1.1.1n-0+deb10u5 CVE-2007-6755 NEGLIGIBLE grype..
The scan still lists two
openssl entries, though only with severity of
NEGLIGIBLE. Therefore, the issue has been fixed.
Check out the KubeClarity GitHub repository to learn about additional use cases, and make sure to star the repository!