CloudBees Security Insights Overview

The CloudBees security insights report provides detailed insights into the results of security scans to users. It helps software development teams, security officers, and CISOs gain insights into security vulnerabilities and bugs so that users can quickly resolve such issues and improve the overall quality of their software. The report provides a single bird's eye view of how vulnerable an organization, sub-organization, or component is.

This blog aims to provide an overview of each widget, explain its value, and demonstrate how to act on the vulnerability insights presented. Users would visit this report to answer several questions:

• How vulnerable are our workflows to breach? What is the severity level?

• When vulnerabilities appear, do we know where they occur for remediation purposes? Are we improving our response time based on this information?

• Which scanner types are we using? What percentage of workflows are these in?

• What percentage of workflows are breaching SLA?

How does all this work?

Each widget uses a scalable and flexible Cassandra database to ingest data from workflows, scans, and integrations like JIRA. The data is indexed by an open search cluster, where an analytics service provides some pre-computation services so that the data is already pre-computed for all the widgets. When a user opens one of the reports, it connects to the report service via a secure API gateway for quick access.

UI overview

Before moving into each widget, let’s discuss some common UI elements:

• Filtering: Use the filters to choose the component, duration, org, and sub-org level. Please note that weeks run from Monday to Sunday. Our overview will focus on data for December 2023, cloudbees-staging org level, and all its components.

• Drill downs: Click any data point in the bold blue font for a deeper dive. Most widgets in this report allow you to view the vulnerability ID, components affected, the scanner user, and insight into the impacted line of code with a direct link to the GitHub repository.

• Hovering: Each report has a tooltip explaining its coverage. You can hover over the data for each graph type to get a breakdown.

• Viewing: All CloudBees platform pages can be viewed in either light or dark mode. We use dark mode for this blog.

Components, workflows, and successful workflow runs

Our first set of widgets aggregates information and tells you how many components and other workflow runs exist, along with their scanner status. You can see there are 489 components with 468 repos, broken out by how many components have a scanner or are without a scanner. So, out of 489 components, 51 have some kind of scanner, and 438 have no attached scanners.

Additionally, I can click on the 51 components, which tells you which scanners exist. For example, in the analytics component, snykcast and sonarqube scanners exist.

The Workflows widget provides a breakdown of workflows, highlighting those equipped with scanners versus those without security scanners. Here, it tells you that 489 components have 3,713 workflow files, composed of 2,221 branches, of which 141 workflows have some kind of scanners, and 3,572 workflows have no scanners.

You can drill down into which workflows have no scanners attached. Similarly, if you click on 246 workflows, you can see which scanner is attached to run as part of this workflow.

These 3,713 workflows ran 12,570 times, of which 504 times included a scan step and 12,066 times there was no scan step.

By better understanding your security scanner coverage across workflows and components, engineering managers, security officers, and CISOs unlock visibility into their threat level and can take action. The summary record shows how often workflows are run with or without a scanner. Typically, running workflows without scanners in the early stages of development is okay. Still, as an enterprise customer, you want to avoid having production deployment of code that was never scanned.

Vulnerabilities overview

The vulnerabilities overview widget tells you how many vulnerabilities are found, reopened, resolved, or open. Let’s define these categories:

• Found: new vulnerabilities encountered for the first time (i.e., not specific to the current duration) for a component. Reopened: vulnerabilities previously encountered and resolved are now showing up again.
• Resolved: vulnerabilities discovered earlier but not appearing in the latest scan of the specified period are considered resolved.
• Open: vulnerabilities appearing in the latest scan of the specified period.

For December, we found 415 vulnerabilities, reopened one, resolved nine, and 406 remain open. Let’s drill down into found vulnerabilities.

When I click on 415, I see a Vulnerabilities Overview screen, which includes ID, discovery date, name, stats, severity, and impacted components. Next, I decided to drill down into the vulnerability ID of CWE-259 since it has a very high severity. I now want to dig deeper into the occurrence from the snykcast scanner and see the exact repo location, if available. In this instance, it exists on line 50. This highlights how granular these reports are and how they allow security teams to troubleshoot for root cause issues.

Open and reopened vulnerabilities
The reopened vulnerabilities widget provides the mean age of open vulnerability occurrences by severity, showing 37 vulnerabilities are very high severity, 105 are high severity, 164 are medium severity, and 100 are low severity. I can further click on these numbers and get a drill down of these 37 very high vulnerabilities, which is where I should prioritize.

For example, I can drill down into CW-295 and notice the vulnerability name is some kind of improper certificate validation that needs fixing. I can click on this number of occurrences, click on number four, and see the repo, file, and line number where this is happening.

This widget also tells you the vulnerability criticality level (very high, high, medium, and low) and how long they've been open (mean, median, max) using a box plot on a candlestick chart—ideally, the higher the severity, the quicker the resolution time.

Scanner types

These widgets focus on better understanding the utilization of your security scanners.

Scan types in workflows
The scan types in workflows widget track the distribution and frequency of four scan types:

• Static application security testing (SAST): SAST takes an “inside-out” approach to finding security vulnerabilities by analyzing the app’s source code in a nonrunning state.

• Dynamic application security testing (DAST): Like SAST, DAST also focuses on finding security vulnerabilities by analyzing the source code. However, DAST differs in that it does this while the application is running. DAST solutions analyze the application from the "outside-in" by simulating cyber attack behaviors and techniques.

• Software composition analysis (SCA): This method achieves open-source software compliance and is used to identify open source components with known vulnerabilities.

• Container scanners: These tools help identify issues with your container images, such as outdated libraries, dependencies, or potential vulnerabilities. They can be used as part of a CI/CD pipeline to catch potential security issues before they make it into production.

The scan types across workflows help ensure consistent security evaluations. It also tells you how many workflow runs are part of which scanner type. For example, 156 workflows with a SAST scanner type execute 470 workflow runs.

Vulnerabilities by security scan type
Whereas the scan types in the workflows widget display the number of workflows and workflow runs by scanner type, the security scan type widget tells you how many of these vulnerabilities were found. This widget further breaks the number of vulnerabilities found by very high or high for different security scan types, so you can see how many of them are high, very high, or low per category of a security scan type.

For example, we found 44 vulnerabilities of SAST, 17 of DAST, 253 from container scans, and 112 vulnerabilities from SCA. You can click further into each of these to determine the affected component and drill down to the GitHub repo and line level.

SLA status overview by occurrences

For the SLA status overview by occurrences widget, we have defined some SLAs in the current version that should fix vulnerabilities within three days.

• On track: less than two days

• At risk: two days

• Breach: after three days, we mark it as breached.

• Please note these SLA statuses are hard-coded for all users but can be modified on an account or user basis. Please reach out to your CloudBees representative to assist here.

For all open vulnerabilities, seven are on track, zero are at risk, and 3,098 open vulnerabilities have breached the three-day marker to break our SLA. This view provides a benchmark for tracking vulnerability resolution time moving forward.

Mean Time to Resolve (MTTR) for vulnerabilities occurrences

The mean time to resolve (MTTR) for vulnerabilities occurrence tracks how long it takes users to fix vulnerabilities based on severity type. If MTTR is improving monthly, that shows the team is solving vulnerabilities more efficiently.

Further, we break these numbers down weekly to see how fast the engineering teams have resolved our vulnerabilities.

CWE top 25 vulnerabilities

The CWE Top 25 is a list compiled by MITRE that lists common security vulnerabilities with the most severe impact. This widget lets you instantly identify impacted components, how many occurrences, and the SLA status to help troubleshoot to understand better how exploitable the system is.

Within the cloudbees-staging organization, we detected five of these vulnerabilities. Further, I can click on any of these components, and this will give me a drill down of what those components are.

Next Steps

The security insights report provides an aggregated view of your vulnerability status across workflows. The report aims to provide software development teams, security officers, and CISOs with insights into security vulnerabilities to help quickly resolve issues and improve software quality. Using this report, you can quickly answer the following questions:

• Which scanners are currently in our workflows?

• How many vulnerabilities are we seeing weekly?

• What is the severity breakdown of our vulnerabilities?

• How quickly are we addressing vulnerabilities?

• What scan types are we using, and how does this relate to severity?

• How often are we breaching SLAs?

Now that you understand this report better, it’s time to start. Try the CloudBees platform for free. For complete documentation on security insights, click here.

To learn more about additional CloudBees analytics reports, visit the documentation below.

• Software delivery activity

• Flow metrics

• DORA metrics