Demystifying the 18 Checks for Secure Scorecards

What are Secure Scorecards for open source projects? And how they help you produce secure software.

“No need to remake the wheel.” The reason this cliche exists is that it’s true. If something already exists and works, then recreating it from scratch is inefficient. Open-source code is the developer’s “wheel” that doesn’t need to be remade. While open-source code can make product development faster, it also comes with security risks. Working to combat this problem, the Open- Source Security Foundation (OpenSSF) and Google are working together on the “Scorecards” project to try and build stronger security throughout the software development life cycle (SDLC). Secure Scorecards act as a set of best practices, building visibility into both actual and potential exploitation of vulnerabilities in an open-source software project.

Who is OpenSSF?

Founded in August 2020, OpenSSF is a Linux Foundation-funded project focused on establishing metrics, tooling, best practices, developer identity validation, and vulnerability disclosures to improve open source software security. The organization provides security researchers a way to collaborate and address open source security supply chain issues.

What are Secure Scorecards for open source projects?

Scorecards are a way for developers to build trust, risk, and security into their products. This proactive security posture improvement initiative consists of automated “pass/fail” checks that provide risk scores. The Scorecards give developers visibility into the security of open-source code used as part of critical projects.

When the Scorecard project was announced in November 2020, OpenSSF explained that some fundamental evaluation metrics included:

• well-defined security policy
• code review process
• Continuous test coverage with fuzzing and static code analysis tools (SAST)

In June 2021, OpenSSF released version 2, which incorporates additional checks intended to help reduce risk, aligning withGoogle’s “Know, Prevent, Fix” framework. In total, V.2 runs eighteen defaultchecks against a target project.

The Eighteen Score Card Checks

Active

When running checks to determine whether an open-source project is active, the automation reviews whether the project got any commits in the last 90 days.

Primarily, this check tries to provide visibility into whether anyone is working with this resource regularly. The more recent the commits, the more likely that people are looking for vulnerabilities and enhancing security.

The scan passes if it has at least 2 commits in the past 90 days.

Automatic Dependency Update

As part of scanning for security, the Scorecards review to see if a project uses a dependabot or renovatebot. This check only reviews whether a project uses these technologies, not whether the project is updated.

Updating dependencies is a primary security practice, but it is also time-consuming. Using automation that provides security alerts for vulnerabilities in dependencies promotes secure code practices while reducing manual processes.

Binary Artifacts

The check attempts to look for binary artifacts that may be compromised. This check helps mitigate security risks from unverified code. Although there is no easy way to check code provenance, this check can help look for binary contents in a project.

To pass this check, a best practice is to build directly from a source.

Branch Protection

The branch protection reviews a project to ensure rule enforcement, like requiring approval review or status checks for pull requests. Fundamentally, these help secure code by controlling who can make changes to the branch.

The check determines whether the following are disabled:

• AllowForcePushes
• AllowDeletions

It also determines whether the following are enabled:

• EnforceAdmins
• RequireLinearHistory
• RequiredStatusChecks (must also have non-empty context enabled)
• DismissStaleReviews
• RequireCodeOwnerReviews

Finally, it also reviews that RequiredPullRequestReviews (>=1).

CI Tests

With this check, the scan reviews whether a project runs tests before merging pull requests. The automation scans for well-known CI-system names across approximately the last thirty days of commits.

If a name contains any of the following, the scan considers the CI-system “well-known”:

• Appveyor
• Buildkite
• Circleci
• e2e
• Github-actions
• Jenkins
• Mergeable
• Test
• Travis-ci

At least 75% of successful pull requests should have at least one successful test to pass the check.

CII Best Practices

The Core Infrastructure Initiative (CII) Best Practices Badge Program is a free, self-certification program that developers can use to assess whether projects are following best practices.

The criteria are grouped into six categories:

• Basics
• Change control
• Reporting
• Quality
• Security
• Analysis

The CII Best Practices check only indicates that a project has self-certified. It does not indicate whether the project earned a gold or silver badge.

Code Review

In some ways, this check overlaps with a few others. It reviews a project to determine whether a code review was completed before merging pull requests.

The project passes if 75% of the commits meet the below criteria:

• Branch-Protection enabled on the default branch
• At least one reviewer for the default branch
• Recent commit having a Github-approved review or merger is from the committer

If the check fails, then the automation analyzes for:

• Review by Prow labeled “lgtm” or “approved”

If that fails, it looks for:

• Gerrit-specific commit messages “Reviewed-on” and “Reviewed-by”

The primary security goal here is to ensure that the project undergoes appropriate review and governance to mitigate security risks. If no one regularly reviews a project’s code, vulnerabilities may exist that can lead to security incidents.

Contributors

This check provides insights into what organizations contribute to a project. It reviews to determine whether recent commits have authors from multiple companies. If contributors are from at least two different companies, the check passes.

This check provides valuable information into whether a project, based on its contributors, is trustworthy.

Fuzzing

This check reviews whether the repository name is listed on the OSS-Fuzz project list to determine whether a project uses fuzz testing.

Fuzzing can uncover security vulnerabilities like buffer overflow. OSS-Fuzz is a free service that helps open-source software developers build security into their code. The project supports:

• libFuzzer
• AFL++
• Honggfuzz
• Sanitizers
• ClusterFuzz

It currently supports the following programming languages:

• C/C++
• Rust
• Go
• Python
• Java/JVM

Frozen Deps

For this check, the automation scans for declared and pinned dependencies. It reviews across all major coding languages, including:

• Golang
• Javascript
• Python
• Ruby
• Rust

First, it scans the following root directory files:

• go.mod
• go.sum
• package-lock.json
• npm-shrinkwrap.json
• requirements.txt
• pipfile.lock
• gemfile.lock
• cargo.lock
• yarn.lock
• composer.lock
• vendor/, third_party/, third-party/

It also engages in reviews for unpinned dependencies in:

• Dockerfiles
• Shell scripts
• GitHub workflows

To pass the check, one of the root directory files and all dependencies must be pinned.

Packaging

Currently, this check only reviews for whether projects are published as downloadable packages as GitHub Packages workflows. Developers can use security advisories in their published packages to allow Dependabot to send alerts to affected repositories.

Pull Requests

The project should require pull requests when any change is made to the default branch. This check reviews recent commits, using the GitHub API to search for associated pull requests.

As part of this check, the automation:

• Discards commits for usernames containing “bot” or “gardener”
• Considers strings containing “Reviewed-on” as being reviewed through gerrit
• Does not check for the corresponding PR

By reviewing whether the projects announce changes to the default branch, the check notifies developers using the code to review any security updates made. This enhances both transparency and security by providing insight into potential code vulnerabilities.

SAST

Static code analysis systems mitigate security risk by scanning code for vulnerabilities. This check reviews projects by looking for GitHub pull requests.

The check reviews recent merged pull requests for GitHub applications named:

• “github-code-scanning” (codeql)
• sonarcloud
• “github/codeql-action”

To pass the check, more than 75% of commits need to be successful.

SAST options analyze the application’s source code or binary (this means that most, if not all, SAST tools are language-dependent). Because they never need to execute the application, they can be used by developers as soon as they’re ready to push their work to the main branch. This means the security-related issues can be identified early on in the software development life cycle (SDLC) when the cost of fixing such issues is significantly reduced. For more information on SAST versus other security testing tools, read this article.

Security Policy

The root directory should have a security policy file named “SECURITY.md” (not case sensitive). The check passes if the project contains the file, and the file is:

• Easily discoverable with a vulnerability reporter
• Contains information on what constitutes a vulnerability
• Contains a way to report the vulnerability securely

Signed Releases

While this check does not verify signatures, it does look to see if a project cryptographically signs release artifacts.

The check reviews the last five GitHub releases for:

• *.minisign
• *.asc
• *.sign

Signed releases provide visibility into versioning. This process enables a project’s users to update their open-source software to the most recent version, reducing the likelihood that they will be using software that contains a known vulnerability.

Signed Tags

Although this check does not verify the signature, it does look for cryptographically signed tags in the last five tags.

Developers often use tags to mark versions. An annotated tag can be stored as a full object in the database, providing information like:

• Tagger name
• Email
• Date

They are often signed and verified using GNU Privacy Guard (PGP). This process offers transparency that can help mitigate security risks by keeping project users updated and mitigating the risks associated with malicious code.

Token Permissions

This check focuses on whether a GitHub workflow applies the principle of least privilege. In this case, that means that the GitHub tokens apply “read-only” by default. The project passes if the check finds the yaml file for permissions keyword is set globally to “read-only.”

Setting permissions as read-only enhances security by giving only the project owners the ability to change the code. This prevents malicious users from placing vulnerabilities or malicious code into the project.

Vulnerabilities

Using the Open Source Vulnerabilities (OSV) service, this check scans for known, open, unfixed vulnerabilities.

OSV contains vulnerabilities from:

• OSS-Fuzz
• Python
• Go
• DWF

ShiftLeft for Securing Open Source Code

ShiftLeft is a member of OpenSSF, committed to establishing application security best practices with the Scorecard project to enhance open-source code security. We support their vision to create a future where participants in the open-source ecosystem use and share high-quality software, with security handled proactively by default.

---