Best practices: 5 Risks to Assess for a Secure CI Pipeline

This article was originally posted on GitGuardian's blog

C.J. May
Information security professional and passionate programmer
with broad interests encompassing many areas of IT.
Twitter | GitHub

As the computing world continues to develop new processes for creating software, criminals continue to evolve their own techniques that exploit the flaws in those processes. DevOps is the latest trend in software development, and it is characterized by high levels of automation. More and more parts of the software development process can occur without human intervention which speeds up development. However, this is not without its drawbacks.

Less human involvement means less oversight from start to finish, and it also means more technologies to potentially exploit or abuse. Most of the risk involved is related to using sensitive information within automation, allowing for several ways to steal secrets. There are also things like code tampering to worry about. To keep your code and secrets safe, you should add the following security practices to your CI pipeline.

Third-party workflows

From the Codecov attack, we can see the impact that a single workflow can have on the security of our overall CI pipeline. Supply chain attacks are on the rise, and supply chain risk is a whole topic in and of itself (learn more about it here). Right now, it’s more important than ever to give extra attention to the things you are exposing your code and infrastructure to.

It’s common to use 3rd party workflows in a CI pipeline, but you need to understand that you are trusting that 3rd party with your code and potentially your secrets when you do so. If possible, you should look at the source code of the images and workflows you are considering using in your CI pipeline. If the workflow is tracked with version control, you could also periodically review changes to ensure that nothing suspicious has slipped in.

Due to time or availability of the source code, this may not always be possible. However, you need to weigh that against accepting the risk involved in blindly trusting a 3rd party’s workflow. At the very least, you should look for workflows that are either published by a trusted party or are popular and widely trusted by others. There will always be some risk involved in trusting a 3rd party, which reiterates the importance of taking measures that limit the damage that can be done.

Access control

The next thing you should consider when securing your CI pipeline is access control. Using key management systems, you can use features like role-based access control to determine who can use which secrets. You should always follow the principle of least privilege when determining who should have access to secrets. Fine-tuning access to the key management system level is very important; however, it’s not the only point of risk in a secret’s lifecycle.

Consider this scenario: you maintain a popular open-source repository on GitHub, and you get pull requests somewhat routinely. To help you save time in evaluating the code that is submitted, you kick off some automated tests when someone submits a new pull request.

One day someone submitted malicious code in a pull request that was grabbing environment variables and shipping them out to the author’s server. They also added a test to make sure their code would run when your CI pipeline ran. As soon as they submit the pull request, the pipeline kicks off and all the environment variables in your CI environment are stolen. Oops.

The above example isn’t even the worst-case scenario. You can read about similar scenarios like the one in this blog. In that instance, the attacker was able to pivot out of the testing environment and steal even more valuable information.

The last thing you want is an untrusted party being able to run arbitrary code in your build environment. It’s not enough that your secrets are protected in a key management system. You also need to make sure that you trust the code that will be using those secrets. Especially if you are an open-source maintainer, don’t run any workflows on pull requests until you have read the code yourself.

Hashing/signing builds

Hashing builds is more important for the security of your users than it is for your own code, but taking the concept a bit further can benefit you in some ways as well. As an example, let’s examine one of the biggest cyberattacks from 2020: the SolarWinds supply-chain attack.

SolarWinds is a leading provider of network monitoring and management tools that are used by countless organizations. In a long-term campaign, an adversary infected SolarWinds’ build servers with a custom piece of malware called Sunspot. Sunspot was a very advanced malware that monitored running processes on the build servers, and it specifically was looking for processes involved in the compilation of SolarWinds’ Orion product.

When Sunspot would see Orion being compiled on the build server, it would inject extra code, later dubbed “Sunburst,” into the compiled software. Sunburst was a backdoor that the attackers used to get access to every organization that ran Orion. You can read more about the SolarWinds attack on CrowdStrike’s blog.

So, what is our takeaway from this? Because the build itself was being infected, it was being signed by SolarWinds and was actually part of the product they were releasing. In order to detect such an attack in the future, SolarWinds has stated that they are investigating ways to validate concurrent builds against each other. This could be a challenge given the sensitivity of hashing algorithms, but it is still something to explore to be able to detect build injections.

Protecting your build environment

From the SolarWinds attack, we can clearly see that the build environment isn’t just a developer sandbox — it’s an extremely sensitive asset that requires protection. Sunspot was able to stay on SolarWinds’ build servers for a long time, but there were definitely IOCs (indicators of compromise) that could have given away the attackers’ presence.

Without getting too deep into computer and network defense, there are some general best practices for securing your build servers. First, make sure that you have proper logging enabled on your build servers and forward them to a SIEM or log aggregator. Second, consider deploying an EDR agent to your build servers to get added telemetry and alerts when something suspicious happens. Third, utilize network prevention and detection tools, and treat your build servers with scrutiny.

There are too many potential layers to protecting computers to list all of them, so we will stop there. If you don’t have much experience in the computer and network defense side of security, reach out to some experts who can provide additional support or guidance. You can also look at a framework like NIST’s Cybersecurity Framework to get you pointed in the right direction.

Secrets management

If you’re an experienced software engineer or security professional, you’ve probably heard of API keys leaking from public code repositories. Maybe you’ve even experienced your own secrets getting leaked after accidentally committing them to an open-source project. Depending on the type of secret that was leaked, it could end up being a costly mistake.

The best way to protect your secrets is to practice good secrets management. A good start is to use secret management tools like Azure Key Vault or Amazon KWS that provide secure storage and identity-based access (learn more here). Using GitHub’s built-in repository secrets manager also works well depending on your use case, but it isn’t as feature-rich as a true key management service.

Another must-have for secrets management is a tool that can tell you right away if you accidentally commit a secret to your codebase. There are some different options out there, but secret detection is GitGuardian ‘s specialty. It has hundreds of built-in secret detectors and is free for open-source projects. Knowing right when you accidentally expose your secrets is crucial in protecting yourself and your code.

Even with the above practices in place, there’s still no guarantee that your secrets are safe. A while back, Codecov’s Docker image was quietly modified to leak the secrets of anyone who used it in their CI pipeline for testing. You can read about it on here, but this impacted tens of thousands of their customers/users.

Even though something like the Codecov attack is impossible to prevent by yourself, you can still limit the impact it would have on your organization. If possible, use different secrets in production than you use in your CI pipeline. That way if something like Codecov’s compromised workflow steals your keys, it won’t affect your production environment.

Conclusion

Unfortunately, cyber adversaries are always evolving their techniques, and the challenge of protecting against novel attacks will always be there. The CI pipeline is among the most newly targeted assets, and there is a lot of opportunity due to the lack of human involvement. Following all these security practices will greatly improve the security of your CI pipeline, but make sure you are always keeping up to date on the new trends in attackers’ techniques as well.