DevSecOps and Shift Left Security: A Guide

What is DevSecOps methodology?

DevSecOps—short for development, security, and operations—adds security-first thinking into every phase of the software development pipeline, helping engineering teams deliver secure software with speed and at scale.

Previously, development teams performed security testing after the development cycle, meaning they handed work over to separate QA and security teams for final inspection. As software development teams adopted DevOps practices, particularly continuous integration and deployment, security reviews created costly bottlenecks by backloading important work at the final stages of the delivery pipelines. Teams only uncovered issues after features had been built, meaning they were costlier and more difficult to debug and fix.

DevSecOps integrates security into every stage of the development pipeline, providing teams with tools and resources at each phase to create safe and secure code. As a result, DevSecOps helps teams address security issues earlier and faster without slowing their organization's software delivery.

By adopting principles of DevSecOps, teams can benefit from:

• Faster and more efficient software delivery
• More secure codebase and proactive security
• Continuous feedback and faster security vulnerability patching
• Highly automated, standardized, and predictable security practices

How does shift-left security fit into DevSecOps?

Before DevSecOps, engineering teams structured their development cycles to be highly sequential, which meant completing all testing and security reviews after the planning, implementation, and integration phases.

Once changes reach the end of the development cycle, they are often more complex to debug, forcing teams to disentangle several factors all at once, such as performance, integration, and more. As a result, uncovering significant issues late in the cycle requires large amounts of rework for development teams. After receiving a long list of complex defects from testers, developers need to design and apply multiple fixes all at once and across thousands of lines of code.

Security testing late in the development life cycle creates especially painful bottlenecks throughout the delivery pipeline:

• Insufficient resource planning. By failing to include testers early in the planning phases, teams may not properly allocate resources later in the development cycle.
• Less time and more debt. Cramming testing at the end of the cycle increases the likelihood of teams skipping testing altogether, increasing technical debt, and deferring problems to later versions.
• Wasted development time. Engineers may continue to work on defective features or changes before receiving any feedback, leading to wasted energy and effort.
• Complex debugging. Systems become more complex later in the development cycle, making testing more challenging and burdensome.

Shifting left means performing testing earlier in the development cycle. In other words, testing is moved to the left on the project timeline.

Importantly, the goal is not to shift security to the left as a discrete phase; instead, teams should integrate security into every phase of development—design, implementation, verification, and so on. Many of these improvements can be introduced by automating security tests, particularly as a part of the continuous integration and deployment pipeline.

According to W. Edwards Deming, an American engineer famous for his key principles on transforming business effectiveness, engineering teams should design more secure products from the moment they start building:

Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.

Why shift left?

The benefits of earlier intervention include faster development speed and improved security throughout the organization's DevOps capabilities. Ultimately, the goal of shift-left security and DevSecOps should be to avoid critical bugs and security defects during the deployment phase, while preserving the fast flow of work.

Greater transparency

Implementing security measures throughout the value stream improves visibility into security coverage, including the number of defects discovered and any potential security blind spots.

The consistent monitoring of threats in the delivery pipeline also introduces a high degree of traceability and auditability, which helps teams iterate and improve their security controls after incidents.

More secure codebase

Shift-left security increases test coverage by encouraging more security testing during the development phase. Additionally, shift-left security enables distributed security, where more team members involved in the development process are responsible for building secure software.

Shift-left security also inspires better software design, instead of a culture of patching and hotfixes, by making teams more aware of security requirements. Teams can code with security in mind from the outset of a project, avoiding ad-hoc and clumsy fixes in the later stages of development.

Lower production costs

Shift-left security reduces the cost of development by resolving issues before introducing additional dependencies. According to IBM researchers, fixing defects is 15 times more expensive to fix during the testing phase than during the design phase. Worse, that number jumps to 100 times more expensive when defects are fixed during the maintenance phase.

Fixing security issues earlier requires less effort and rework than if teams wait until after implementation.

Organizational learning

Shift-left security introduces key security practices to development teams and equips them with tools that provide fast feedback on their work. These insights help spread knowledge about security best practices throughout the organization, creating a more security-conscious culture.

Improved response time

Better collaboration between development and security means issues can be discovered faster and fixed sooner. When coupled with continuous integration, shift-left security helps teams patch newly identified security vulnerabilities in faster release cycles, decreasing the time threat actors can take advantage of them.

Standardization and automation

DevSecOps encourages teams to adopt highly repeatable and predictable security workflows. They can apply security controls consistently across environments, including containers, databases, serverless functions, and more. Such standardization prevents security issues from accidentally slipping through security controls due to human error.

Shift-left security also introduces a high degree of automation into the software development life cycle, shortening feedback loops and eliminating handoffs. Teams can continuously decrease the workload of manual testers by iteratively and incrementally offloading routine testing to automated tooling.

Decreased time to market

Overall, shift-left security can increase delivery speed. Optimized security workflows and automation mean less wait time for developers and fewer bottlenecks when shipping new features.

How to shift left in DevSecOps

Adopting shift-left security is not a linear, binary outcome with a strict timeline; instead, it is a continuous process of security improvements over the long-term.

Define a strategy

Before changing any security workflows, engineering and security teams must first agree on standards and expectations for their shift-left initiatives. It's important they decide what shift-left security looks like at their organization, creating a roadmap of iterative, long-term improvements.

They can also perform a risk-based analysis to decide where to start implementing testing and tooling, balancing potential risk to the organization's security and resources required to implement new workflows. Incident history and can provide a list of high-priority and high-impact areas for improvement.

Understand the flow of work in your organization

Next, teams need to understand the flow of work through the software delivery pipeline. The goal is to provide context to security teams about the tools and environments used by developers. They can then create a unified test strategy outlining the tests, tools, and data required for each new security requirement.

They can also identify opportunities to improve existing workflows so they are more compatible with shift-left security. For example, teams can adopt trunk-based development to streamline automated security tests.

Implement security guardrails

When adding security guardrails, the goal is to make security a seamless part of daily work. Teams can introduce these changes into the development pipeline by:

• De-emphasizing acceptance and system level testing, while refocusing on unit testing and integration testing.
• Reducing toolchain complexity with pre-approved tools.
• Introducing testing automation, including security workflows directly into continuous integration pipelines.
• Hardening continuous integration systems with checklists to ensure teams follow best practices.

Continuously train and improve

Shift-left security requires continuous improvement to respond to the ever-changing needs of development and security teams. As part of the ongoing shift left, security teams can be more involved in the early phases of design, reviewing implementation details to advise on potential testing requirements, architecture considerations, and security considerations.

Development teams can also be kept up to date on the OWASP Top 10, testing best practices, and other security trends.

Shift left security best practices

Shift-left security needs to be implemented with careful consideration for its impact on the development process. Like all DevOps capabilities, it works best when combined with visibility, monitoring, and continuous improvement.

Optimize test environments

Security testing should also follow the test pyramid to provide the most valuable feedback to developers at the right time. It's important to implement smaller, faster tests earlier, such as static code analysis, unit tests, and smoke tests. By starting with these smaller tests, developers avoid the need to create full builds to reveal certain security issues.

Teams should run more comprehensive test suites—such as API, integration, and cross-browser testing—later in the development pipeline. These test suites should be designed to fail fast, stopping builds early if they uncover catastrophic issues.

Teams should also consider using short-lived testing environments. Long-running environments can quickly become obsolete or require significant upkeep, while temporary environments avoid maintenance costs and outdated data. For example, containers for smoking tests can be created and destroyed quickly according to predefined configurations.

Monitor pipeline performance

Shift-left security should be highly automated, with tools and tests added directly to the build process. Teams will need to maintain security tests and plan for their upkeep each development cycle.

As a result, an important part of shift-left security is consistently monitoring pipeline performance. Teams should highlight slow pipelines and jobs, as well as their success rates. The goal is to fine tune their configuration to eliminate false positives, increase speed, or flag outdated tests. They can also identify recurring security weaknesses that require additional tooling earlier in the development cycle.

Security teams involved in design

Requesting feedback from security teams during the design phase helps teams flag potential issues and highlight areas of focus during implementation. Security teams can help teams prevent complex security issues before developers have a chance to build them into the codebase.

Increase pre-approved tools

Standardization and pre-approval for dependencies, packages, toolchains, libraries encourages their use and avoids rework. Overall, it decreases the surface area for security issues by eliminating vulnerable toolchains before they're added into the codebase.

Develop a culture of visibility

Above all, teams should improve visibility into their organization's shift-left security practices. They should make it easy for all team members to understand their organization's security posture, including its strengths and areas for improvements.

Measuring shift left security

To measure the effectiveness of shift-left security, teams should improve visibility into:

• Security review time. The goal should be to decrease the time required for security reviews so they do not slow the development process.
• Percentage of security requirements integrated into automated testing. Teams should increase the coverage of security requirements performed as part of test automation, which provide faster feedback to developers.
• Percentage of development tools in use that have been pre-approved. Teams should increase their approval coverage until most or all tools used by development teams have been pre-approved by security teams.

Shift left security tools

Organizations can implement several different types of tools to help developers shift security to the left. The four main security testing methods include:

• Static Application System Testing (SAST) involves scanning source code for vulnerabilities. It's typically fast and cheap because it does not require code to compile or run.
• Dynamic Application Security Testing (DAST) involves black-boxing testing, exposing an application to common attacks.
• Interactive Application Security Testing (IAST), a hybrid of SAST and DAST, analyzes running applications and watches behaviors initiated by both manual and automated tests.
• Runtime Application Self Protection (RASP) integrates with an application to prevent attacks during runtime, blocking or preventing execution based on traffic and user behavior. Two popular tools include OpenRASP and Sqreen.

Teams can also implement security practices to further secure the software supply chain:

• Secrets detection: Teams can use tools to scan code for secrets, like API keys, encryption keys, and credentials, to prevent accidental exposure.
• Dependency scanning: Tools like WhiteSource and FOSSA scan dependencies and open source projects for security issues or identify outdated packages.

Start small, start today

As teams work in increasingly complex development environments—relying on more dependencies, tools, and open source technology—security is more important than ever. With widespread security scares, such as Log4j and SolarWinds, creating a resilient and security-focused organization can be a daunting task. In 2021 alone, the US-CERT Vulnerability database recorded more than 18,000 vulnerabilities, breaking previous records.

In spite of these challenges, DevSecOps and shift-left security testing offers teams a way to confidently and consistently write more secure code and build safer software. With small, continuous improvements, any engineering team can inspire a culture that prioritizes better security.