Major DevOps Challenges and How to Address Them

The genesis of DevOps comes from the need to break down the silos and get better ownership of the delivered product and better collaboration across teams. It entails two major components of the business space - Development and Operations.

Typically, DevOps is the practice of the development and operations teams working together from the start of the software development lifecycle (SDLC) and through deployment and operations.

This is done to increase the organization’s speed of delivery as well as have better ownership (and corresponding, better quality) of the final product.

DevOps enables enterprises to serve their customers in a better manner with continuous delivery and an enhanced quality of deliverables. However, with the many benefits that DevOps offers, there are also challenges that you may encounter while implementing DevOps.

Whether it is aligning the goals and priorities to promote cross-functional team collaboration or shifting older infrastructure models, DevOps poses certain challenges to enterprises.

Before your organization adopts DevOps, it is essential to understand these challenges and how you may address them.

1: Communication Issues Between Security and Development Teams

Developers and security teams chase seemingly contrasting goals.

Developers aim at pushing the software out of the delivery pipeline as fast as possible.

Security teams focus on security over speed, where delivering secure applications is their top priority. Which often means spending relatively significant amounts of time reviewing applications prior to each release.

Lack of collaboration and proper communication between the security and development teams often leads to confusion, delayed deliveries, and frustration from both teams.

DevOps advocates for the early involvement of security teams in the SDLC. However, there is friction between the development and security teams in the beginning stage as the developers are frequently not aware of security principles nor how to address security threats. Development teams want to quickly spin up new servers and security teams want to slow down and ensure each server is sufficiently hardened, has proper logging in place, and so forth.

While delivering high-quality services is one of the top priorities of DevOps, it also calls for enhanced security measures.

Often, security teams do not communicate properly with the development team, leaving them oblivious of the proper security actions that need to be taken. Similarly, security teams oftentimes don’t embrace the automated nature of the DevOps approach.

2: The Difficult of the Security Team Keeping Pace with the DevOps Cycle

DevOps focuses on fast delivery speed and short development cycles. Security teams seek to be very thorough in reviewing the security of the applications and their environments, for it frequently takes just one vulnerability to severely compromise an organization.

With this need to be thorough, it can take a much longer time to assess the code and its environment than it takes to develop or modify it.

While DevOps aims for rapid continuous delivery, organizations are often pushed to leave out security for the sake of speedy deliveries. Putting speed first allows misconfigurations, potential bugs, unaddressed threats, and vulnerabilities in the application, exposing it to security breaches and malicious attacks.

3: Cultural Resistance to Security

Traditionally, security testing occurred towards the end of the SDLC, right before the deployment phase. But with DevOps, security teams are integrated throughout the SDLC.

This early integration can lead to strife as the development teams are accustomed to working quickly on their own during the development stage of the lifecycle.

Development teams experience immense pressure from management to deliver development updates as fast as possible and frequently view any interaction with the security team during development as a hindrance to delivering functionality that management desires.

While the development team works towards this goal, the security of the application is often sacrificed in the fast-paced process. Many believe that integrating security early in the process can produce delayed deliveries and hence they avoid the security aspect of an application.

This is oftentimes a cultural issue, especially if the security team is viewed as the “naysayers” who continually say you can’t do something rather than saying how you can do something securely.

If the security team has developed a reputation as a “naysayer,” then it can be difficult to overcome that and build a close collaborative relationship in the DevOps environment.

4: Avoiding Risks Related to Containers and Other Tools

A DevOps environment frequently relies on cloud infrastructure and deployments, which often leaves the application exposed to potential security threats if proper measures are not put into place. Many open-source, immature, and new tools are used in the DevOps environment.

In the fast-paced delivery pipeline of DevOps, a simple bug or misconfiguration can lead to spectacular failures (such as organizations publicly exposing their administration consoles for their orchestration software like Tesla did).

A DevOps team will utilize various tools such as Ansible, Salt, Chef, Puppet, etc. along with many others. One of the most commonly utilized tools/technologies used by DevOps teams are containers.

Containers are ultra-lightweight portable packaging platforms that make it simple to deploy applications. Unfortunately, it can be difficult for security teams to assess the security of these containers.

Are safe libraries being used, are properly hardened services being spun up? Are secrets being securely stored and managed?

Frequently, these questions are not fully addressed and answered and the use of containers may introduce new risks into an organization.

But the issue isn’t just with containers, all of the tools associated with deployment need to be addressed and secured, since they are instrumental in creating the deployed application and environment.

All too often, the keys of the kingdom are associated with orchestration software and these need to be carefully scrutinized to ensure that they are secure.

5: Poor Access Controls and Secrets Management

With highly automated builds and deployments, secrets management and tight access controls are essential.

Secrets may include API tokens, SSH Keys, privileged account credentials, etc. These might be used by containers, services, employees and many more entities.
All too frequently, these critical passwords and keys are poorly managed (exposed) and are frequent targets of attackers.

Additionally, to ensure a smooth and quick workflow, DevOps teams often allow almost unrestricted access to privileged accounts such as admin, root, etc.

When multiple individuals use and share credentials of confidential accounts, and when processes run with elevated privileges, the possibility of these excessive permissions being abused increases significantly.

Best Practices to Address These Challenges

While DevOps may give rise to some security vulnerabilities and pose compatibility issues between various teams in the SDLC, there are ways to tackle these challenges.

To strengthen DevOps security, while maintaining a balance between different teams, and the need for agility, consider implementing the following practices in your organization.

1: Enforce Security-Focused Policies

The implementation of governance and effective communication is crucial in building holistic security environments.

You should define a concise, easy to understand, and transparent set of cybersecurity procedures and policies for areas such as access controls, code review, firewalls, and configuration management.

The DevOps teams should adhere to these security policies, and work together collaboratively towards a secured application.

Additionally, the concept of “infrastructure as code (IaC)”, is a cornerstone of DevOps.

IaC is the definition of the setup and configuration of virtual machines, networks, load balancers, and connection topology as code that uses the same versioning as the DevOps teams use for its application code.

While this may seem scary, it can be extremely powerful, as code (the infrastructure, the servers, routers, the configurations, etc.) can be reviewed and assessed easier to ensure that the environment is in the correct hardened configuration.

Similar to the principle where the same code generates the same binary, an IaC model also generates the same environment when it is applied.

Do you want to ensure that you have the server in the correct configuration? Easy. Deploy a new server from the approved version with the hardened configuration that is stored as code.

IaC solves the problem of environment drift in the delivery pipeline. Without IaC, teams have to maintain the settings of each deployment environment.

Inconsistency between different environments can lead to issues in the release phase. With the integration of IaC, DevOps teams can easily administer and manage the security of their applications and environments.

DevOps teams that integrate IaC work together with a unified set of security practices and tools to support infrastructure and deliver applications reliably, rapidly, and at scale.

2: Adopt a DevSecOps Model

Effective DevOps security can be achieved by encouraging cross-functional collaborations throughout the entire DevOps lifecycle. DevOps teams should not just work in-sync but also actively participate in the development lifecycle to achieve common goals of enhanced security.

Security should not be the sole responsibility of one team, instead, it should be a culture deep-rooted within the organization. When security is culturally imbued throughout an enterprise, it is known as “DevSecOps.”

It is a culture within organizations where everyone takes responsibility for adhering to security practices.

DevSecOps consists of cybersecurity functions and governance to reduce the possibility of security breaches via loose account controls and other security vulnerabilities. It goes way beyond technical tools and software, ensuring that security is a core principle of the organization.

DevSecOps encourages various teams to learn about basic security principles. All members of a team should have some core security training.

In addition to training, developers should learn how to use automated tools and software to run quick security checks. Security professionals should also be able to write code and work with APIs so that they can script and automate security checks, especially with regards to IaC.

Security teams can get involved and develop approved and hardened versions of the infrastructure for the development team to use. They can also enforce the configurations by monitoring the infrastructure code through automated means.

3: Use Automation for Speed and Scalability

Automation plays a crucial role when it comes to creating secure applications and secure environments. Automation helps mitigate the risks arising from manual errors and reduces the associated vulnerabilities and downtime.

Without automated security tools and processes, it becomes difficult for the security team to keep pace with the DevOps team. Automated tools can be used for several processes such as configuration management, vulnerability management, privileged credentials/secrets management, and code analysis among others.

Along with implementing automated tools and processes in your DevOps, another thing that’s critical is the selection of automated tools and processes.

Automated tools used for creating a secure DevOps workflow should:

• Be easy to understand and manage
• Not require security expertise
• Not give a high false-positive rate of issues
• Be integrated into the CI/CD pipeline

The goal is to help the DevOps team work efficiently and in an easier manner, not to overload them with dozens of tools or alien processes from their working environment.

The smaller the gap between the speed of the security and the DevOps team, the easier it will be to embed security as a core principle in your organization.

4: Manage Vulnerabilities Effectively

Incorporating security at the beginning phase of the SDLC helps facilitate the early detection of bugs and vulnerabilities.

With these identified vulnerabilities, you will need an efficient vulnerability management system so that you can track and prioritize how each vulnerability should be addressed (remediation, acceptance, transfer, etc).

There are 4 primary stages to a vulnerability management program:

• Determine the criticality of an asset, owners of the assets, frequency of scanning, and establish achievable timeline for remediation.
• Discover and inventory assets on the network.
• Identify vulnerabilities on the discovered assets.
• Report and remediate the identified vulnerabilities.

While you work with a vulnerability management program, you may notice a fairly high vulnerability score with time consuming remediation cycles in the beginning. However, the key is to show progress quarter by quarter, and year by year.

As teams become more familiar and educated about the vulnerability management program, the time for remediation and vulnerability scores should eventually decrease.

The most successful vulnerability management programs continuously adopt and comply with the latest risk reduction goals of the cybersecurity guidelines and policies within the organization.

5: Adopt Effective DevOps Secrets Management

Secrets are passwords, keys, and other sensitive information that must be carefully controlled.

With the move towards fast automated deployment, DevOps teams have frequently resorted to very poor secrets management, such as storing passwords in files in containers.

In the race for fast automated deployments, teams can oftentimes take shortcuts that leave very sensitive passwords and keys exposed.

For effective DevOps secrets management, you should remove confidential data such as credentials from the code, files, accounts, services and in various platforms and tools.

This involves eliminating the passwords from the code and storing them in a centralized password safe when not in use.

You can use products like Cyberark, Azure key vault, AWS secrets manager, Thycotic Secret Server, and others to store your passwords when not using them.

Privileged password management solutions will ensure that scripts and applications request use of the password from a centralized password safe. Additionally, by implementing APIs in the system, you can gain control over code, scripts, files, and embedded keys.

6: Adopt Effective Privileged Access Management

Restricting privileged access to accounts can significantly reduce the opportunities and risks for internal as well as external attackers to exploit the system. Technically, this means eliminating access to administrative or privileged accounts on end-user machines.

You should monitor every privileged account session to ensure that they are legitimate and adhere to compliance mandates.

Enforcing a restrictive privileged model also consists of limited access for developers and testers to certain development, production, and management systems.

But it should still allow them appropriate access and permissions to build images and machines from approved templates, and deploy, modify, and remediate vulnerability issues in the system.

Consider implementing a cutting-edge privileged access management solution such as OpenIAM that can automate the control, monitoring, and auditing of privileged access throughout the development lifecycle.

It should also be capable of tracking the full lifecycle of privileged credentials/secrets management.

Takeaways

DevOps has propelled organizations towards a better future by providing efficient solutions that aid in faster delivery, encourage collaboration between teams, and foster an Agile environment.

While DevOps offers ample benefits, it also introduces challenges. One of the most prominent issues with DevOps is the difficulty many organizations face integrating security into the DevOps process.

But security must be integrated. Early and effective implementation of security in DevOps can help identify vulnerabilities quickly and remediate operational weaknesses before they become an issue.

By integrating security early in the DevOps lifecycle, you can ensure that it is embedded at the very core of the system and runs throughout the lifecycle of the product. It will secure the code from the risks of data breaches and cybersecurity attacks that exploit weaknesses in the system.

This post was originally published at CypressDataDefense.com.