With the proper implementation of DevOps, enterprises can deliver applications and services at high velocity. According to a survey conducted by Upguard, 63% experienced an improvement in the frequency and quality of their software deployments. However, if it's not implemented strategically, it can cause catastrophic damage to your business and its reputation. According to a prediction made by Gartner, 75% of DevOps initiatives will fail to meet their expectations by the end of 2022. There are many horror stories of organizations that suffered massive backslash from DevOps failure. Let's analyze a few of the case studies and determine why they failed.
Knight Capital: Company went bankrupt within 45 minutes
In 2014, a real-time stock trading company known as "Knight Capital" suffered one of the worst nightmares during their DevOps Implementation. The company had to pay a hefty $440 million due to failed deployment. The company used an internal application called SMARS to handle buy orders in the stock market. Unfortunately, the application had many obsolete parts in its codebase. One such outdated feature called Power Peg was laying dormant in the codebase. As a result, when a new code was written in the application, it mistakenly called the Power Peg feature, which made buy orders worth billions of dollars within the next 45 minutes. Adding to the existing problem, when Knight's staff was notified about it via email, they marked it as an urgent system alert.
Lesson learned:
- Automation is a powerful tool and should be used responsibly to prevent a major crisis.
- Old processes/features need to be removed before introducing new code to contain conflicting interactions.
Gitlab: Insufficient testing led to failed restore process
Gitlab experienced a major service outage on 31st January 2017. It was due to the accidental removal of production data during routine database maintenance. Usually, a simple backup restoration can fix the problem within no time under such circumstances. However, to Gitlab's surprise, none of their previous backup procedures were executed correctly. In addition, they weren't testing the backups enough. So, when the time came to take a backup, the system failed. This was a huge wake-up call for all the tech companies.
Lesson learned:
- Backups need continuous monitoring and will work only if you test your restore processes regularly.
- Automate your backup and restoration pipelines to run them once every day.
Workflowy: Performance issue while decompressing database
A simple and elegant productivity tool known as Workflowy suffered performance issues while decompressing a single extensive database into a cluster of small databases. The staff was making architectural changes to cope with their growing business, and then they encountered the issue. They found out that decompressing databases is slowing down queries and blocking data access to users. Troubleshooting the problem, they found out that decompressing data takes up too many resources, which gave rise to all the performance issues.
Lesson learned:
- Decompressing databases can be a resource-hungry task. It can lead to performance issues or even outages.
- While performing such a task on a live website, one should avoid "slow queries" in the database.
Conclusion
There is no doubt that implementing DevOps the right way can be a boon to your organization. However, there are millions of ways things can go wrong. Whether it's databases, infrastructure, cloud vendors, or outdated code without a proper implementation strategy, your organization can suffer unrepairable damages. I have listed down three horror case studies of failed DevOps implementation. Can you relate to any of the stories? I would like to hear about your experiences!
Top comments (0)