The DevOps Handbook – Enabling Safe Deployments

We learn the secrets of a safe deployment practice while continuing to study The DevOps Handbook as Joe is a cartwheeling acrobat, Michael is not, and Allen is hurting, so much.

For those of you that are reading these show notes via their podcast player, you can find this episode’s full show notes at https://www.codingblocks.net/episode140.

Sponsors

• Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard.
• Secure Code Warrior – Start gamifying your organization’s security posture today, score 5,000 points, and get a free Secure Code Warrior t-shirt.

Survey Says

News

• Rob Pike’s Rules of Programming (utexas.edu)

No, you click the button …

Enable Feedback to Safely Deploy Code

• Without a quick feedback loop:
  • Operations doesn’t like deploying developer code.
  • Developers complain about operations not wanting to deploy their code.
  • Given a button for anyone to push to deploy, nobody wants to push it.
• The solution is to deploy code with quick feedback loops.
  • If there’s a problem, fix it quickly and add new telemetry to track the fix.
  • Puts the information in front of everyone so there are no secrets.
• This encourages developers to write more tests and better code and they take more pride in releasing successful deployments.
  • An interesting side effect is developers are willing to check in smaller chunks of code because they know they’re safer to deploy and easier to reason about.
  • This also allows for frequent production releases with constant, tight feedback loops.
• Automating the deployment process isn’t enough. You must have monitoring of your telemetry integrated into that process for visibility.

Use Telemetry to Make Deployments Safer

• Always make sure you’re monitoring telemetry when doing a production release,
• If anything goes wrong, you should see it pretty immediately.
  • Nothing is “done” until it is operating as expected in the production environment.
• Just because you improve the development process, i.e. more unit tests, telemetry, etc., that doesn’t mean there won’t be issues. Having these monitors in place will enable you to find and fix these issues quicker and add more telemetry to help eliminate that particular issue from happening again going forward.
• Production deployments are one of the top causes of production issues.
  • This is why it’s so important to overlay those deployments on the metric graphs.

Pager Duty – Devs and Ops together

• Problems sometimes can go on for extremely long periods of time.
• Those problems might be sent off to a team to be worked on, but they get deprioritized in lieu of some features to be added.
  • The problems can be a major problem for operations, but not even a blip on the radar of dev.
  • Upstream work centers that are optimizing for themselves reduces performance for the overall value stream.
    • This means everyone in the value stream should share responsibility for handing operational incidents.
• When developers were awakened at 2 AM, New Relic found that issues were fixed faster than ever.
• Business goals are not achieved when features have been marked as “done”, but instead only when they are truly operating properly.

Have Developers Follow Work Downstream

• Having a developer “watch over the shoulder” of end-users can be very eye-opening.
  • This almost always leads to the developers wanting to improve the quality of life for those users.
• Developers should have to do the same for the operational side of things.
  • They should endure the pain the Ops team does to get the application running and stable.
  • When developers do this downstream, they make better and more informed decisions in what they do daily, in regards to things such as deployability, manageability, operability, etc.

Developers Self-Manage Their Production Service

• Sometimes deployments break in production because we learn operational problems too late in the cycle.
• Have developers monitor and manage the service when it first launches before handing over to operations.
  • This is practiced by Google.
  • Ops can act as consultants to assist in the process.
• Launch guidance:
  • Defect counts and severity
  • Type and frequency of pager alerts
  • Monitoring coverage
  • System architecture
  • Deployment process
  • Production hygiene
• If these items in the checklist aren’t met, they should be addressed before being deployed and managed in production.
• Any regulatory compliance necessary? If so, you now have to manage technical AND security / compliance risks.
• Create a service hand back mechanism. If a production service becomes difficult to manage, operations can hand it back to the developers.
  • Think of it as a pressure release valve.
  • Google still does this and shows a mutual respect between development and operations.

Resources We Like

• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (Amazon)
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win (Amazon)
• The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data (Amazon)
• Improve mobile user experience with Datadog Mobile Real User Monitoring (Datadog)

Tip of the Week

• Configure an interpreter using Docker (JetBrains)
  • JetBrains describes how to connect PyCharm to use Docker as the interpreter.
• BONUS: Why Date-ing is Hard (episode 102)
  • We discuss using the venv Python module to create seperate virtual environments, allowing each to have their own version dependencies. (docs.python.org)
  • To use venv,
    • Create the virtual environment: python -m venv c:\path\to\myenv
    • Activate the virtual environment: c:\path\to\myenv\Scripts\activate.bat
    • NOTE that the venv module documentation includes the variations for different OSes and shells.
• Node Anchors in YAML (yaml.org)
• Tweaks (Visual Studio Marketplace)
  • Install Tweaks to gain features, such as Presentation Mode, for Visual Studio.
• Angular state inspector (chrome web store)
• Angular Language Service (Visual Studio Marketplace)
• Angular Snippets (Version 9) (Visual Studio Marketplace)
  • NOTE that the author has similar plugins available for different Angular versions.

Episode source