Getting started with CI/CD has become more easy with the DevOps platform era. GitLab provides a new pipeline editor to help with onboarding, and allow to prevent common configuration mistakes.
Pipelines and their definitions are always changing, and a work in progress similar to source code development. It is easy to create a long lasting pipeline which blocks reviews from fast feedback, or waste ten thousands of dollars of cloud compute resources.
The overall question is - how to create efficient DevSecOps pipelines, and make use of all available options: Configuration, resources, CI/CD infrastructure, Security and Observability.
Based on the talk I had created after creating the CI/CD Pipeline Efficiency docs after a CI monitoring webinar ...
- Continuous LifeCycle 2021: Efficient DevSecOps Pipelines in a Cloud Native World
- AWS UG Nuremberg: Efficient DevSecOps Pipelines
- cdCon: Dev loves CI/CD: Efficient Sec and Ops Pipelines
- Tweakers Dev Summit: Devs love CI/CD: Efficient DevSecOps Pipelines
- 2. German Connect Day: Efficient Sec and Ops Pipelines
- IT-Tage 365: Devs ❤️ CI/CD: Efficient Sec and Ops Pipelines
- Open Source Automation Days: Developers love CI/CD: The Sec and Ops Sequel
- Webinar: Identify, analyze, action! Deep monitoring with CI
... I have now built a 10+ hours workshop for Open Source Automation Days 2021 with old and new best practices for everyone to learn and benefit. I've added new topics focussing on auto-scaling and cloud native deployments, whilst enriching the configuration examples with many hidden gems.
Yesterday, a discussion about a great blog post "Let's make faster GitLab CI/CD pipelines" on Hacker News inspired me to share the workshop and now blog about it. You can find all workshops in the GitLab Developer Evangelism handbook.
You can learn async at your own pace, with the slides providing the exercises and the solutions in a workshop project. Note that there is one exercise which needs preparations for auto-scaling which is described in the slides.
The following topics will be practiced:
- Introduction: CI/CD meets Dev, Sec and Ops
- CI/CD: Terminology and first steps
- Pipeline Editor
- Analyse & Identify
- Learn using the GitLab CI Pipeline Exporter to monitor the exercise project throughout the workshop.
- Efficiency actions
- Config Efficiency: CI/CD Variables in variables, job templates (YAML anchors, extends), includes (local, remote), rules and conditions (if, dynamic variables, conditional includes),
!referencetags (script, rules), maintain own CI/CD templates (include templates, override config values), parent-child pipelines, multi project pipelines, better error messages to fix failures fast - Resource Use Efficiency: Identification, max pipeline duration analysis, fail fast with stages grouping, fail fast with async needs, analyse blocking stages pipeline (solution with needs), matrix builds for parallel execution (pratice: combine matrix and
extends, combine matrix and!reference),extendsmerge strategies (with and without!reference) - CI/CD Infrastructure Efficiency: Optimization ideas, custom build images, optimize builds with C++ as example, GitLab runner resource analysis (sharing, tags, external dependencies, Kubernetes), local runner exercise, resource groups, storage usage analysis, caching (Python dependency exercise, including
when:alwayson failed jobs) - Auto-scaling: Overview, AWS auto-scaling with GitLab Runner with Terraform, insights into Spot Runners on AWS Graviton
- Config Efficiency: CI/CD Variables in variables, job templates (YAML anchors, extends), includes (local, remote), rules and conditions (if, dynamic variables, conditional includes),
- Group discussion
- Deployment Strategies: IaC, GitOps, Terraform, Kubernetes, registries
- Security: Secrets in CI/CD variables, Hashicorp Vault, secrets scanning, vulnerability scanning
- Observability: CI/CD Runner monitoring, SLOs, quality gates, CI/CD Tracing
- More efficiency ideas: Auto DevOps, Fast vs Resources, Conclusion and tips
Take your time with learning, the workshop has a lot of topics inside. Let me know how it goes, and join https://forum.gitlab.com/ for questions and feedback (please tag @dnsmichi ). 🤗
Everyone Can Contribute - learned a new gem from this workshop? Blog about it and share on social tagging @dnsmichi @gitlab! 💡
Blog originally published on https://dnsmichi.at/2021/12/10/efficient-devsecops-pipelines-in-a-cloud-native-world-free-workshop/
Top comments (1)
Hey there!
While we appreciate you sharing posts here, please see DEV's Content Policy available in our Terms and be careful not to share content that is too promotional. The terms state:
Going forward, be careful to avoid sharing articles that are too promotional in nature. While it's generally okay to promote your product, business, personal brand, etc., just make sure that the posts you share have value beyond promotion.
If promotion is your primary goal, then consider signing up for Pro Tools which gives you access to:
Hope you understand our reason for limiting promotional content and that you continue to enjoy DEV!