This article highlights ten open source tools that have gained significant attention amongst infrastructure engineers and are considered essential for professionals in Platform Engineering/DevOps/Site Reliability engineering.
These tools cover a wide range of functionalities, including Infrastructure as Code management, secret management, distributed filesystems, internal developer portals, continuous integration and deployment (CI/CD), and self-hosted Git services.
Each of these open-source projects, from Digger's Infrastructure as Code platform to Gitea's self-hosted Git service, represents a key component in the modern DevOps toolkit, helping engineers to build, deploy, and maintain scalable and efficient software systems.
The tools are:
- Digger - an Open Source Infrastructure as Code management platform.[Infrastrucutre as code automation]
- Git Secret - A bash-tool to store your private data inside a git repository. [Secret Management]
- Infisical - Open source end-to-end encrypted secrets sync for teams and infrastructure. [Secret Management]
- Lade - Automatically load secrets from your preferred vault as environment variables. [Secret Management]
- Ceph - Highly scalable object, block and file-based storage under one whole system. [Distributed Filesystems]
- Backstage - An open platform for building developer portals. [Internal Developer Portal]
- Kraken CI - Modern CI/CD, open-source, on-premise system that is highly scalable and focused on testing. [CI/CD]
- Buildbot - automate all aspects of the software development cycle. [CI framework]
- Gogs - A self-hosted Git service. [Git]
- Gitea - Another self-hosted Git service. [Git]
Now lets dive into each tool one by one:
Digger
Digger is an IaC management tool for Terraform and OpenTofu, addressing the complexities often encountered with specialized IaC CI systems like Terraform Cloud and Atlantis.
Its unique approach integrates Terraform/OpenTofu directly into your existing CI infrastructure, leveraging its asynchronous jobs, compute, orchestration, and logging capabilities.
This integration not only enhances security by keeping cloud access secrets within your CI environment but also proves cost-effective by eliminating the need for extra compute resources. Digger's feature set includes Terraform plan and apply within pull request comments, private runners utilizing existing CI compute environments, Open Policy Agent (OPA) support for robust access control, and PR-level locks to prevent race conditions. Additionally, it supports advanced functionalities like Terragrunt, multiple Terraform versions, and drift detection, making it an all-encompassing solution for managing Terraform/OpenTofu deployments efficiently and securely.
Check out self hosting documentation
Git Secret
Git Secret is an essential bash tool for developers and DevOps professionals, offering a robust solution for secret management within a Git repository. This open-source tool effectively encrypts sensitive files and data, ensuring that confidential information like passwords, keys, and credentials are securely stored in the repository.
By encrypting files with the public keys of allowed users, Git Secret ensures that only authorized personnel can access and decrypt these secrets. This method not only enhances security but also simplifies the process of sharing sensitive data among team members. It's particularly valuable in collaborative environments, where managing access to sensitive information is crucial for maintaining security and compliance. Git Secret stands out as a practical, secure, and efficient way to handle private data in code repositories.
Infisical
Infisical is an open source secret management platform tailored for teams to centralize crucial data such as API keys, database credentials, and configurations. Aimed at making secret management accessible to everyone, not just security experts, it redesigns the entire developer experience. The platform offers a user-friendly dashboard for managing secrets across various projects and environments, client SDKs for on-demand secret retrieval, and a CLI tool for integrating secrets into any framework during local development.
Infisical includes native integrations with platforms like GitHub, Vercel, and Netlify, and features such as automatic Kubernetes deployment secret reloads, self-hosting options on different infrastructures, secret versioning, Point-in-Time Recovery, comprehensive audit logs, Role-based Access Controls, simplified on-premise deployments to AWS and Digital Ocean, along with secret scanning and leak prevention capabilities.
Lade
Lade is a practical tool designed to enhance secret management by automatically loading secrets from a user's chosen vault into environment variables or files. This functionality is key in minimizing the exposure of sensitive information, as it restricts access to secrets only for the duration of a specific command's execution. By ensuring that secrets are only available when absolutely necessary, Lade significantly reduces the risk of unauthorized access or leaks. This approach is particularly beneficial in environments where security and data privacy are paramount. Lade is part of the Metatype ecosystem. Consider checking out how this component integrates with the whole ecosystem and browse the documentation to see more examples.
Ceph
Ceph stands out in storage technology, offering a scalable and reliable solution where traditional systems fall short. It supports object, block, and file storage in one system, adaptable for various environments including on-premises, cloud, or container-native setups. Key benefits include scalability, enabled by the CRUSH algorithm, allowing for expansion without typical downtime. This makes Ceph suitable for businesses and institutions needing to grow their storage capacity rapidly.
Ceph is also notable for its reliability. It is self-managing and self-healing, with Monitor and Manager daemons enhancing data availability. The CRUSH algorithm reduces failure risks, ensuring a robust storage solution.Performance-wise, Ceph's customizable deployment suits diverse needs without compromising efficiency. As a software-defined system, it performs well regardless of the infrastructure, addressing the limitations of traditional storage systems.
Backstage
Backstage is an innovative open platform designed for creating internal developer portals, streamlining the developer experience within organizations. As a centralized hub, it allows teams to manage software components, monitor services, and access tools and documentation from a single interface.
This enhances collaboration and increases efficiency by reducing the complexity often associated with accessing various development tools and resources. By providing a unified, customizable environment, Backstage fosters a more organized and coherent workflow. Its open-source nature invites contributions and adaptations to suit specific organizational needs, making it an invaluable tool for companies looking to optimize their internal software development processes.
Kraken CI
Kraken CI is a modern CI/CD system that operates on the Continuous Integration philosophy, focusing on pre-commit and post-commit phases in software development. In the pre-commit phase, developers and testers prepare code changes, aiming to minimize the risk of breaking production code. Kraken CI facilitates this by providing a validation environment that simplifies testing, making it easier to produce quality code. It reduces the likelihood of large, risky changes and helps manage code integration more effectively.
In the post-commit phase, the emphasis is on maintaining the stability of production code. Kraken CI's effective post-commit validation delivers clear, unambiguous information about the production code, reducing the time to feedback and allowing for quick response to any issues. This results in greater stability and release-readiness of the production code. By improving both pre-commit and post-commit phases, Kraken CI fosters a culture shift in software development. It moves away from a gate-focused approach, where each stage of development is a barrier, to a more fluid process where small changes are made frequently. This shift reduces the impact of breaks and improves the overall quality and efficiency of the engineering process, allowing teams to focus on innovation and delivering unique value to customers.
Buildbot
Buildbot is a versatile CI framework designed to automate all aspects of the software development cycle, enhancing efficiency and reliability. As an open-source platform, it is highly customizable, allowing teams to tailor the automation process to their specific needs. Buildbot excels in integrating various stages of development, from code integration, testing, to deployment, ensuring a seamless and coherent workflow. This framework supports multiple development environments, making it adaptable to different technologies and project requirements. Its ability to streamline complex processes and foster continuous integration and deployment makes Buildbot a valuable tool for teams seeking to optimize their software development lifecycle.
Gogs
The Gogs project is dedicated to creating a simple, stable, and extensible self-hosted Git service, emphasizing ease of setup. Utilizing Go, Gogs offers an independent binary distribution compatible across multiple platforms, including Linux, macOS, Windows, and ARM systems. The platform features a comprehensive user dashboard, profile, and activity timeline, and supports repository access through SSH, HTTP, and HTTPS.
It includes robust management tools for users, organizations, and repositories, alongside webhooks and Git hooks. Gogs facilitates repository issues, pull requests, wiki, and collaboration features. It also offers migration and mirroring of repositories, a web editor for repository files, Jupyter Notebook and PDF rendering, and supports various authentication methods including SMTP, LDAP, and GitHub integration. Additionally, Gogs is customizable, supports a range of databases like PostgreSQL and MySQL, and is localized in over 31 languages, making it a versatile and user-friendly solution for Git hosting.
Gitea
Gitea is a versatile tool for creating and managing git-based repositories, streamlining Code Review to enhance code quality for users and businesses. It integrates a CI/CD system, Gitea Actions, compatible with GitHub Actions, allowing users to create workflows in YAML or use existing plugins. Gitea's project management features include issue tasks, labeling, and kanban boards for efficient management of requirements, features, and bugs. These tools integrate with branches, tags, milestones, assignments, time tracking, and dependencies to plan and track development progress. Furthermore, Gitea supports over 20 package management types, such as Cargo, Composer, NPM, and PyPI, catering to a wide range of public or private package management needs. This comprehensive suite of features makes Gitea a powerful platform for managing development projects and packages.
Top comments (12)
Would you mind elaborating on the reasons, or say why you are an authority in this field ?
Thank you Mike for the details.
Your response urged me to a little bit of research. With enough security on passphrase and/or encryption size, the "little bit of extra money and time" is still multiple years at best. And I do not have NSA level secrets to store.
I will take care of those parameters and consider myself fine with the risk !
Your dedication to security practices is wonderful !
On my side, I don't take myself and my projects too seriously, the world is in of the verge of apocalyptic events way more deep than my problems.
Thanks to you, I learned some stuff about the encryption key size, the passphrase length, and if someday someone is able to crack 256-bits encryption keys under a few weeks, he won't go to first to my clients' repos, we are too small fishes to catch 😊
Digger sounds interesting! On which type of projects should it be use it? E.g., I'm developing my own side project SaaS, reflectdaily.app/, and currently deploy to Fly via Wasp. At which point could I benefit from Digger?
Thanks for sharing !
git-secretis what I was searching for 🎉I have implemented this myself for now.
Ooh, good list!
This is a decent list. Ceph and Minio are legit S3 alternatives.
Great article!
Infisical 🚀
Some comments may only be visible to logged-in visitors. Sign in to view all comments.