Although GitOps is a relatively new trend, developers have been using Git to automate processes for a long time, and so Gitops is becoming not just popular but a de facto standard very quickly. Case in point - At Permit.io, we use Gitops as a critical feature as part of our product- empowering our customers to manage policy as code across environments and setups. Git's branching mechanism, conflict resolution, native file versioning support, and advanced features make it a fantastic tool for our users while fulfilling the single-source-of-truth promise of Policy-As-Code.
As more organizations adopt GitOps as a DevOps practice, we believe it's an opportune time to share our experience building a GitOps-enabled tool. In this article, we'll discuss our experience developing GitOps as a feature for our users, providing best practices for you to excel in implementing it for your DevOps needs. Whether planning to use an existing GitOps tool or build your own, learning from our homegrown experience is invaluable. We hope you find our experience worthwhile.
Why GitOps?
Over the past few years, two trends have emerged that enable developers to expedite software development: low code and Infrastructure as Code. Low-code tools empower developers to incorporate a wide range of functionalities into their applications with minimal coding. On the other hand, Infrastructure as Code allows developers to declare and manage their infrastructure through coding and automated tools.
Initially, managing infrastructure and applications was relatively straightforward, and developers only needed a few lines of code. However, as the tools and requirements advanced, the amount of code necessary to manage infrastructure and applications increased. Furthermore, the need for a continuous deployment process complicated matters further by creating multiple code states.
As developers began using Git to manage their code manually, they started utilizing it as an operations automation tool. GitOps emerged as a result of this. While various tools can assist in GitOps implementation, we believe that the fundamental concept behind GitOps is a mindset. In the same manner that DevOps automated all operations, GitOps automates operations in a Git manner. This entails using Git's versioning and branching mechanisms to manage infrastructure and applications.
Since the primary principle of GitOps is to use a specific Git state as the sole source of truth for infrastructure and applications, we suggest adopting the mindset of pulling changes to environments instead of the traditional approach of pushing changes to environments. With a firm grasp of GitOps' core principles, let's delve into the best practices we've discovered while implementing GitOps.
GitOps Best Practices
Enforce Automated Tests and Checks
Automated tests and checks are essential in GitOps to ensure code changes are thoroughly tested before deployment. This approach shifts control of code versioning to the environment, eliminating the need for human intervention. However, reliable and effective automated tests and checks are critical to prevent potential disasters. Enforcing a 100% automated test pass rate is recommended to avoid unexpected malfunctions and downtime. It's important to perform automated testing at every stage of the process, including different environments, to avoid manual errors. A continuous integration and deployment pipeline that automates testing on code changes is highly recommended for thorough testing before production deployment.
Use Preview Environments
Using GitOps and automated deployments can come with the risk of downtime in production due to the discovery of bugs. This not only entails the time required to fix the issue but also the disruption to workflows beyond developer control. Preview environments offer a solution to this challenge by providing an environment that mimics production to test code changes before deploying. When making changes that require infrastructure modifications, submitting a pull request will automatically create a preview environment. This environment is identical to production, providing confidence that code will function as expected during deployment. This approach helps to identify and resolve issues or conflicts that arise during automated deployment quickly.
Encourage Frequent Merges
To achieve rapid software development through GitOps, frequent merges from the main branch are crucial. This reduces the risk of merge conflicts and allows for real-time feedback. Frequent merges also establish a feedback loop between developers, making error detection and course correction easier. One way to encourage frequent merges is by implementing local hooks, which can be customized to trigger based on specific actions and provide automated feedback. By making frequent merges a routine, development teams can streamline their GitOps process for faster and more efficient software development.
Deploy Quickly, Deploy Often
To ensure efficient deployment and avoid bottlenecks in our GitOps tool, we prioritized the independence of applications and services, allowing for seamless integration and continuous updates. We implemented a strategy of deploying small changes and updating configurations regularly to simplify the process and achieve real-time deployment. By prioritizing speed and collaboration, we were able to deliver high-quality services quickly and maintain a steady pipeline.
Define and Document a Conflict Strategy
In GitOps, defining a conflict strategy is a crucial best practice, especially as teams and infrastructure scale. A documented conflict resolution process enables efficient and effective conflict management. This process should be tailored to the organization's specific needs and may specify the order of preference between changes or the role of stakeholders. It's important to ensure that all team members understand and follow the documented process. Automated conflict resolution tools can help address conflicts, but their outputs should be reviewed and tested regularly to ensure they meet the organization's requirements.
Use "Pull" Hooks for State Management
When implementing GitOps, it's crucial to have manual hook endpoints for state management to prevent any potential issues. These hooks allow the DevOps team to manually manage the state of the application or infrastructure in case of any problems. The GitOps way of hooks is to use "pull" hook endpoints to maintain consistency and avoid conflicts, as this method lets the team update the state of the application or infrastructure from a specific Git state. This approach ensures that the Git state is the only source of truth, and changes made are consistent with it. Manual hook endpoints for state management should be created in a read-only flavor to minimize the risk of conflicts or mistakes when managing the state of the application or infrastructure. This ultimately ensures that the GitOps process remains consistent and reliable, enabling teams to efficiently deliver high-quality applications and infrastructure.
Define Naming Conventions for Branches
When implementing GitOps, it's crucial to establish clear naming conventions for branches. This practice enables quick identification of branch types and purposes and simplifies management. Consistent naming also reduces the likelihood of errors and inconsistencies among developers. Standardizing branch naming for environments, features, hotfixes, and releases is essential. For example, "env/" can denote environments, and "feat/" can denote features. Following these conventions streamlines development and improves versioning.
Every Developer Works on Separate Branches for Every Feature
Developers should work on separate branches for each feature to maintain a clean codebase and enable independent work without affecting the main branch. A single collaborator should be assigned to each branch to prevent conflicts and ensure isolated feature development. Collaboration branches open only to pull requests and not for commits. This practice is a best practice for GitOps developers to minimize conflicts and ensure feature testing and development are isolated before merging into the main branch. All this enforcement is easy to configure with your source control manager such as GitHub or GitLab.
Link Logical Environments
In GitOps, it's crucial to link logical environments to deployment environments for a seamless code flow from development to production. To achieve this, representing logical environments as branches in the Git repository is ideal. This approach enables the use of Git's branching and versioning capabilities to track changes consistently across environments.
Branches help manage different versions of code, configuration, and infrastructure in multiple environments, such as development, testing, staging, and production. You can create separate branches for each environment, track changes, and easily roll back changes if necessary. This approach also avoids confusion and minimizes errors when working with different environments, providing a clear view of which branch represents each environment.
Separate Repos in Complex Environments
To manage complex environments effectively, it's best to create separate repositories for different types of code, such as Infrastructure as Code (IaC), Policy as Code (PaC), application code, and framework code. This allows for a clear separation of responsibilities and reduces the likelihood of conflicts. Having distinct repositories ensures changes made in one don't negatively affect others, making it easier to manage changes across environments. For GitOps, where code is managed with code, it's crucial to have a clear separation of concerns. While a complete multi-repo setup isn't necessary, having a logical separation of repositories is crucial for effective GitOps workflows.
Use Policy as Code to Control Pipelines
GitOps simplifies infrastructure and application deployments by using a single Git repository as a source of truth. However, as the system's complexity increases, it's important to separate code and pipeline concerns. This is where Policy as Code (PaC) comes in. PaC allows developers to manage policies and permissions in one place, ensuring pipelines follow organizational guidelines. By treating policies as code, developers can use version control practices, like pull requests and code reviews, to maintain transparency and consistency in pipeline operations. This ensures pipeline configurations are correctly and effectively maintained.
package main
default allow = false
# The input is the request object that contains all the necessary information
# about the delivery request
allow {
input.branch == "main"
input.hasTestsPassed
input.hasOneCollaboratorPerBranch
input.hasRunDevSecOpsTools
}
# Define a rule to check if there is only one collaborator per branch
hasOneCollaboratorPerBranch {
# Count the number of collaborators for each branch
count(collaborator) == 1
# For each branch
input.branches[branch] == collaborator
}
# Define a rule to check if all DevSecOps tools have been run
hasRunDevSecOpsTools {
# Check if all the required DevSecOps tools have been run
input.DevSecOpsTools.runAllTools
}
# Define a rule to check if all tests have passed across the whole pipeline
hasTestsPassed {
# Check if all the tests have passed in each stage of the pipeline
input.pipeline.allTestsPassed
}
What Next?
One of the most effective ways to learn new fields is to be involved in open-source projects and communities. OPAL, the GitOps-enabled policy-as-code management tool we built, is open source and available on GitHub. We encourage you to check it out and see how you could contribute.
Contribution to open source is not must be complex feature writing, and you can always start with small things like fixing typos, adding documentation, and more. You can refer to our issues page and look for issues tagged with good first issue tag, which is suitable for starting to contribute to OPAL. You can join our slack community, as we are always happy to help new contributors. Another option to contribute to open-source projects is to star them on GitHub. Star repositories make them more visible and encourage other developers to contribute.
If you have any questions or comments, feel free to reach out to us on our community channels. We will happily help with any GitOps design questions, even if you are not using OPAL.
Top comments (0)