Terraform drift detection and remediation - a primer

We know Terraform/OpenTofu is the most widely used IaC tool for automating and managing cloud infrastructure.

However, with the power of managing infrastructure as code, comes the challenge of managing 'drift' - the divergence between the intended state of infrastructure as defined in Terraform configurations and its actual state in the cloud.

Drift can manifest for various reasons, from manual interventions to overlapping tool functionalities, and poses significant risks including security vulnerabilities, compliance issues, and financial overheads. This article delves deep into Terraform drift detection and remediation, exploring the underlying causes, the potential impacts, and the strategies to detect and rectify this aspect of cloud infrastructure management.

Common Causes of Drift

The occurrence of drift in Terraform-managed infrastructure can be attributed to several factors. A primary cause is manual changes made directly in the infrastructure through interfaces like the AWS Console, which are not recorded in the Terraform state file. This discrepancy leads to a state of drift. Another significant factor contributing to drift is the use of multiple automation tools with overlapping capabilities.

For example, using Terraform alongside Ansible, which also possesses infrastructure provisioning capabilities, might result in inconsistencies and drift if not managed properly.

Additionally, user-defined scripts, such as bash or shell scripts in cloud platforms, can also lead to drift by modifying network configurations or other resources.

Emergency changes or hotfixes applied directly through interfaces like the AWS Console can bypass Terraform's regular processes, further leading to drift.

A lack of adequate training among non infra savvy developers in the team (yes - "shifting left" gone wrong) regarding Terraform's use can result in them making direct updates through cloud service consoles, inadvertently introducing drift.

Implications of Drift

Drift in Terraform can have far-reaching implications. It can expose security vulnerabilities, leading to potential data breaches and system compromises. In terms of compliance, drift can lead to violations, especially when it results in unauthorized access or exposure of sensitive data. Operationally, drift can introduce challenges, increasing system downtime and impacting performance. Untracked changes can also lead to increased operational costs, particularly if they involve the use of higher-spec resources that were not part of the original plan.

Detecting and Managing Drift

Detecting and managing drift in Terraform is a multifaceted process. The use of Terraform commands such as terraform refresh and terraform plan plays a critical role in identifying drifts. Additionally, periodic monitoring of the infrastructure using these commands can aid in early detection and prevention of larger issues. Tools like Digger and Terraform Cloud offer dedicated drift detection and remediation mechanisms. These tools provide continuous monitoring and notifications, enabling teams to stay informed about the state of their infrastructure.

Drift Detection in Digger - Slack Alerts

Drift Remediation Approaches

There are two primary approaches to remediate drift in Terraform. The first involves reconciling the changes, which means restoring the infrastructure to the state defined in the Terraform code. This typically involves reapplying the configuration. The second approach is aligning the Terraform code with the current state of the infrastructure. This is often necessary when changes made outside of Terraform are deemed necessary and should be maintained. This involves updating the Terraform code to reflect the real-time state of the infrastructure.

In conclusion, managing infrastructure drift in Terraform environments requires diligent monitoring, effective use of tools, and clear policies and training for team members. This comprehensive approach is vital to ensure that the infrastructure remains secure, compliant, and aligned with the organizational goals and requirements.

Digger

Thank you for reading until the end. Before you go, just wanted to share the following:

• We're building an Open Source Tool that helps you orchestrate Terraform within CI/CD systems such as GitHub Actions while providing RBAC via OPA, Drift Detection and Concurrency with a self hostable orchestrator backend. Would love your feedback!

• Star us on GitHub | Check out Docs | Blog | Slack