Terraform Provider for SAP BTP - Remote State and Drift Detection

Introduction

In a recent DSAG (German-speaking SAP User Group) event ("DSAG Betriebstage" - can be translated as Ops Days) I had the opportunity to give a talk about Infrastructure as Code (IaC) and how it can help you with governance topics on the SAP Business Technology Platform (BTP).

One important topic in this context was of course the automated detection of configuration drifts. The scenario basically was that an SAP BTP subaccount was (perfectly) set up using Terraform following all company-specific guidelines. However, a user with the necessary permissions changed the configuration manually in the SAP BTP cockpit. This led to a deviation between the automated configuration and the actual state of the subaccount. This is what we call a configuration drift. How can we find out about this drift?

While I showed how this can be detected via Terraform, we did not go into the technical details of the process. To make this theoretical description more tangible we will take a closer look on how we can achieve this from a technical perspective.

Prerequisites

To make the following examples work I leveraged the following tools and services:

• A GitHub repository for storing the Terraform configuration
• An Azure Blob Storage for centrally storing the Terraform state
• GitHub Actions to automate the Terraform workflow especially the drift detection

You can of course also use different backends like AWS S3. You find a complete list of supported backends in the Terraform documentation. The same is true for the CI/CD pipeline. You can use e.g., GitLab CI/CD or any other CI/CD tool you prefer to enable the automated flow.

The GitHub Repository

I created a GitHub repository that you find at https://github.com/btp-automation-scenarios/btp-terraform-drift. This repository contains the Terraform configuration for an SAP BTP subaccount.

The configuration is very basic and only creates a subaccount with a few entitlements to show the drift detection flow. The infrastructure configuration is stored in the infra directory of the repository.

The sensitive information for the setup like username and password is stored in GitHub Secrets.

Setup for Remote State

The detection of a configuration drift relies on the storage of the Terraform state. This state file contains the configuration of the infrastructure and is used by Terraform to determine what must be changed in the infrastructure.

In a real-life setup this state is stored centrally to make it available to the team of BTP administrators. This can be an Azure Blob Storage, AWS S3 or any other supported backend. One important point here is that the state needs to be encrypted as it might contains sensitive information. In this blog post we will use an Azure Blob Storage.

The first step is to create such a storage on Microsoft Azure. The Microsoft documentation provides a very good guide on how to setup such a storage account for Terraform state. You find the documentation here.

For the sake of simplicity, we will shortly walk through the steps using the Azure CLI:

1. Create a resource group on Azure:

   az group create --name rg_terraform_state_sapbtp --location westeurope 

1. Create a storage account in the resource group:

   az storage account create --resource-group rg_terraform_state_sapbtp --name sasapbtptfstate --sku Standard_LRS --encryption-services blob

1. Create a blob container in the storage account:

   az storage container create --name tfstate --account-name sasapbtptfstate

This results in the following setup for the storage account:

and the blob container (that already contains the state file):

You can also do the setup via the Azure Portal or Terraform.

Be aware that:

• Azure storage comes with automatic encryption. You can also use customer managed keys for that.
• Azure state storage blobs are automatically locked before any operation that writes to the storage account. This prevents concurrent writes to the state file. Details are described in the documentation.

Next, we must make Terraform aware of the fact that should store the state in the Azure Blob Storage. This is done by the following configuration to the provider.tf file:

terraform {
  required_providers {
    btp = {
      source  = "sap/btp"
      version = "~>1.1.0"
    }
  }
  backend "azurerm" {
    resource_group_name  = "rg_terraform_state_sapbtp"
    storage_account_name = "sasapbtptfstate"
    container_name       = "tfstate"
    key                  = "dev.terraform.tfstate"
  }
}

The relevant block is the backend block. We use a predefined backend provided by Terraform for Azure. The resource_group_name, storage_account_name and container_name are the values we used to define the Azure Blob Storage. The key is the name of the state file.

Next, we define the workflows to create and destroy the subaccount.

GitHub Actions for the Setup

To execute the creation and the deletion via GitHub Actions I added two workflows to the repository:

• .github/workflows/setup-subaccount.yml: this workflow executed the setup of the subaccount in the SAP BTP based on some input variables. The configuration is given by:

name: Basis Subaccount via Terraform

on:
  workflow_dispatch:
    inputs:
      PROJECT_NAME:
        description: "Name of the project"
        required: true
        default: "sample-proj-drift"
      REGION:
        description: "Region for the sub account"
        required: true
        default: "eu10"
      COST_CENTER:
        description: "Cost center for the project"
        required: true
        default: "1234567890"
      STAGE:
        description: "Stage for the project"
        required: true
        default: "DEV"
      ORGANIZATION:
        description: "Organization for the project"
        required: true
        default: "B2B"

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Setup
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform Apply 
      id: terraform_apply
      shell: bash
      run: |
        export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} apply -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -auto-approve -no-color

• .github/workflows/destroy-subaccount.yml: this workflow destroys the subaccount in the SAP BTP. The configuration is given by:

name: Destroy Subaccount via Terraform

on:
  workflow_dispatch:

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Deletion
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform Apply 
      id: terraform_apply
      shell: bash
      run: |
        export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} destroy -var globalaccount=${{ secrets.GLOBALACCOUNT }} -auto-approve -no-color

There is one important thing to note here. The Terraform CLI needs to authenticate not only to SAP BTP but also to Azure to access the Blob storage. I used the storage access key for that. This key is stored in the GitHub Secrets along with the other sensitive information. To make the access key available to Terraform we must export it in the Terraform steps as ARM_ACCESS_KEY.

With these GitHub Actions we can setup and destroy the subaccount in the SAP BTP including a central storage of the Terraform state in the Azure Blob Storage. The next step is to automate the detection of a configuration drift.

Drift Detection

The detection of a configuration drift is done by comparing the actual state of the subaccount in the SAP BTP with the state defined in the Terraform configuration. This sounds complicated, but indeed it is quite easy.

The only thing we need to do is to execute a terraform plan inside of a GitHub Action. If the stored state matches the configuration on SAP BTP the planning will basically state "nothing to do". In case there is a deviation the planning will come up with changes that it would like to apply to bring the state and the configuration on SAP BTP back in sync. We use this to find out if any unwanted changes have been made.

To make things easier for the handling in a CI/CD flow we provide an additional parameter to the terraform plan command namely the -detailed-exitcode. This parameter tells the CLI to provide more granular information about what the resulting plan via the exit code:

• 0 = Succeeded with empty diff, so no drift detected
• 1 = Error - this should not happen, but at least good to know that things went completely of the rails
• 2 = Succeeded with non-empty diff, so changes are present, and a drift is detected.

With this we create a new workflow for the drift detection:

name: Check for Subaccount Drift via Terraform

on:
  workflow_dispatch:
    inputs:
      PROJECT_NAME:
        description: "Name of the project"
        required: true
        default: "sample-proj-drift"
      REGION:
        description: "Region for the sub account"
        required: true
        default: "eu10"
      COST_CENTER:
        description: "Cost center for the project"
        required: true
        default: "1234567890"
      STAGE:
        description: "Stage for the project"
        required: true
        default: "DEV"
      ORGANIZATION:
        description: "Organization for the project"
        required: true
        default: "B2B"

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Drift Check
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        export export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform Plan 
      id: terraform_plan
      shell: bash
      continue-on-error: true
      run: |
        export ARM_ACCESS_KEY=${{ secrets.ARM_ACCESS_KEY }}
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} plan -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -no-color -detailed-exitcode

    - name: Create issue
      if: steps.terraform_plan.outcome == 'failure'  
      uses: actions/github-script@v7
      env:
        PROJECT_NAME: ${{ github.event.inputs.PROJECT_NAME }}
        REGION: ${{ github.event.inputs.REGION }}
        STAGE: ${{ github.event.inputs.STAGE }}
        COST_CENTER: ${{ github.event.inputs.COST_CENTER }}
        RUN_ID : ${{ github.run_id }}
      with:
        script: |
          const issueTitle = `Configuration Drift Detected for ${process.env.PROJECT_NAME}`
          const issueBody = `A drift has been detected for ${process.env.PROJECT_NAME} in ${process.env.REGION} region. Stage is ${process.env.STAGE} and cost center is ${process.env.COST_CENTER}. Find more information in the run https://github.com/btp-automation-scenarios/btp-terraform-drift/actions/runs/${process.env.RUN_ID}`

          github.rest.issues.create({
            owner: context.repo.owner,
            repo: context.repo.repo,
            labels: [
              'automated issue', 'drift detected'
            ],
            title: issueTitle,
            body: issueBody
          })

    - name: State deviation - Set run to failed
      if: steps.terraform_plan.outcome == 'failure'
      uses: actions/github-script@v7
      with:
        script: |
            core.setFailed('A configuration drift was detected!')    

The main part of the workflow is the step with the id terraform plan that executes the planning and provides us the exit code that we use to check if a drift is detected. As follow-up actions we:

• Create an issue in the GitHub repository to inform the team about the drift
• Set the run to failed to also make it visible in the GitHub Actions overview

The step for the creation is defined as:

  - name: Create issue
      if: steps.terraform_plan.outcome == 'failure'  
      uses: actions/github-script@v7
      env:
        PROJECT_NAME: ${{ github.event.inputs.PROJECT_NAME }}
        REGION: ${{ github.event.inputs.REGION }}
        STAGE: ${{ github.event.inputs.STAGE }}
        COST_CENTER: ${{ github.event.inputs.COST_CENTER }}
        RUN_ID : ${{ github.run_id }}
      with:
        script: |
          const issueTitle = `Configuration Drift Detected for ${process.env.PROJECT_NAME}`
          const issueBody = `A drift has been detected for ${process.env.PROJECT_NAME} in ${process.env.REGION} region. Stage is ${process.env.STAGE} and cost center is ${process.env.COST_CENTER}. Find more information in the run https://github.com/btp-automation-scenarios/btp-terraform-drift/actions/runs/${process.env.RUN_ID}`

          github.rest.issues.create({
            owner: context.repo.owner,
            repo: context.repo.repo,
            labels: [
              'automated issue', 'drift detected'
            ],
            title: issueTitle,
            body: issueBody
          })

This step is executed only if the plan state failed which we ensure via the if condition if: steps.terraform_plan.outcome == 'failure'.

We use the GitHub REST API to create an issue with the corresponding labels. As an example, we provide some additional information, but you can of course also provide more detailed information about the detected drift.

Make sure that within the settings of the repository (or the organization) the GitHub Actions are allowed to create issues by enabling the "Read and write permissions" for GitHub Actions:

In the current state the workflow would be reported as "successful". I personally prefer that the run is marked as failed in case of a drift. This must be done explicitly by the following step:

    - name: State deviation - Set run to failed
      if: steps.terraform_plan.outcome == 'failure'
      uses: actions/github-script@v7
      with:
        script: |
            core.setFailed('A configuration drift was detected!')    

With this we have an automated drift detection in place that informs the team about any deviations between the actual state of the subaccount in the SAP BTP and the configuration defined in the Terraform configuration.

Once we introduce a drift, we will see the following in the GitHub Actions overview:

As we can see the exit code is as expected a "2" indicating that changes would be applied.

And consequently, an issue is created in the GitHub repository:

Conclusion and Outlook

With only minor configurations the Terraform built-in functionalities of state storage and planning enable us to detect configuration drifts in the SAP BTP to automatically detect deviations of the configuration with respect to the company-specific guidelines. This ensures that configuration is not changed without the knowledge of the administrator team especially to ensure governance and compliance.

Now that we have detected the drift, how to deal with it? This usually cannot be handled automatically but needs to be analyzed in detail. You have basically two options that bring things back in sync:

• Reconcile the state with the configuration in the Terraform configuration. This can be done by executing a terraform apply. This will apply the necessary changes to the infrastructure and bring it back in sync with the state.
• The manual changes on the platform were okay (e.g., due to an emergency fix). In this case the state file must be updated to reflect the actual state. This can be done by executing a terraform apply and a terraform apply with the -refresh-only flag. This will update the state file with the actual state of the subaccount and not do any changes to the infrastructure.

Another aspect worth to discuss is the integration into existing processes especially on the SAP side of the house. We implemented the flow and the "handling" (namely the creation of a GitHub issue) in an isolated fashion. However, we also received the ask to integrate the flow into the SAP solutions that might already deal with such governance tasks like SAP Cloud ALM. based on that we have a feature request open (link) in the repository of the Terraform Provider for SAP BTP. If you would like to share your point of view or vote for it, this is the place to go.

With this, nothing more to say than: Happy Terraforming!