DEV Community

Cover image for Map Azure Devops runtime variables to Terraform input variables
Arindam Mitra
Arindam Mitra

Posted on • Updated on

Map Azure Devops runtime variables to Terraform input variables

#azure #devops #terraform #iac

Greetings my fellow Technology Advocates and Specialists.

In this Session, I will demonstrate -

  1. How to Map Azure DevOps Runtime Variables to Terraform Input Variables.
  2. If at all we need to put the values in variables.tf or in tfvars.

I had the Privilege to talk on this topic in ONE Azure Communities:-

NAME OF THE AZURE COMMUNITY TYPE OF SPEAKER SESSION
Boston Azure User Group Virtual
EVENT ANNOUNCEMENT:-
Image description
LIVE RECORDED SESSIONS:-
LIVE DEMO was Recorded as part of my Presentation in BOSTON AZURE USER GROUP Forum/Platform
Duration of My Demo = 41 Mins 07 Secs
REQUIREMENTS:-
  1. Azure Subscription.
  2. Azure DevOps Organisation and Project.
  3. Service Principal with Delegated Graph API Rights and Required RBAC (Typically Contributor on Subscription or Resource Group)
  4. Azure Resource Manager Service Connection in Azure DevOps.
  5. Microsoft DevLabs Terraform Extension Installed in Azure DevOps.
CODE REPOSITORY:-

GitHub logo arindam0310018 / 23-May-2022-DevOps__Runtime-Variables-To-Terraform-Input-Variables

MAP AZURE DEVOPS RUNTIME VARIABLES TO TERRAFORM INPUT VARIABLES

MAP AZURE DEVOPS RUNTIME VARIABLES TO TERRAFORM INPUT VARIABLES

Greetings my fellow Technology Advocates and Specialists.

In this Session, I will demonstrate -

  1. How to Map Azure DevOps Runtime Variables to Terraform Input Variables.
  2. If at all we need to put the values in variables.tf or in tfvars.

I had the Privilege to talk on this topic in ONE Azure Communities:-

NAME OF THE AZURE COMMUNITY TYPE OF SPEAKER SESSION
Virtual Boston Azure Virtual
EVENT ANNOUNCEMENT:-
Image description
LIVE RECORDED SESSIONS:-
LIVE DEMO was Recorded as part of my Presentation in BOSTON AZURE USER GROUP Forum/Platform
Duration of My Demo = 41 Mins 07 Secs
IMAGE ALT TEXT HERE
REQUIREMENTS:-
  1. Azure Subscription.
  2. Azure DevOps Organisation and Project.
  3. Service Principal with Delegated Graph API Rights and Required RBAC (Typically Contributor on Subscription or Resource Group)
  4. Azure Resource Manager Service Connection in Azure DevOps.
  5. Microsoft DevLabs Terraform Extension Installed in Azure DevOps.
HOW DOES MY CODE PLACEHOLDER
…
View on GitHub
HOW DOES MY CODE PLACEHOLDER LOOKS LIKE:-
Image description
OBJECTIVE:-
Deploy a Resource Group and User Assigned Managed Identity from the values provided by user in the DevOps Runtime Variables Parameters and not providing it again in Terraform variables.tf or tfvars
PIPELINE CODE SNIPPET:-
AZURE DEVOPS YAML PIPELINE (azure-pipelines-usr-mid-v1.0.yml):- 
trigger:
  none

######################
#DECLARE PARAMETERS:-
######################
parameters:
- name: SubscriptionID
  displayName: Subscription ID Details Follow Below:-
  default: 210e66cb-55cf-424e-8daa-6cad804ab604
  values:
  -  210e66cb-55cf-424e-8daa-6cad804ab604

- name: ServiceConnection
  displayName: Service Connection Name Follows Below:-
  default: amcloud-cicd-service-connection
  values:
  -  amcloud-cicd-service-connection

- name: RGNAME
  displayName: Please Provide the Resource Group Name:-
  type: object
  default: <Please provide the required Name>

- name: USRMIDNAME
  displayName: Please Provide the User Assigned Managed Identity Name:-
  type: object
  default: <Please provide the required Name>

######################
#DECLARE VARIABLES:-
######################
variables:
  TF_VAR_RG_NAME: ${{ parameters.RGNAME }}
  TF_VAR_USR_MID_NAME: ${{ parameters.USRMIDNAME }}
  ResourceGroup: tfpipeline-rg
  StorageAccount: tfpipelinesa
  Container: terraform
  TfstateFile: UMID/usrmid.tfstate
  BuildAgent: windows-latest
  WorkingDir: $(System.DefaultWorkingDirectory)/Usr-MID
  Target: $(build.artifactstagingdirectory)/AMTF
  Environment: NonProd
  Artifact: AM

#########################
# Declare Build Agents:-
#########################
pool:
  vmImage: $(BuildAgent)

###################
# Declare Stages:-
###################
stages:

- stage: PLAN
  jobs:
  - job: PLAN
    displayName: PLAN
    steps:
# Install Terraform Installer in the Build Agent:-
    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
      displayName: INSTALL TERRAFORM VERSION - LATEST
      inputs:
        terraformVersion: 'latest'
# Terraform Init:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM INIT
      inputs:
        provider: 'azurerm'
        command: 'init'
        workingDirectory: '$(workingDir)' # Az DevOps can find the required Terraform code
        backendServiceArm: '${{ parameters.ServiceConnection }}' 
        backendAzureRmResourceGroupName: '$(ResourceGroup)' 
        backendAzureRmStorageAccountName: '$(StorageAccount)'
        backendAzureRmContainerName: '$(Container)'
        backendAzureRmKey: '$(TfstateFile)'
# Terraform Validate:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM VALIDATE
      inputs:
        provider: 'azurerm'
        command: 'validate'
        workingDirectory: '$(workingDir)'
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'
# Terraform Plan:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM PLAN
      inputs:
        provider: 'azurerm'
        command: 'plan'
        workingDirectory: '$(workingDir)'
        commandOptions: "--var-file=usrmid.tfvars --out=tfplan"
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'

# Copy Files to Artifacts Staging Directory:-
    - task: CopyFiles@2
      displayName: COPY FILES ARTIFACTS STAGING DIRECTORY
      inputs:
        SourceFolder: '$(workingDir)'
        Contents: |
          **/*.tf
          **/*.tfvars
          **/*tfplan*
        TargetFolder: '$(Target)'
# Publish Artifacts:-
    - task: PublishBuildArtifacts@1
      displayName: PUBLISH ARTIFACTS
      inputs:
        targetPath: '$(Target)'
        artifactName: '$(Artifact)' 

- stage: DEPLOY
  condition: succeeded()
  dependsOn: PLAN
  jobs:
  - deployment: 
    displayName: Deploy
    environment: $(Environment)
    pool:
      vmImage: '$(BuildAgent)'
    strategy:
      runOnce:
        deploy:
          steps:
# Download Artifacts:-
          - task: DownloadBuildArtifacts@0
            displayName: DOWNLOAD ARTIFACTS
            inputs:
              buildType: 'current'
              downloadType: 'single'
              artifactName: '$(Artifact)'
              downloadPath: '$(System.ArtifactsDirectory)' 
# Install Terraform Installer in the Build Agent:-
          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: INSTALL TERRAFORM VERSION - LATEST
            inputs:
              terraformVersion: 'latest'
# Terraform Init:-
          - task: TerraformTaskV2@2 
            displayName: TERRAFORM INIT
            inputs:
              provider: 'azurerm'
              command: 'init'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF/' # Az DevOps can find the required Terraform code
              backendServiceArm: '${{ parameters.ServiceConnection }}' 
              backendAzureRmResourceGroupName: '$(ResourceGroup)' 
              backendAzureRmStorageAccountName: '$(StorageAccount)'
              backendAzureRmContainerName: '$(Container)'
              backendAzureRmKey: '$(TfstateFile)'
# Terraform Apply:-
          - task: TerraformTaskV2@2
            displayName: TERRAFORM APPLY # The terraform Plan stored earlier is used here to apply only the changes.
            inputs:
              provider: 'azurerm'
              command: 'apply'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF'
              commandOptions: '--var-file=usrmid.tfvars'
              environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'
Enter fullscreen mode Exit fullscreen mode

Now, let me explain each part of YAML Pipeline for better understanding.

PART #1:-
BELOW FOLLOWS PIPELINE RUNTIME VARIABLES CODE SNIPPET:- 
######################
#DECLARE PARAMETERS:-
######################
parameters:
- name: SubscriptionID
  displayName: Subscription ID Details Follow Below:-
  default: 210e66cb-55cf-424e-8daa-6cad804ab604
  values:
  -  210e66cb-55cf-424e-8daa-6cad804ab604

- name: ServiceConnection
  displayName: Service Connection Name Follows Below:-
  default: amcloud-cicd-service-connection
  values:
  -  amcloud-cicd-service-connection

- name: RGNAME
  displayName: Please Provide the Resource Group Name:-
  type: object
  default: <Please provide the required Name>

- name: USRMIDNAME
  displayName: Please Provide the User Assigned Managed Identity Name:-
  type: object
  default: <Please provide the required Name>
Enter fullscreen mode Exit fullscreen mode
THIS IS HOW IT LOOKS WHEN YOU EXECUTE THE PIPELINE FROM AZURE DEVOPS:-
Image description
NOTE:-
Please Provide the Name of the Resource Group
For Example: AMTESTMIDRG
Please Provide the Name of the User Assigned Managed Identity
For Example: AMMID100
PART #2:-
BELOW FOLLOWS PIPELINE VARIABLES CODE SNIPPET:- 
######################
#DECLARE VARIABLES:-
######################
variables:
  TF_VAR_RG_NAME: ${{ parameters.RGNAME }}
  TF_VAR_USR_MID_NAME: ${{ parameters.USRMIDNAME }}
  ResourceGroup: tfpipeline-rg
  StorageAccount: tfpipelinesa
  Container: terraform
  TfstateFile: UMID/usrmid.tfstate
  BuildAgent: windows-latest
  WorkingDir: $(System.DefaultWorkingDirectory)/Usr-MID
  Target: $(build.artifactstagingdirectory)/AMTF
  Environment: NonProd
  Artifact: AM
Enter fullscreen mode Exit fullscreen mode
IMPORTANT TO NOTE:-
User Input Values from DevOps Runtime Parameters are referenced to DevOps Variables.
Notice the the variables TF_VAR_RG_NAME and TF_VAR_USR_MID_NAME.
Azure DevOps Variables gets automatically mapped to Environment Variables in Azure DevOps Build Agent.
Environment Variables which Starts with TF_VAR_ gets automatically mapped to Terraform Input Variables
Refer the link to find more: https://www.terraform.io/cli/config/environment-variables
GENERAL INFORMATION:-
Please feel free to change the values of the variables.
The entire YAML pipeline is build using Parameters and variables. No Values are Hardcoded.
PART #3:-
PIPELINE STAGE DETAILS FOLLOW BELOW:-
  1. This is a Two Stage Pipeline with 4 Runtime Variables - 1) Subscription ID 2) Service Connection Name 3) Resource Group Name and 4) User Assigned Managed Identity Name
  2. The Names of the Stages are - 1) PLAN and 2) DEPLOY
PIPELINE STAGE - PLAN:- 
- stage: PLAN
  jobs:
  - job: PLAN
    displayName: PLAN
    steps:
# Install Terraform Installer in the Build Agent:-
    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
      displayName: INSTALL TERRAFORM VERSION - LATEST
      inputs:
        terraformVersion: 'latest'
# Terraform Init:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM INIT
      inputs:
        provider: 'azurerm'
        command: 'init'
        workingDirectory: '$(workingDir)' # Az DevOps can find the required Terraform code
        backendServiceArm: '${{ parameters.ServiceConnection }}' 
        backendAzureRmResourceGroupName: '$(ResourceGroup)' 
        backendAzureRmStorageAccountName: '$(StorageAccount)'
        backendAzureRmContainerName: '$(Container)'
        backendAzureRmKey: '$(TfstateFile)'
# Terraform Validate:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM VALIDATE
      inputs:
        provider: 'azurerm'
        command: 'validate'
        workingDirectory: '$(workingDir)'
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'
# Terraform Plan:-
    - task: TerraformTaskV2@2
      displayName: TERRAFORM PLAN
      inputs:
        provider: 'azurerm'
        command: 'plan'
        workingDirectory: '$(workingDir)'
        commandOptions: "--var-file=usrmid.tfvars --out=tfplan"
        environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'

# Copy Files to Artifacts Staging Directory:-
    - task: CopyFiles@2
      displayName: COPY FILES ARTIFACTS STAGING DIRECTORY
      inputs:
        SourceFolder: '$(workingDir)'
        Contents: |
          **/*.tf
          **/*.tfvars
          **/*tfplan*
        TargetFolder: '$(Target)'
# Publish Artifacts:-
    - task: PublishBuildArtifacts@1
      displayName: PUBLISH ARTIFACTS
      inputs:
        targetPath: '$(Target)'
        artifactName: '$(Artifact)'
Enter fullscreen mode Exit fullscreen mode
PLAN STAGE PERFORMS BELOW:-
## TASKS
1. Terraform Installer installed in Azure DevOps Build Agent.
2. Terraform Init
3. Terraform Validate
4. Terraform Plan
5. Copy the Terraform files (Most Importantly Terraform Plan Output) to Artifacts Staging Directory.
6. Publish Artifacts
PIPELINE STAGE - DEPLOY:- 
- stage: DEPLOY
  condition: succeeded()
  dependsOn: PLAN
  jobs:
  - deployment: 
    displayName: Deploy
    environment: $(Environment)
    pool:
      vmImage: '$(BuildAgent)'
    strategy:
      runOnce:
        deploy:
          steps:
# Download Artifacts:-
          - task: DownloadBuildArtifacts@0
            displayName: DOWNLOAD ARTIFACTS
            inputs:
              buildType: 'current'
              downloadType: 'single'
              artifactName: '$(Artifact)'
              downloadPath: '$(System.ArtifactsDirectory)' 
# Install Terraform Installer in the Build Agent:-
          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: INSTALL TERRAFORM VERSION - LATEST
            inputs:
              terraformVersion: 'latest'
# Terraform Init:-
          - task: TerraformTaskV2@2 
            displayName: TERRAFORM INIT
            inputs:
              provider: 'azurerm'
              command: 'init'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF/' # Az DevOps can find the required Terraform code
              backendServiceArm: '${{ parameters.ServiceConnection }}' 
              backendAzureRmResourceGroupName: '$(ResourceGroup)' 
              backendAzureRmStorageAccountName: '$(StorageAccount)'
              backendAzureRmContainerName: '$(Container)'
              backendAzureRmKey: '$(TfstateFile)'
# Terraform Apply:-
          - task: TerraformTaskV2@2
            displayName: TERRAFORM APPLY # The terraform Plan stored earlier is used here to apply only the changes.
            inputs:
              provider: 'azurerm'
              command: 'apply'
              workingDirectory: '$(System.ArtifactsDirectory)/$(Artifact)/AMTF'
              commandOptions: '--var-file=usrmid.tfvars'
              environmentServiceNameAzureRM: '${{ parameters.ServiceConnection }}'
Enter fullscreen mode Exit fullscreen mode
DEPLOY STAGE PERFORMS BELOW:-
## TASKS
1. DEPLOY Stage will Execute only if PLAN Stage completed successfully. If not, DEPLOY Stage will get Skipped Automatically.
2. DEPLOY Stage will Execute only after Approval. The Approval is integrated with Environment defined in the Pipeline Variable Section (Environment: NonProd) and applied in DEPLOY Stage Jobs (environment: $(Environment)).
3. Download the Published Artifacts.
4. Terraform Installer installed in Azure DevOps Build Agent.
5. Terraform Init
6. Terraform Apply
DETAILS AND ALL TERRAFORM CODE SNIPPETS FOLLOWS BELOW:-
TERRAFORM (main.tf):- 
terraform {
  required_version = ">= 1.2.0"

   backend "azurerm" {
    resource_group_name  = "tfpipeline-rg"
    storage_account_name = "tfpipelinesa"
    container_name       = "terraform"
    key                  = "UMID/usrmid.tfstate"
  }
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.2"
    }   
  }
}
provider "azurerm" {
  features {}
  skip_provider_registration = true
}
Enter fullscreen mode Exit fullscreen mode
TERRAFORM (usrmid.tf):- 
## Azure Resource Group:-
resource "azurerm_resource_group" "rg" {
  name     = var.RG_NAME
  location = var.rg-location
}

## Azure User Assigned Managed Identities:-
resource "azurerm_user_assigned_identity" "az-usr-mid" {

  name                = var.USR_MID_NAME
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location

  depends_on          = [azurerm_resource_group.rg]
  }
Enter fullscreen mode Exit fullscreen mode
TERRAFORM (variables.tf):- 
variable "RG_NAME" {
  type        = string
  description = "Name of the Resource Group"
}

variable "rg-location" {
  type        = string
  description = "Resource Group Location"
}

variable "USR_MID_NAME" {
  type        = string
  description = "Name of the User Assigned Managed Identity"
}
Enter fullscreen mode Exit fullscreen mode
IMPORTANT TO NOTE:-
The Variable name of the Resource Group and User Assigned Managed Identities in usrmid.tf and variables.tf are in upper case.
This is because Azure DevOps Pipeline variables which automatically references to Build Agent Environment Variables gets converted to uppercase
If the variables are not defined as above, the Pipeline waits for Resource Group and User Assigned Managed Identity Name as Input.
Image description
The Pipeline is then cancelled manually
Image description
TERRAFORM (usrmid.tfvars):- 
rg-location     = "West Europe"
Enter fullscreen mode Exit fullscreen mode
IMPORTANT TO NOTE:-
There is No Resource Group and User Assigned Managed Identity Name Value provided in tfvars or in variables.tf
ITS TIME TO TEST:-
DESIRED RESULT: Stages - PLAN and DEPLOY should Complete Successfully. Resource Group and User Assigned Managed Identity Resources should get deployed. Remote State file gets created.
PIPELINE RUNTIME PARAMETERS WITH POPULATED VALUES:-
Image description
PIPELINE STAGE PLAN EXECUTED SUCCESSFULLY:-
Image description
PIPELINE STAGE DEPLOY WAITING APPROVAL:-
Image description
Image description
PIPELINE STAGE DEPLOY EXECUTED SUCCESSFULLY:-
Image description
PIPELINE OVERALL EXECUTION STATUS:-
Image description
Image description
VALIDATE RESOURCES DEPLOYED IN PORTAL:-
Image description
VALIDATE REMOTE TERRAFORM STATE FILE:-
Image description

Oldest comments (0)

Read next

env0team profile image

Using Terraform, Kubernetes, and Helm: The Power Trio

env0 Team -

matthijsschuurman profile image

Why is coding so ridiculously overcomplicated?

Matthijs -

acwwat profile image

How To Implement AWS SSB Controls in Terraform - Part 1

Anthony Wat -

mjrowles profile image

A better way to do environment variables

Matt -