VM provisioning and Config Management in Azure with ARM Templates

ARM Template is a great mechanism to provision quickly infrastructure in Azure. But when it comes to VM provisioning, this is not the end. If most PaaS services can be set up almost entirely by using ARM templates or Terraform. For IaaS it’s another story. Configuration and application delivery are still needed.

So, is ARM Template or Terraform apply is the end of the story for us?

Infrastructure as code gives us some values. It enables repeatable, idempotent and fast deployments. However, these values are only valuable for you but not for your end-users.

Simply, imagine this situation, you have your ARM template ready to deploy, vnet, nsg, vm and all things needed to create the infrastructure. The deployment is correct and operates in 20 minutes. But if the team responsible to configure and install the application in the VM is busy doing something less, firefighting somewhere else or deploy other complex applications, it ends up to a huge delay for developers or office users to consume the infrastructure you have build. Your 20 minutes deployments aren’t visible. Your end-users will not notice. They only see the 10 days delays to have their services.

Because the true value of infra as code is not only to satisfy our goals but also to give our end users a better service. This is the way to be seen as a problem solver and not as part of the problem. If the servers, network and other stuff are deployed within the day, but you need to wait 5 days to have someone to configure the servers and another one to install applications, your wait time will fives days plus one, six. This mean more than a week.

Having 2 teams, one for the infra, infrastructure services and another for applications is common today. But in the age of Cloud computing, where services are deployed "à la carte", it doesn’t make sense anymore (did it made sense a day?).

What solution can we use?

Fortunately, ARM Template allows executing some tasks after provisioning jobs. It's Microsoft.Compute/virtualMachines/extensions. This resource lets you execute a script and other stuff after the first run of a newly created VM.

This is a generic resource. With it, you can install an anti-malware, a monitoring agent or run a script.

It looks like

{
    "name": "Vmname/MyVmScript",
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "apiVersion": "2019-03-01",
    "location": "[parameters('location')]",
    "dependsOn": [
        "[variables('vmName')]"
      ],
    "properties": {
      "publisher": "Microsoft.Compute",
      "type": "CustomScriptExtension",
      "typeHandlerVersion": "1.8",
      "autoUpgradeMinorVersion": true,
      "settings": {

      },
      "protectedSettings": {}
    }
}

Where:

Type, the extension type, like CustomScriptExtension or DSC
typeHandlerVersion, the version of the handler, the tools behind the extension
AutoUpgradeMinorVersion, Boolean, indicate if the ARM should update the handler to a newer version if available
Settings, the extension settings
ProtectedSettings, just like a secure string, data in protectedSettings cannot be retrieved in logs.

Virtual Machine Extension is a child resource. It can’t be deployed without the parent resource, the VM. You cannot configure something in a VM if the VM it’s not online.

As a child resource, a virtualMachines/extensons can be inside the VM resources parameters.

  {
      "apiVersion": "2018-04-01",
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[variables('vmName')]",
      "location": "[parameters('location')]",
      "properties": {
       //****
      },
      "resources": [
        {
          "type": "extensions",
          "name": "MyVmScript",
          "apiVersion": "2019-03-01",
          "location": "[parameters('location')]",
          "dependsOn": [
            "[variables('vmName')]"
          ],
          "properties": {
            "publisher": "Microsoft.Compute",
            "type": "CustomScriptExtension",
            "typeHandlerVersion": "1.8",
            "autoUpgradeMinorVersion": true,
            "settings": {
        // *** 
            }
          }
        }
      ]
    }

It can be built outside the parent VM, but in this case, the name of the extension resource must include the parent resource name, a slash (/) and the name of the extension.

{
    "name": "Vmname/MyVmScript",
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "apiVersion": "2019-03-01",
    "location": "[parameters('location')]",
    "dependsOn": [
        "[variables('vmName')]"
      ],
    "properties": {
      "publisher": "Microsoft.Compute",
      "type": "CustomScriptExtension",
      "typeHandlerVersion": "1.10",
      "autoUpgradeMinorVersion": true,
      "settings": {
          // ****
      },
      "protectedSettings": {}

    }
  }

We can use this extension resource to configure and deploy applications inside a server.

Let's try to create a web server. It's a simple task but it will illustrate how to enable IaaS os configuration with an ARM template.
When thinking about automating the installation of IIS in a Windows server I think about PowerShell.
It’s a simple command

Install-WindowsFeature -name Web-Server

Let’s put it in a script named install-iis.ps1

To be able to run it from an ARM Templates it must be globally available. It can be on GitHub or inside a blob container in a storage account.
First step put the named-install-iis.ps1 script in a blob container named config.

$AzStorageContext = New-AzStorageContext -StorageAccountName myaccountname -StorageAccountKey $SasKey

Set-AzStorageBlobContent -File "install-iis.ps1" -Container "config" -Blob "install-iis.ps1" -Context $AzStorageContext -Force  

Now we can build our server using ARM Template.

    {
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "webserver01/installScriptIIS",
      "apiVersion": "2019-03-01",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', 'webserver01')]"
      ],
      "properties": {
            "publisher": "Microsoft.Compute",
            "type": "CustomScriptExtension",
            "typeHandlerVersion": "1.10",
        "autoUpgradeMinorVersion": true,
        "settings": {
          "fileUris": [
                "[variables('configVmScriptFile')]"
              ],
              "commandToExecute": "[concat('powershell -ExecutionPolicy Unrestricted -File ', variables('scriptName'))]"
        },
        "protectedSettings": {
            "storageAccountName": "[variables('storageAccountName')]",
            "storageAccountKey": "[variables('storageAccountKey')]"
        }
      }
    }

The configVmScriptFile variable contains the URI to the script in the blob container. To reach this URI, the storage account name and the key is needed. To protect these two values, you can pass them by the protectedSettings properties.

        "protectedSettings": {
            "storageAccountName": "[variables('storageAccountName')]",
            "storageAccountKey": "[variables('storageAccountKey')]"
        }

As you see, fileUris is an array. You can send one or more files to the VM. These files don’t need to be PowerShell script. But you will need at least one to execute.

File are downloaded into the VM in C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension<version>\Downloads

Execution is controlled by the commandToExecute parameter. The extension will run in the last location.

You will need to build your command. Just remind you, you are on a new VM with no configuration and with PowerShell it means you need to configure the execution policy.
You can monitor the script execution like any other resource deployment.

If an error occurs in your scripts, the ARM deployment will fail. Be sure to include error control in your scripts. Any exception will end as a failed deployment but only for the extension deployment.

Inside the VM you can check the execution of scripts. Simply go to X:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension{Version}\RuntimeSettings and read the files X.Settings where X represents the execution instances.

You will have the execution setting with the parameter. Protected Parameters are encrypted.

You can also check x:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension{version}\Status
And read the X.status, it's a JSON file with the output and error code from the script execution.

You are not limited to one script extension. You can do more than one customScript extension and more you can chain them by using dependOn.

But there is something important when using a script extension. ARM Templates are, by definition, idempotent (it performs actions only if needed) and declarative (it tells the system how the state should be without any imperatives instruction).
A script, most of the time, isn't idempotent. It's just a set of instructions telling how to build something. The script we used, install-iis.ps1, isn’t idempotent (it will not check if the service is installed or not) or declarative (it’s just an install script). It’s a paradox to run it in the context of ARM deployment.

Can we do better?

Using a script isn’t the only way to install IIS on a Windows server. There is a declarative and idempotent option, Desired State of Configuration or DSC. And fortunately, there is an ARM extension for DSC.

DSC, Desired State of Configuration, is the configuration management platform based on PowerShell. It's idempotent and declarative like in ARM templates. When deploying an IIS service in a windows server, DSC test if the service is already installed or not and install it if needed, in other cases, it will do nothing.

A DSC is based on a configuration script, where you describe the state you want to your server. Configuration file uses DSC resources to describe and enable the configuration you just describe. Finally, configuration files are compiled to MOF documents.

A DSC configuration look like this.

Configuration IISConfig {

    Import-DscResource -ModuleName 'PSDesiredStateConfiguration'

    Node "localhost"
    {
        WindowsFeature WebServer
        {
            Ensure  = 'Present'
            Name    = 'Web-Server'
        }
    }

}

This configuration install IIS on a server.

Just like the script extension configuration files need to be available globally. But with DSC configuration we have more challenges to solve. You may have more than one configuration file and/or more than one configuration in a file. How to tell Azure which configuration you want to run for this deployment.

The DSC extension has a property to indicate which configuration file and which configuration you need to use in this deployment.

The other challenge is about DSC resources. The new VM may not have all the resources needed to enable the configuration. DSC resource is a PowerShell module. You can download them by using PowerShell gallery or private repo. But how can you get these modules inside a newly created VM?

To get these modules, the solution in the DSC extension is to send them in the same way you send the configuration files.

In other terms, we need to build a package, a ZIP file, with the resource modules and the DSC configuration files.
To create this package, you can use this

save-module -Name ResourceModuleName -Path $BuildPath -Repository PSGallery 

copy-item -Path iisconfig.dsc.ps1 -Destination $BuildPath  

We need to create the package as a ZIP file

Compress-Archive -Path "$($BuildPath)\*" -DestinationPath configPackage.zip  

$AzStorageContext = New-AzStorageContext -StorageAccountName myaccountname -StorageAccountKey $SasKey

Set-AzStorageBlobContent -File configPackage.zip  -Container "config" -Blob configPackage.zip   -Context $AzStorageContext -Force  

Unlike the Script extension, the DSC extension needs an SAS key to access the storage. You can not just use the storage account name and the storage key. You need to generate a SAS key and add it to the configuration package file URI.

$AzStorageSASKey= New-AzStorageContainerSASToken -Name config -Context $AzStorageContext -Permission r -StartTime (get-date).AddHours(-1) -ExpiryTime (get-date).AddMonths(1)

we can build the URI

        "configFile": "configPackage.zip",
        "configVmScriptFile": "[concat('https://omcextarmscr01.blob.core.windows.net/','config/', variables('configFile'), variables('SasKey'))]",

Now we can build our extension in the ARM template.

 {
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "webserver01/DSCManageIIS",
      "apiVersion": "2019-03-01",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', 'webserver01')]"
      ],
        "properties": {
            "publisher": "Microsoft.Powershell",
            "type": "DSC",
            "typeHandlerVersion": "2.19",
            "autoUpgradeMinorVersion": true,
            "settings": {
                "ModulesUrl": "[variables('configVmScriptFile')]",
                "ConfigurationFunction": "iisconfig.dsc.ps1\\IISConfig",
                "Properties": {}
            },
            "protectedSettings": {
            }
        }
    }

The configuration is very similar to what we saw on the script extension but instead of having a fileUris setting, we have a ModulesUrl. This setting is a string an not an array, like in the custom script extension. You cannot have more than one, be sure to put all your config files and all the modules inside the package.

An important setting is ConfigurationFunction. This property tells how to manage the package and which file and configuration to use in the VM.
The configuration may need parameters. Parameters can be passed to the configuration by using Properties.

                "Properties": {
                    "testParam": "myTest"
                }

DSC configuration may need a credential to perform some operation. Thinks about an online domain join. You can protect this credential with the protectedSettings property. Data will not show in the log.

There are two steps to get data from protectedSetting to the configuration. First, you need to create an item in protectedSetting.

            "protectedSettings": {
                "Items": {
                    "AdminPassword": "[parameters('demoCredentiel')]"
                }
            }

Second, you can create a reference to the data by using PrivateSettingsRef.

"Properties": {
                    "testParam": "myTest",
                     "Credential": {
                        "Username": "test-dsc\\labadmin",
                        "password": "PrivateSettingsRef:AdminPassword"
                    }

After the deployment and the first start of the VM.

The package is downloaded in C:\Packages\Plugins\Microsoft.Powershell.DSC<Version>\DSCWork on the machine you can check the status of the operation in C:\Packages\Plugins\Microsoft.Powershell.DSC<Version>\

Enabling a configuration during the VM provisioning is one thinks. It's great because it allows us to deliver the VM and its configuration at the same time, but what if we want to change this configuration?

In Azure, you can leverage this difficulty by using Azure Automation State Configuration. This service act as a DSC pull server and handle configuration, modules management, and DSC compilation. As a Pull server, VM can be configured to report if there is any drift within the present configuration and the desired state and if there is a new configuration to implement.

To perform that we need an Azure Automation account. As we work with IIS we will need the xWebAdministration resource module.
The latest version in the PowerShell Gallery is 3.2.0-preview

New-AzAutomationModule -AutomationAccountName MyAutomationAccount -ResourceGroupName MyRg -Name xWebAdministration -ContentLinkUri "https://www.powershellgallery.com/api/v2/package/xWebAdministration/3.2.0-preview0001"

Now we can build our configuration.

Configuration IISConfig {

    Import-DscResource -ModuleName 'PSDesiredStateConfiguration'
    Import-DscResource -ModuleName @{ModuleName = 'xWebAdministration';ModuleVersion = '3.2.0'}

        WindowsFeature WebServer
        {
            Ensure  = 'Present'
            Name    = 'Web-Server'
        }

        xWebSite DefaultSite
        {
            Ensure          = 'Present'
            Name            = 'Default Web Site'
            State           = 'Stopped'
            ServerAutoStart = $false
            PhysicalPath    = 'C:\inetpub\wwwroot'
            DependsOn       = '[WindowsFeature]WebServer'
        }
}

The module name is completed with the module version to be sure to work with the right version in the Automation account repository.

After adding and compiling the configuration we can use it in a template deployment.

{
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "webserverAsDSC/DSCManageIIS",
      "apiVersion": "2019-03-01",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', 'webserverAsDSC')]"
      ],
        "properties": {
            "publisher": "Microsoft.Powershell",
            "type": "DSC",
            "typeHandlerVersion": "2.80",
            "autoUpgradeMinorVersion": true,
            "settings": {
                "Properties": [
                      {
                        "Name": "RegistrationKey",
                        "Value": {
                          "UserName": "PLACEHOLDER_DONOTUSE",
                          "Password": "PrivateSettingsRef:azureAutomationPrivateKey"
                        },
                        "TypeName": "System.Management.Automation.PSCredential"
                      },
                      {
                        "Name": "RegistrationUrl",
                        "Value": "[variables('azureAutomationURI')]",
                        "TypeName": "System.String"
                      },
                      {
                        "Name": "NodeConfigurationName",
                        "Value": "IISConfig.localhost",
                        "TypeName": "System.String"
                      },
                      {
                        "Name": "ConfigurationMode",
                        "Value": "ApplyandAutoCorrect",
                        "TypeName": "System.String"
                      },
                      {
                        "Name": "RebootNodeIfNeeded",
                        "Value": true,
                        "TypeName": "System.Boolean"
                      },
                      {
                        "Name": "ActionAfterReboot",
                        "Value": "ContinueConfiguration",
                        "TypeName": "System.String"
                      }
                    ]
            },
            "protectedSettings": {
               "Items": {
                      "azureAutomationPrivateKey": "[variables('azureAutomationKey')]"
                    }
            }
        }
    }

In the property section, we need to define the same properties we find in Azure DSC GUI when we add a node. The DSC configuration mode (ApplyandMonitor, ApplyOnly, and ApplyandAutoCorrect), Action After Reboot and Reboot if Needed. We need to provide the Azure Automation URL with RegistrationURL. This URI is located in the Key property in the Account Settings of Azure Automation.

We need to authenticate to the Azure Automation web services. We need to provide the Key to allow the VM to join to the service by employing registrationKey property. It contains 3 values. The UserName, not used by the services, you can set anything, but it must be present in the template. The password, it's the Azure Automation key. The typeName, the data type if the final value, which is a constant, System.Management.Automation.PSCredential.

To secure the Azure Automation key we can use protected settings. You need to add a key/pair value in the Items part.

"protectedSettings": {
               "Items": {
                      "azureAutomationPrivateKey": "[variables('azureAutomationKey')]"
                    }
            }

Ideally, the value should be stored in a Key Vault, just like the VM admin password.
We can now reference it in the registrationKey.

"Value": {
                          "UserName": "PLACEHOLDER_DONOTUSE",
                          "Password": "PrivateSettingsRef:azureAutomationPrivateKey"
    }

After the deployment, the VM will register to the Azure Automation service. The configuration is applied on the VM and Azure Automation, acting as a pull server will monitor any drift in the configuration.
You will find the log, just like in the first case we use.

With ARM Template we can go beyond VM provisioning. We can use an extension to deliver what final users demands, configure VM and make the service work. In this post, I used PowerShell, DSC, and Azure Automation. But you can also use other extensions like Azure Pipeline with Azure DevOps, Chef, Octopus Deploy and Puppet. You can also combine different extensions in the same template (but you cannot add the same extension).
There are many possibilities to close the gap between teams. In the age of Cloud, no needs to have 2 silos, one for the infrastructure and another for the app. They must work together.