Secure DevOps with Pulumi and Azure AD

Pulumi, Terraform, Bicep etc are tools that enable developers and IT Pros to build and deploy infrastructure to the cloud using code. Infrastructure as Code is a great way to implement elaborate solutions using tools and languages you already know (in most cases).

We recently invited Sean Whitesell on the #425Show to show us how to get started with Pulumi and Azure. If you wish to catch the video, it's now available on YouTube

This blog expands on that streams and focuses on how to securely deploy infrastructure to Azure using Pulumi and Azure Key Vault.

When it comes to protecting sensitive information and secrets in Pulumi, there are two main options:

1. Using a Service Principal to query Azure Key Vault at deployment time
2. Using an encryption provider to store secrets

1.1 Create a service principal to use for the deployment:

First of all, we need to create a Service Principal in Azure AD. Open the Azure CLI and type the following:

az login
az ad sp create-for-rbac -n pulumi-deployment-sp --scopes /subscriptions/<YourSubscriptionId>

The result should look like this:

{
  "appId": "1fdd1348-ccb0-4ec5-9799-edd14dd72e13",
  "displayName": "pulumi-deployment-sp",
  "name": "http://pulumi-deployment-sp",
  "password": "<it's a secret>",
  "tenant": "e801a3ad-3690-4aa0-a142-1d77cb360b07"
}

Make sure to keep this information safe as we'll need to use it later in the set up

1.2 Configure the Key Vault Access policy

The Service Principal can now be used to deploy resource but doesn't have access to our Key Vault secrets. We will use Key Vault to store the sensitive information such as passwords, keys and connection strings to decouple the Pulumi code and the config settings. That way, the code becomes a lot more portable and IT Admins can audit the usage of secrets to ensure there's no foul play.

In the Azure portal, open your Key Vault and navigate to the Access Policies blade. Add a new Policy as per the image below:

The Service Principal should now be able to List and Get secrets from our Key Vault

1.3 Configure Pulumi to authenticate to Azure using a Service Principal

Next, we need to set up Pulumi to use the Service Principal in order to authenticate to the Azure Resource Manager to deploy our resources. Since Pulumi uses Terraform behind the scenes, the environment variables that we need to configure are exactly the same as per this TerraForm document

In our PowerShell session (I don't want these stored permanently on my machine) we set up the following environment variables using the output information that was provided when we created our Service Principal:

$env:ARM_CLIENT_ID="<Service Principal App ID>"
$env:ARM_CLIENT_SECRET="<Service Principal secret/password>"
$env:ARM_SUBSCRIPTION_ID="<Subscription ID>"
$env:ARM_TENANT_ID="<Azure AD Tenant ID>"

1.4 The actual code

To access our Azure Key Vault secrets during the deployment process, we can use the following code:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

interface Data{
    azurerm_key_vault: {
        id:string
    }
};

let config = new pulumi.Config();
let data = config.requireObject<Data>("data");

const sql_username_secret = azure.keyvault.getSecret({
    name: "pulumi-sql-username",
    keyVaultId: data.azurerm_key_vault.id
});

const sql_password_secret = azure.keyvault.getSecret({
    name: "pulumi-sql-password",
    keyVaultId: data.azurerm_key_vault.id
});

export const sql_username = sql_username_secret.then(secret => secret.value);
export const sql_password = sql_password_secret.then(secret => secret.value);

In case you're wondering where we got the data object, this is something we initialize using the Pulumi CLI. Alternatively, we can edit the Pulumi.<environment>.yaml directly. Currently, the data object holds the Azure Key Vault resource id which we can set via the Pulumi CLI with this command:

 pulumi config set --path 'data.azurerm_key_vault.id' /subscriptions/<mySubscriptionId>/resourceGroups/<myResourceGroup>/providers/Microsoft.KeyVault/vaults/<myKeyVaultName>

Running this command will update the Pulumi.dev.yaml file like this:

config:
  azure:environment: public
  azure:location: WestUS
  pulumiwithaad:data:
    azurerm_key_vault:
      id: /subscriptions/d801110....

If we run the Pulumi up command with the code provided above, we should receive the following output

2. Using the KeyVault Secrets provider for Pulumi

The other option, to keep our config settings secret, is to use an Secrets Encryption Provider to encrypt our sensitive data. By default, this is provided out of the box by the Pulumi Service but there is an option to explicitly set your own Encryption Provider. The available options are:

• default
• passphrase
• awskms
• azurekeyvault
• gcpkms
• hashivault

Of course, I'll chose to go with the Azure Key Vault provider because I can then use Managed Identities or Service Principals to run my deployment tasks.

There are a few steps needed to achieve this. For local development, we need to:

2.1 Enable Azure Key Vault auth via the CLI

First we need to configure the following environment variable and set it to true. Open the PowerShell session you will be running your tasks and execute the following command:

$env:AZURE_KEYVAULT_AUTH_VIA_CLI = "true"

Next, we need to ensure that the Azure CLI is using correct Service Principal. To do this, we can run the following command. In the same PowerShell session log in to the Azure CLI with the following command:

az login --service-principal --username <Service Principal Client Id> --tenant <Azure AD Tenant Id> --password <Service Principal Client Secret>

Alternatively, we could use a certificate to authenticate our Service Principal, which is more secure than passing credentials around.

2.2 Configure the Secrets Provider

Next we need to update Pulumi to use Key Vault as the secrets provider. This can be accomplished by running the following command in the Pulumi CLI:

pulumi stack change-secrets-provider "azurekeyvault://<yourKeyVaultName>.vault.azure.net/keys/<TheKeyToUseForEncryptionDecryption>"

2.3. Update the Access Policy in Key Vault

If we try to run our Pulumi script now, it's guaranteed that we'll receive an 'Unauthorized' error message. That's because our Service Principal needs to be able to use the designated Key Vault key to encrypt and decrypt data.

To fix this error, we need to update the Azure Key Vault Access Policy as per the image below:

• Add a Key Vault backed secret to our config file

2.4 Add a secret

With Pulumi primed to use Key Vault as its secret provider, we can now add a new secret to our code. Open the Pulumi CLI and run the following command:

pulumi config set --secret someOtherSecret Hunter2!

This will add an entry: pulumiwithaad:someOtherSecret in our Pulumi.dev.yaml file as well as references to our Secrets Provider and EncryptedKey

secretsprovider: azurekeyvault://<myKeyVault>.vault.azure.net/keys/pulumiKey
encryptedkey: aGxiU1Jq...<omitted for brevity>...VHVLNUE=
config:
  azure:environment: public
  azure:location: WestUS
  pulumiwithaad:data:
    azurerm_key_vault:
      id: /subscriptions/d8011108-23b2-40d8-8bc4-1f3f77abe795/resourceGroups/dotnetconf/providers/Microsoft.KeyVault/vaults/cmdeploymentdemo
  pulumiwithaad:someOtherSecret:
    secure: v1:6oxydcsaDaemg43V:fV6Oy11srnDjxdlRVwsPb5xMo6nS/y6Q

If we run pulumi preview then we can see how our secret is set as expected:

In our Pulumi code, we can reference the secret by calling:

const kvSecret = config.requireSecret("someOtherSecret");

We are now ready to deploy our SQL server to Azure! Let's do this:
Pulumi up

It works!

Source Code

You can find this sample code on this GitHub repo

Summary

Sean was great on the show and we learned quite a bit about Pulumi and how to use it with Azure AD. This blog post took things a bit further to explain how to further lock down and secure your DevOps pipeline by removing secrets and relying on Service Principals and Azure Managed Identity to authenticate to Azure and run your deployment. Let us know in the comments if you have any questions!