Using PowerShell and Pester to validate Azure Policy syntax

I write a lot of custom Azure Policy, and like everyone, I believe, I made some mistakes, and I want to avoid waiting until I try to deploy policies to catch these errors.
But what kind of error to avoid, in Azure Policy editing there are 3 levels of errors:

• Syntax error, when you have a malformed JSON document or a mistake in your parameter
• Integration error, when something doesn’t work in the Policy, an object type that doesn’t exist for example
• Logical error, when you can deploy and assign the policy, but the policy doesn’t work as expected

Let’s try to address the first problem. What can we do to avoid and report syntax errors? I use PowerShell and PowerShell has a testing framework, Pester. An Azure Policy is A JSON file that can be easily manipulated in PowerShell, PowerShell can extract parts of the policy and test them.

What to test:

• If the file is a valid JSON file
• If the policy has a valid displayName
• If the policyRule contains an IF and a THEN clause
• If the policy has a valid effect
• If the policy mode is valid
• If all parameters are used

If one, at least, of these tests fails, the policy will not work.

To test a policy, I use a pester script. The script reads the policy JSON file and performs these tests.
As we are dealing with external data, such as the Azure Policy, we need to make sure that data are available at all levels in the pester script. Pester has scope where data are available at IT or a Describe Level and data available inside It block.
For the first one, I use beforeDiscovery

BeforeDiscovery {
    $policyRawContent = Get-content -Path "..\policies\auditVMExtension.json" -Raw 
    $policyContent = $policyRawContent | ConvertFrom-Json -Depth 16
    $parametersList = $policyContent.parameters | Get-Member -MemberType NoteProperty | select-object -ExpandProperty name
}

Before running any test for each file, we need to extract some data from the policy. Pester can do that with BeforeAll in the context section.

beforeAll {
    $policyRawContent = Get-content -Path "..\policies\auditVMExtension.json" -Raw 
    $policyContent = $policyRawContent | ConvertFrom-Json -Depth 16
    $policyIfContent = $policyContent.policyrule.if.allOf.count + $policyContent.policyrule.if.anyOf.count
    $policyThenContent = $policyContent.policyrule.then.count
    $policyRuleString = $policyContent.policyRule | ConvertTo-Json -Depth 10    
    $azPolicyEffect = @("Append","deny","audit","auditIfNotExists","DenyAction","deployIfnotexist","disabled","modify")
    $policyMode = @("indexed", "all")
}

The testing starts with a describe section that includes the policy display name

describe "test Policy for policy <policyContent.displayName>" {

The first test is to valid that the file is a valid JSON file

    it "Should be a valid json file" {
        $policyRawContent | test-json -ErrorAction SilentlyContinue | should -Be $true
    }

The second test, checks if the displayName is valid, here if the length is more than 10 characters.

    it "Should have a valid displayName" {
        $policyContent.displayName.length | Should -BeGreaterThan 10
    }

The third test determines if the “if” section is valid or not. In the BeforeAll section the "$policyIfContent" counts the number anyOff and allOff clauses, it should have at least one, if not the policy is not valid.

    it "Should have a valid if section" {
        $policyIfContent | should -BeGreaterThan 0
    }

The same for the then section, it should not be empty.

    it "Should have a valid then section" {
        $policyThenContent | should -BeGreaterThan 0
    }

After that, the script tests if the policy has a valid mode either All or Indexed. It is a best practice Indexed includes only resources that have a tag and a location, the all keyword meaning every resource. The “$policyMode” variable is declared in the beforeAll section.

    it "Shoud have a valid mode" {
        $policyContent.mode | Should -BeIn $policyMode
    }

The next test determines if the effect is valid, The value should be in the $azPolicyEffect defined in the BeforeAll section

    it "Should have a valid effect" {
        $policyEffect | Should -BeIn $azPolicyEffect
    }

The last test is about parameters. We try to see if all defined parameters are used in the policy rule. The list of parameters is generated in the BeforeDiscovery section (because it is in the IT scope)

$policyContent = $policyRawContent | ConvertFrom-Json -Depth 16
$parametersList = $jsonContent.parameters | Get-Member -MemberType NoteProperty | select-object -ExpandProperty name

This array is used in a loop to test if the parameter is present in the policyRule. In the It declaration you can use “<>” to print the tested parameter, inside the section you will use the “$” variable.
The test checks if the parameter is present in the policyrule (transformed in a String representation) by using the contains method of the string object>

    it "Parameter <_.>  Should be present in the present in the policyRule section" -ForEach $parametersList {
        $b.Contains("[parameters('$_')]") | Should -BeTrue 
    }

The complete file is available here

This is a simple way to test the sanity of a policy file. Here tests are only about the syntax and what we should expect to find in an Azure Policy file. But this test will not validate the policy itself, and will not ensure that you can deploy the policy, assign it, and work as expected. That is another part of Azure Policy testing.