In my previous blog, How do you prove that your infrastructure is compliant.
I explained how you can prove your infrastructure is compliant using CloudFormation Guard.
But, how do you write those rules? And even more important, how do you test your rules? If you look at the repository CloudFormation Guard.
You will notice that the project itself offers a testing framework.
Alright! Let’s build a ruleset and write some tests for it!
Let’s build a ruleset for S3
We will start with a sample for S3, create a file called: rules/s3.guard.
let buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]
rule BucketEncryption when %buckets !empty {
%buckets.Properties {
BucketEncryption.ServerSideEncryptionConfiguration[*] {
ServerSideEncryptionByDefault.SSEAlgorithm IN ["AES-256", "aws:kms"] <<Ensure all S3 buckets use encryption-at-rest>>
}
}
}
Now we will write the test, called: rules/s3_tests.yaml.
---
- name: "Skip if not present"
input:
Resources: {}
expectations:
rules:
BucketEncryption: SKIP
- name: "No encryption used"
input:
Resources:
MyBucket:
Type: AWS::S3::Bucket
expectations:
rules:
BucketEncryption: FAIL
- name: "Bucket with KMS Encryption"
input:
Resources:
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
expectations:
rules:
BucketEncryption: PASS
- name: "Bucket with AES-256 Encryption"
input:
Resources:
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES-256
expectations:
rules:
BucketEncryption: PASS
- name: "Bucket with UNKNOWN Encryption"
input:
Resources:
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: UNKNOWN
expectations:
rules:
BucketEncryption: FAIL
Now it’s time to run the tests:
cfn-guard test --rules-file rules/s3.guard --test-data rules/s3_tests.yaml
This will give the following output:
Test Case #1
Name: "Skip if not present"
PASS Rules:
BucketEncryption: Expected = SKIP, Evaluated = SKIP
Test Case #2
Name: "No encryption used"
PASS Rules:
BucketEncryption: Expected = FAIL, Evaluated = FAIL
Test Case #3
Name: "Bucket with KMS Encryption"
PASS Rules:
BucketEncryption: Expected = PASS, Evaluated = PASS
Test Case #4
Name: "Bucket with AES-256 Encryption"
PASS Rules:
BucketEncryption: Expected = PASS, Evaluated = PASS
Test Case #5
Name: "Bucket with UNKNOWN Encryption"
PASS Rules:
BucketEncryption: Expected = FAIL, Evaluated = FAIL
You might be thinking, that looks nice! But the output is quite verbose, and you need to execute a command per rule/test set.
It gets even worse when a test does fail. For this example I altered the rules/s3.guard file to use aws:kmz instead of aws:kms.
Then it looks like this:
Test Case #1
Name: "Skip if not present"
PASS Rules:
BucketEncryption: Expected = SKIP, Evaluated = SKIP
Test Case #2
Name: "No encryption used"
PASS Rules:
BucketEncryption: Expected = FAIL, Evaluated = FAIL
Test Case #3
Name: "Bucket with KMS Encryption"
FAILED Rules:
BucketEncryption: Expected = PASS, Evaluated = FAIL
Test Case #4
Name: "Bucket with AES-256 Encryption"
PASS Rules:
BucketEncryption: Expected = PASS, Evaluated = PASS
Test Case #5
Name: "Bucket with UNKNOWN Encryption"
PASS Rules:
BucketEncryption: Expected = FAIL, Evaluated = FAIL
If you look at Test Case #3 it says: BucketEncryption: Expected = PASS, Evaluated = FAIL. That makes it hard to spot what test failed.
cfn-guard-test
For this reason I created a python package called cfn-guard-test. This package
makes it easier to run many rule/test sets. This is especially useful in the CI/CD pipelines.
Please read the repository for the installation instructions.
Let’s run the tests again!
cfn-guard-test --verbose
The output now looks something like:
===== Analyzing Results =====
Running rules/s3_tests.yaml
Passed 5
Failed 0
That is a nice and clean overview. It lists the number of passed and failed tests. For the next example I altered the
rules/s3.guard file to use aws:kmz instead of aws:kms. This will again trigger a failure:
===== Analyzing Results =====
Running rules/s3_tests.yaml
Passed 4
Failed 1
Rule BucketEncryption failed on #3 "Bucket with KMS Encryption" in rules/s3.guard
Well you have to agree. That immediately sums up how many passed and failed. And more it shows that the BucketEncryption
rule fails. And it failed on the test case "Bucket with KMS Encryption".
You don’t want to spend time figuring out what is wrong. This is why we use tooling, to make our life easier.
This tool also has the ability to generate JUnit report.
cfn-guard-test --verbose \
--junit-path "reports/cfn-guard.xml"
Conclusion
cfn-guard-test can help you run many tests. And get an aggregated report of
the results. This tool will help you detect bugs in your rules easier and quicker.
Top comments (0)