loading...
Cover image for stackmanager: Yet another tool for CloudFormation
Leading EDJE

stackmanager: Yet another tool for CloudFormation

andrewdmay profile image Andrew May ・5 min read

I recently started building stackmanager, an Open Source project for managing CloudFormation stacks.

There are already a number of other tools for doing this, and I've used or tried several of them, but none quite fit how I wanted them to work.

Functional Requirements

Managing all the configuration used to create/update a CloudFormation stack in a single file

The CloudFormation create-stack command has lots of separate values you need supply, and with the CLI it's a pain to supply Stack Name, Template File, Parameters, Targets, Capabilities, etc.

Support for deploying the same stack to different environments (e.g. dev, prod, or different regions) while reducing duplication of configuration

Almost all the CloudFormation I write is used in multiple different environments (often different accounts), and sometimes is deployed to multiple regions for failover.

Uses ChangeSets, with control over whether to immediately apply or apply later

ChangeSets allow you to preview (to a limited degree) the changes you're making, and especially if you're using the AWS Serverless Application Model transformation, or a CloudFormation macro, you don't necessarily know exactly what resources are going to be created until the CloudFormation service process the template.

In some CI/CD workflows, there may be an approval requirement before changes go into production, and the ChangeSet allows you to preview changes.

Log progress

Print a summary of the ChangeSet and show the CloudFormation events whether the stack update succeeds or fails. I really like how SAM shows this information.

Support Lambda Functions

This is somewhat outside the core functionality, but as I started writing this while working on a project that deployed Lambda functions using CloudFormation (where the S3 Key is changed when there is new code, triggering an update), I wanted to be able to use a single tool to build, upload and deploy Lambda functions, similar to the SAM CLI, but better suited to more general CloudFormation management and CI/CD.

Lambda

Non-functional requirements

This is not the first utility of this sort I've written, but previous versions have belonged to the client who I wrote them for. I wanted to create an Open Source utility that I could use at multiple clients and for Leading EDJE infrastructure.

Writing stackmanager from scratch allowed me to improve on what I'd written previously, and include things I'd previously neglected like unit tests.

Even if I end up the only one contributing to the code base I also wanted to run it like a regular open source project with a list of issues, pull requests and releases when I reached a suitable milestone.

Issues

Getting to 1.0

A lot of Open Source projects never reach a 1.0 release, but I just wanted to release something relatively complete and well tested and not necessarily include everything that stackmanager could possibly do. The hardest thing to finish in the list of issues I'd included in the 1.0 milestone was to get acceptable code coverage, as my experience writing Python unit tests is somewhat limited, especially when it comes to mocking.

Releases

Using stackmanager

I don't really want to rewrite the README here, but stackmanager allows you to do things like:

$ stackmanager --profile dev --region us-east-1 \
build-lambda --source-dir integration/functions/python/hello_world/ --output-dir /c/dev/temp/ --runtime python3.7 \
upload --bucket stackmanager-lambda-files-us-east-1 --key python/hello_world.zip \
deploy --environment dev --config-file integration/functions/python/config.yaml --auto-apply

Building python3.7 Lambda function from integration/functions/python/hello_world/

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Built Lambda Archive C:\dev\temp\hello_world.zip
2020-10-28 20:33:21 Found credentials in shared credentials file: ~/.aws/credentials

Uploaded C:\dev\temp\hello_world.zip to s3://stackmanager-lambda-files-us-east-1/python/hello_world.zip

Stack: d-StackManager-PythonFunction, Status: does not exist

Creating ChangeSet c8a1e3704d0da48f69903bafcaf239c86

Action    LogicalResourceId    ResourceType           Replacement
--------  -------------------  ---------------------  -------------
Add       FunctionRole         AWS::IAM::Role         -
Add       Function             AWS::Lambda::Function  -

Executing ChangeSet c8a1e3704d0da48f69903bafcaf239c86 for d-StackManager-PythonFunction

ChangeSet c8a1e3704d0da48f69903bafcaf239c86 for d-StackManager-PythonFunction successfully completed:

Timestamp                  LogicalResourceId              ResourceType                ResourceStatus      Reason
-------------------------  -----------------------------  --------------------------  ------------------  ---------------------------
2020-10-28 20:33:24-04:00  d-StackManager-PythonFunction  AWS::CloudFormation::Stack  REVIEW_IN_PROGRESS  User Initiated
2020-10-28 20:33:35-04:00  d-StackManager-PythonFunction  AWS::CloudFormation::Stack  CREATE_IN_PROGRESS  User Initiated
2020-10-28 20:33:39-04:00  FunctionRole                   AWS::IAM::Role              CREATE_IN_PROGRESS  -
2020-10-28 20:33:39-04:00  FunctionRole                   AWS::IAM::Role              CREATE_IN_PROGRESS  Resource creation Initiated
2020-10-28 20:33:53-04:00  FunctionRole                   AWS::IAM::Role              CREATE_COMPLETE     -
2020-10-28 20:33:56-04:00  Function                       AWS::Lambda::Function       CREATE_IN_PROGRESS  -
2020-10-28 20:33:57-04:00  Function                       AWS::Lambda::Function       CREATE_IN_PROGRESS  Resource creation Initiated
2020-10-28 20:33:57-04:00  Function                       AWS::Lambda::Function       CREATE_COMPLETE     -
2020-10-28 20:33:59-04:00  d-StackManager-PythonFunction  AWS::CloudFormation::Stack  CREATE_COMPLETE     -
Enter fullscreen mode Exit fullscreen mode

This built a Python Lambda function, uploaded the zip file to S3 and then deployed a CloudFormation stack that used the file in S3, previewing the change-set and then running it.

In this case the functionality is similar to the SAM CLI for Lambda functions, but the focus of stackmanager is really the deploy command that is for any CloudFormation stack and not just Lambda functions.

At the heart of stackmanager is the configuration file that can encapsulate the configuration for multiple environments. For the example I ran above this is the configuration file:

---
Environment: all
StackName: "{{ EnvironmentCode }}-StackManager-PythonFunction"
Parameters:
  Environment: "{{ Environment }}"
Tags:
  Application: StackManager
  Environment: "{{ Environment }}"
Template: integration/functions/python/template.yaml
Capabilities:
  - CAPABILITY_IAM
---
Environment: dev
Region: us-east-1
Variables:
  EnvironmentCode: d
Enter fullscreen mode Exit fullscreen mode

In this simple example the inheritance between the dev environment and the all environment isn't particular useful, but once I have multiple environments and a mix of common and unique parameters this really cuts down on the total amount of configuration (and the number of separate files to edit).

Is stackmanager right for you?

If you're already using CloudFormation, but you have a mess of parameter files and other arguments you're using to run your CloudFormation then I'd like to think this will help you clean up the mess.

Moving forward there's a possibility that CDK will obsolete writing CloudFormation directly, but there's a lot of CloudFormation already out there so it's going to be around for a while.

Leading EDJE dev.to

Discussion

pic
Editor guide