Speeding up the creation of infrastructure code

Building cloud applications is hard. The number of services available from public cloud providers is astonishing, and these offerings can dramatically increase your team’s velocity. And yet, integrating these services into a product is hard: documentation is not always as clear or well-organized as you’d like, and all the more when a particular product feature might need to direct requests to 5 or 6 different AWS services, each of which requires piles of JSON to configure correctly. Here's an approach I've used successfully across several different applications recently:

Step 1: Create whatever you need in the AWS Web Console

Amazon’s web console holds your hand. This is a good thing, but should be used with care. The goal is not to do it once, then forget about it, but to do it the easy way, and the capture this in code.

You probably know from perusing the docs what AWS services you're going to need: so fire up the console and start clicking away! For instance, you might make that S3 bucket with a dozen different life-cycle rules, or that ECR repo with a score of tag expiration settings. Depending on what you're doing, this might require coordinating resources across different AWS services (e.g. making an SQS queue, and a lambda).

Step 2: Use the AWS CLI tools to dump the configuration of the stuff you’ve made

Trying to manually figure out how to map from AWS's Web Console to CloudFormation or API Client code takes forever. Fortunately, the AWS-CLI provides a much better way. You can use this to dump the JSON of all the entities you created in step 1. Maybe you're working with CloudFront. You can do this to list your distributions:

aws cloudfront list-distributions > distributions.json

The exact command varies, of course, depending on the service. aws <servicename> help will generally lead you to the right command. Generally you want one that starts with list or describe (e.g.

aws ec2 describe-instances
aws dynamodb describe-table --table-name <TABLE_NAME>

Remember that the aws-cli tools are fundamentally just convenience wrappers to REST APIs, which is why they often aren’t terrible user friendly. REST APIs and traditional *nix tools have different structures and conventions (For instance, a *nix command would usually have required params expressed as positional arguments rather than named ones).

Step 3: Craft this into either into CloudFormation or client SDK code

Now that you have a JSON representations of the entities you need, you can write these up as code, either as CloudFormation/CDK/Terraform templates, or with the AWS client SDKs. In my particular case, I wanted to create these resources in response to a REST API call, so I wrote up the creation logic in TypeScript using the AWS -v3 JavaScript SDK. These are awesome to use for a lot of reasons. I was able to copy and paste much of the JSON from step and just tweak the params that were different for each entity. And since it was TypeScript, I was told at compile time if a parameter was named incorrectly. It made juggling hundreds of lines of JSON configuration much less error-prone and frustrating.

Conclusion: The AWS CLI is your friend

I’ve used this strategy to put together ECS blue/green deploy code and to set up some complicated video workflows with AWS's Elemental service suite. Juggling mountains of JSON is not fun. Creating what you need in the web console, by contrast, and then exporting via the CLI will speed up your develop of infrastructure-facing code, especially if you can’t find good examples to follow. Good luck!