One of the common challenges in providing sufficient and understandable information about AWS infrastructures is the creation of architectural diagrams. The process of creating these diagrams often involves certain assumptions about the audience's technical level and their familiarity with the diagram's content. This list is not meant to be exhaustive, but it highlights the complexities involved.
Creating these diagrams can be quite tedious and time-consuming. Additionally, AWS environments can change rapidly, which increases the workload if you want to keep everything up to date.
Approaches to Infrastructure Deployments with Visuals
Classic Approach
The classic approach involves creating the diagram upfront. This method is particularly useful when negotiating with stakeholders from IT security, customers, or data protection teams. While the diagram remains accurate until the first release date, its relevance tends to diminish over time.
Database Approach
To address this, other approaches focus on creating a database of the current AWS infrastructure. AWS itself provides solutions for this, but there are also AWS partners, like Lucid with their Lucid Spark product, which allows you to gather resources. While this approach keeps pace with the dynamic nature of AWS environments, the collection process is often not purposeful, as it typically gathers all resources indiscriminately. Consequently, a post-hoc collection and rearrangement process is necessary.
Visual Creation
A third approach involves visually creating the infrastructure and compiling the corresponding artifacts. This method is especially useful for beginners, as it allows them to interact with AWS indirectly and with a low barrier to entry. However, challenges arise when attempting to implement CI/CD processes or deploy multiple environments simultaneously.
Diagrams During or After Deployment
The fourth approach focuses on creating diagrams during or after deployment. This method aims to generate a resource representation based on your Terraform or CloudFormation scripts. While this approach keeps the diagrams aligned with the current architectural design, it lacks interactivity, which can be a significant drawback. The resource selection must be meaningful to foster a proper understanding of architecture.
Diagrams as Companions of the Cloud Development Kit (CDK)
The Cloud Development Kit (CDK) offers a powerful toolkit for creating complex and dynamic deployments in various ways. The CDK utilizes CloudFormation in combination with AWS services like S3, ECR, and Lambda, creating a dynamic abstraction that aids in deployment creation. In relation to the four approaches mentioned earlier, using the CDK aligns closely with the fourth approach.
One of the great features of the CDK is that it automatically builds a class tree of resources by default. This resource tree provides an opportunity to create a visual representation of the stack. Another advantage is the vibrant supporting community, where many talented individuals contribute new features daily. Finally, with support for native programming languages, I have yet to encounter a deployment task that could not be implemented with the CDK.
I recently found an open-source project that has the potential to improve the fourth approach significantly. The parent project is called cdk-graph.
Features of cdk-graph
- Synthesizes a serialized graph (nodes and edges) from CDK source code.
- Provides a runtime interface for interacting with the graph (in-memory database-like graph store).
- Provides a plugin framework for additional tooling to utilize and extend the graph . The goal of this framework is to enable bespoke tooling to be built without having to first traverse the CDK Tree and Metadata to build a graph. Projects like cdk-dia generate a bespoke in-memory graph that is then utilized to generate diagrams. By standardizing on the graph interface necessary to build complex tooling, we can more rapidly build new tooling that focuses on its core value.
Implementing AWS PDK with Python
Let's create the CDK sample app:
XYZ@123:~/dev-to$ mkdir -p sample-app
XYZ@123:~/dev-to$ cd sample-app/
XYZ@123:~/dev-to/sample-app$ cdk init --language=python
This creates a simple CDK repository with a single SQS queue. I then added an example to my sample_app_stack.py. You can take another example if you like: AWS CDK Examples.
To run everything accordingly, we need to install these packages:
XYZ@123:~/dev-to/sample-app$source .venv/bin/activate
(.venv) XYZ@123:~/dev-to/sample-app$ pip install -r requirements.txt
(.venv) XYZ@123:~/dev-to/sample-app$ pip install aws-pdk asyncio
Now let's take a look at app.py, as this file needs to be changed a bit:
#!/usr/bin/env python3
from aws_pdk.cdk_graph import CdkGraph, FilterPreset, Filters, FilterValue
from aws_pdk.cdk_graph_plugin_diagram import CdkGraphDiagramPlugin
import asyncio
import aws_cdk as cdk
from sample_app.sample_app_stack import SampleAppStack
async def main():
"""Orchestration of CDK creation process. This function is needed to make things work in an asynchronous way."""
app = cdk.App()
SampleAppStack(
app,
"SampleRepoStack"
)
graph = CdkGraph(
app,
plugins=[
CdkGraphDiagramPlugin(
diagrams=[
{
"name": "diagram - 1 - filter compact",
"title": "Diagram SampleRepoStack",
"filterPlan": {
"preset": FilterPreset.COMPACT,
},
},
{
"name": "diagram - 2 - no filter",
"title": "Diagram SampleRepoStack",
"filterPlan": {
"preset": FilterPreset.NONE,
},
},
{
"name": "diagram - 3 - focus on API Gateway",
"title": "Diagram SampleRepoStack",
"filterPlan": {
"filters": [
{
"graph": Filters.include_cfn_type([
FilterValue(regex=r"AWS::ApiGateway::*")
]),
},
],
},
}
]
),
],
)
app.synth()
graph.report()
if __name__ == "__main__":
"""We call the orchestration function via asyncio lib to make things work in an asynchronous way."""
asyncio.run(main())
diagram - 1 - filter compact
This mode is always agood starting point shows meaningful defaults
diagram - 2 - no filter
This diagram cannot bee shown here (due to image restrictions), but holds all resources from the app apart from IAM Policies.
diagram - 3 - focus on API Gateway
This daigram uses a RegEX filter to strip down the diagramm to all resource starting with r"AWS::ApiGateway::*"
Due to the current architecture of the plugin, the entire process of CDK synthesis needs to be asynchronous. Based on the results of the process, the pictures are generated. From the code perspective, you need to implement a main() function around app = cdk.App() and app.synth(). With that, we add graph = CdkGraph( … ) and after the app.synth() command, the graph.report() command. Finally, we introduce a boilerplate to call the main() safely and asynchronously.
As you can see in this example, we can leverage a wide range of filters as shown here: CDK Graph Plugin Diagram. These filters allow you to create one or multiple versions of visual representations of your architecture. The filters can be explicit, static, or design-based regular expressions.
Final Words
The plugin can be implemented quickly, increases the understanding of the CDK app, and brings additional value regarding the documentation of your work for third-party people. I found two situations where it’s currently not working. Some L3 Constructs, like those from AWS Service Catalog, do not support image creation, as the graph creation process breaks due to multiple steps in the sequence of the app creation. You can find the issue here: GitHub Issue. Besides that issue, you may face errors while including predefined CloudFormation stacks in your CDK app.
All in all, the package adds additional value to our pull request reporting, and if you are using CDK, take a look. And if you want to go even futher check out cdk-nag and the Cdk Graph Threat Composer Plugin. Security guys will love you.
Happy Coding :-)
Top comments (0)