Putting DevOps On Autopilot With LEGO-Like Automation

Original Post: https://blog.totalcloud.io/post/putting-devops-on-autopilot-with-lego-like-automation?utm_source=dev-to&utm_medium=referral&utm_campaign=flagship

The Workflow Editor, our core offering, can be thought of as a DevOps engineer’s personal drawing board, among other things. It primarily enables you to create completely customized workflows from scratch to automate all cloud actions. Before we delve into it, let’s break it down. The ‘editor’ is the platform and a ‘workflow’ is a logical sequence of steps that you create on the editor to achieve any desired output. A fitting analogy would be LEGOs. The LEGO baseplate, on which you actually build your LEGO blocks is equivalent to our editor; and whatever object you build is comparable to a completed workflow. Each block of lego is akin to a ‘node’, which we’ll get into later. Look at it as Ryan Van Duzor putting together thousands of LEGO pieces to build an intricate coral reef . The editor can create workflows as complex as the reef; but without the need for a thousand pieces, just a bunch of our nodes.

Notice how we like to think of the editor as a platform, and not just a tool because it’s a medium that enables automation for your cloud with limitless flexibility. The editor helps with something as simple as starting an EC2 instance to something as arduous as predicting Lambda costs daily, or finding and attaching missing CloudWatch alarms to resources. The flexibility is as vast as LEGOs themselves, letting you craft almost any kind of workflow in a completely intuitive manner. A single workflow can be used to perform multiple actions on multiple resources.

The Editor can be considered to be an automation-enabler, because it has been built in a way that completely eliminates the need to code. You need not build, test and deploy scripts to perform any workflow. This not only lets you automate simple repetitive tasks, but also extremely complex actions, all in a matter of minutes (it’s as easy as creating a flowchart). The editor enables you to visualize your use-cases in the form of a flowchart, instead of code, to give you a better overview of your entire action.

What a workflow involves

In simple terms, any cloud use-case can be developed using a workflow. For instance ‘Delete Unattached EBS Volumes’ is a use-case; and this can be broken down into its workflow elements; which are:
when you want to delete these volumes
the resource that you’re performing the action on (in this case EC2 volumes)
what the action actually is (in this case deleting unattached EBS volumes)

The image below is what this ‘workflow’ looks like on our editor.

Each element of a workflow is called a ‘node’. So in this use-case, the ‘when’ becomes the ‘trigger node’, the ‘what’ becomes the ‘resource’ and ‘filter nodes’, the actual ‘action’ becomes the ‘action node’. These constitute our basic nodes, to enable the creation of any workflow.

While creating a workflow, you can think of the sequence of actions you’d take on the AWS console or CLI to quickly decide the nodes you pick. We’ve also built out multiple other nodes that take customization a level up. Every node can be edited and configured according to your requirements.

How It Works

As we’ve highlighted, workflows enable you to achieve any use-case to efficiently manage your cloud. But cloud management’s primary requirement is insight into your cloud. These insights help you gain complete visibility into the security, cost and operational efficiency of your cloud. Upon receiving these insights, you can choose to take action on those insights. The editor enables you to evolve from gaining insight to taking action on it, with extremely simple workflows. Usually, when an insight exposes an issue, you’ll want to act on it to remediate it. So naturally, workflows can largely be of two types, insights and actionable workflows.

Let’s understand this with an example. Take the use-case - Missing CloudWatch Alarms. This involves two parts, finding or identifying the missing alarms on resources, which constitutes an insight; and the second part, taking action to attach those missing alarms to the resources, which becomes an actionable workflow.

Part 1: Insight - Finding and Reporting Missing CloudWatch Alarms

AWS allows you to set up CloudWatch alarms to oversee a CloudWatch metric. The alarm can notify you depending on the value of the metric relative to the defined threshold. Some examples are CPU usage alarms and load balancer latency alarms. If a resource is missing an alarm that was set to notify you of a certain threshold, it can lead to system outages or performance blockades, adversely affecting the reliability and availability of your infrastructure. The editor can be used to create a workflow that generates a report of all the resources that are missing alarms, and exactly which alarms are unattached. The workflow looks something like this:

• The workflow can be initiated through a trigger that you define, for instance, everyday.
• The resource nodes select ‘alarms’ as an addon to all the resources for which you want to check missing alarms.
• The filter nodes’ parameters are defined to filter out alarms which do not exist
• A custom node emits this data, with details of the specific alarms missing, into the report node
• A report node is used to generate the collective report

Part 2: Action - Attaching Missing CloudWatch Alarms to Resources

Having received information about the missing CloudWatch alarms through the insight, you can choose to automate the remediation of this use-case. A workflow can be created to attach the missing alarms. This is where the action node comes in. This node can be configured to attach missing alarms to each individual resource. Take the example of EC2 instances, and an organization using 1000 such instances. If the report generated in Part 1 found that 100 out of these 1000 instances are missing different alarms, a single workflow can be set up to attach different alarms to different instances. You can also add a user approval node in the workflow prior to the action node, if you have the organizational obligation to receive permission before attaching the alarms.

What’s next for TotalCloud

With more than 170 templates, we enable you to simply plug and play the most complex of use cases, and we’re adding more as we progress, like CloudTrail based workflows.

Having developed the powerful Workflow Editor, we now strive to enhance and easen your experience of cloud automation. We’re releasing ‘Insights’ as a bundled solution, to give you visibility into security, cost, and efficiency of your cloud. Each insight bundle can be adopted as a whole, and can be auto-scanned every 24 hours.

With a comprehensive resource scheduling solution on its way, we enable huge cost savings. It not only allows basic scheduling, but also real-time scheduling based on metric thresholds, along with schedule and rightsizing recommendations.

External integrations will become easier, with tools like Jenkins, Freshdesk, Zendesk, Jira and Telegram configurations in progress. This will enable you to not only trigger workflows from any of these external platforms with HTTP trigger, but also send and receive data through API calls.

Conclusion

TotalCloud’s editor not only empowers you with information about your cloud through insights, but also equips you to take action on it. The amenability of the action node is so wide, that it lets you solve simple and complex use-cases alike. A single workflow can be created for multiple resources, irrespective of the number; enabling you to scale easily. With the focus on automation, your devops engineers can spend their time on innovation, instead of worrying about scalability. You can get started with creating a workflow right away with a TotalCloud account. If you have any feedback or ideas, let us know in the comments section!