Devops Automation Examples in Practice

Original post By: Adam Duvander

Cloud native engineering is a flexible discipline. However, the many DevOps tool options and integration methods can be confusing. More complexity often means more difficulty automating tasks. DevOps programs should evolve toward process improvement. Improvements in code delivery will drive greater transparency in project communication, wiser adoption of shared resources, and make more room for quality assurance.

In this post, we’ll look at three examples for DevOps automation. They use Jira, Jenkins, and Docker—but the concepts of event-triggered workflows will translate to whatever tools you use.

Build Resources (and Transparency) From Work Tickets

Transparency is key to open communication within DevOps teams. Imagine a world where any project manager can see and build information in a business-friendly context. Taking some time to synchronize your Jenkins pipelines with JIRA, for example, would give a PM real-time insight to EC2 deployment jobs, and prevent hail storms of status pulse checks. The following steps will promote Jira from a 2D digital cork board into a mission control interface, giving orders and taking receipts. You could use a similar approach with your project management and build tools.

To set up the automation, prepare your Jira project to trade information to Jenkins through a webhook. Let’s say your project is an app migration and you need to repeatedly provision a suite of AWS services for testing. Instead of manually uploading CloudFormation scripts, add custom fields in the Issue configuration console that will tell Jenkins where to find those scripts and what parameters to provision them with. Then create a Build Status field that will be controlled by updates from Jenkins. Verify that those new fields now appear in your tickets. Next, under the Advanced configuration menu, create a connection with Jenkins by plugging its server into Webhooks configuration form. Select “Created” and “Updated” as Events triggers.

Now it’s Jenkins’ turn. We’re going to set it up to receive build parameters and pass build statuses. We are assuming the pipelines are already built and we simply need to get them working with the Jira webhook. To do so, install the following plugins and configure them to connect to your Jira account, and more specifically, the project prefix for the app migration board:

• Jira
• Jira Pipeline Steps
• Jira Trigger Plugin
• jira-ext Plugin

These will allow Jenkins to listen for the appropriate updates from Jira, as well as expose functionality to communicate with the triggering ticket from the build.

With both applications now locked into each other, update your pipeline to handle the Jira payload and hand it off to the build. Write a JQL filter in the pipeline configuration so it can only be triggered by “Create” and “Update” webhook events. Next, map the CloudFormation template location and the other parameters as Custom Fields and make sure the Jira ticket key is also captured as an Issue Attribute Path field. The CloudFormation template and parameter are now available as environmental variables to guide that build. Also, the ticket key value is available for the new functions exposed by the Jira plugins to build status updates and log info directly into the ticket for all stakeholders to see.

An automation like this can help you do less repetitive work. It does take some effort to set up and maintain the connections between your tools. To simplify tedious workflows and find example designs, be sure to check out the event-driven automation tool that Puppet is building.

Streamline Your Docker Builds with Dynamic Variables

Docker, like CloudFormation, is a powerful tool for rebuilding apps and infrastructure on demand. Here we’re going to extend the power of parameterization into Docker-based pipeline builds. Ideally, you could transport a Docker container as-is into any environment. In practice, however, there will be differences. Maybe you have different database endpoints for development, testing, and production. If your application’s integration tests rely on hitting the right endpoint per environment, we can use build-specific parameters to keep your Docker containers environment-agnostic.

Here’s how you can route parameters for a test environment build:

pipeline {
    // ...
    agent {
        dockerfile {
            filename "my-dockerfile"
            label "my-docker-label"
            args "-v DB_URL= ${env.DB_URL}" // set to 'test_db.example.com'
        }
    }
    // ...
}

Let’s assume we’re passing a controlled set of parameters from the triggering action into Jenkins, like in the previous example. We’re running a test pipeline, so Jenkins received the database endpoints as DB_URL=test_db.example.com from the triggering action. We can access this value in our Jenkinsfile from the env object.

Now we must funnel the DB_URL value into the Docker build. We begin by declaring a build agent. To maximize the use of your team’s Docker program, pull in a custom-built Dockerfile, by declaring your custom Dockerfile as your agent. Next we must pass the environment variable as an interpolated string to the args: args “-v DB_URL= ${env.DB_URL}”. In your custom Dockerfile, map the DB_URL ARG instruction to an ENV instruction. Now that URL will be accessible as a variable in your containerized test scripts without having to modify the file at all.

Of course, these are the sorts of common cloud wiring that Relay is built to support. Anything that happens frequently based on events is a good match, which is why testing is another area ripe for automation.

Make Testing Routine Inside of your Containerized Build

Testing is a non-negotiable pillar of any software development program, but it can become a bottleneck to deployment. DevOps Teams often mitigate the logjam with a less-than-comprehensive testing regime. For example, a combination of functional and unit testing on updates to an API may suffice to verify the CRUD functionality with a minimum passable integration to a locally spun up datasource. However, you can replicate all of the APIs’ dependencies and run an extensible suite of tests within a one-off docker-compose job with little overhead.

For this example, let’s imagine you are building a small Node app that reads and writes to MongoDB. It will be easy to represent the stack in a docker-compose file. Docker-compose files are YAML scripts that tell Docker what to inject into a Dockerfile build. In this case, we’re going to build a Node.js image in the Dockerfile and declare two services in our docker-compose.yml: the API service and a MondoDB instance. Save this docker-compose file and a Node.js Dockerfile to build it into an integration-test subdirectory in the project repo.

You can now perform some sanity checks by running docker-compose against the integration-test directory by running Postman actions against the local API and checking MongoDB on whichever port you exposed it to. More importantly, we’re just a few short files away from automatically testing the full integrated build. Create an index.js file with your test scripts. Point your API calls and database operations at the names you gave your services in the docker-compose file and return results within the container. Lastly, create a shell script to manage the docker containers and output the test results. Now you can automatically trigger the tests in other CI/CD pipelines, version control them, and add more services or tests as the business logic evolves.

Quickly Deploy Event-driven Workflows

As you can see, your workflows are more efficient when you build enough overhead to scale out event-triggered behavior. We’ve seen how dynamically orchestrating Docker builds from Jira can kick off extensible testing programs within Jenkins. A good optimization principle to consider now is iteration. Some of these integrations may work or falter for your needs. Being able to swap out and remap your services without turning off the pipelines keeps your DevOps workflows flowing.

Relay by Puppet seeks to manage that for you by automating modular event-driven workflows. If one integration stops meeting your needs, you can swap tools or reroute the workflow and automate your DevOps environment.