Build a CockroachDB Control Plane using Ansible Tower

CockroachDB can be easily deployed on the public cloud via Cockroach Cloud, Cockroach Labs DBaaS offering. With a few clicks, your Dedicated or Serverless cluster is ready in minutes if not seconds. Connect, and profit.

Some customers however face restrictions with regards to public cloud usage, preferring their own private cloud instead. Installing and deploying a CockroachDB cluster is very easy, but it's very hard to beat the convenience of a Control Plane if your goal is wide company adoption and streamlined maintenance (i.e. software upgrades).

In this blog, I use Ansible Tower as the cornerstone system to create a simple Control Plane. Tower's enterprise grade features allow you to create users and permissions, manage credentials and environments for executing basically any script or script workflow. Also, it has a little handy feature to create basic GUIs, the Survey, which I leverage to take user inputs.

The idea is that any user in the organization can login into Tower, create a cluster specifying its characteristics (size, number of nodes, regions, etc..) and receive the database connection string. Tower will then use an existing inventory of servers (or create new VMs), install and deploy CockroachDB, and return the connection details. You can of course add pre- and post-installation scripts at your will.

Setup

Ansible Tower, now evolved to a new product called Red Hat Ansible Automation Platform, is a licensed product. So I installed AWX instead, the upstream project that is free and open source.

AWX requires a Kubernetes environment, and on my laptop I created a single node K8s cluster using Minikube.

With AWX up and running, it is time to configure it and integrate it with other components.

Execution environment

First, we need an execution environment, that is, the k8s Pod image that Tower will use to run our "CockroachDB cluster create" script. Ansible provides a base image called ansible/awx-ee which I have extended using below Dockerfile and published as fabiog1901/awx-ee on Dockerhub.

FROM quay.io/ansible/awx-ee:latest

USER root

RUN pip install --upgrade pip

RUN ansible-galaxy collection install ansible.posix

RUN ansible-galaxy collection install community.general

RUN pip install boto boto3 botocore 

RUN pip install google-api-core google-auth google-cloud-compute googleapis-common-protos 

RUN pip install azure-common azure-core azure-identity azure-mgmt-compute azure-mgmt-core azure-mgmt-network azure-mgmt-resource

RUN pip install cockroachdb-cloud-client

USER 1000

Nothing crazy, just adding a few basic Ansible Collections and making sure the cloud provider python SDKs are installed. I mentioned before that the idea was to create a CockroachDB cluster for the private cloud, but as I don't have one, I use the public cloud instead. Assume that for the private cloud we will also install the private cloud python SDKs, for example, the OpenStack SDK or the VMWare vSphere SDK; the idea is to create a docker image that has all the libraries required by the script.

Here I added that docker image as the Default execution env in Tower

Repository Project

What has to be executed is provided to Tower via a Project. Tower integrates with GitHub so we can use a repository as our Tower Project. The sample repo I created contains a playbook to create a cluster and another playbook to destroy a cluster. The playbooks require an Ansible Collection, cockroachdb-collection, which has roles and modules to create VMs and deploy CockroachDB. As we have put the details of the collection in the collections/requirements.yml file, the collection is fetched automatically at runtime by Tower.

Template

A Tower Template is what describes what needs to be run. I created a "CockroachDB Create" template that collects the cluster information via a Survey, and runs the create.yaml file in my Project.

Please note that for the sake of brevity, there are many other details that I'm glossing over, like Credentials (AWS keys, SSH key, etc..), local variables, user creation, etc..

Here the details of the Survey:

Run

It's time to run our Template! I have created a user fabio that has only permission to run the 'CockroachDB Create' job. Now I am logged in with such user and click on the rocket icon to launch the job.
Please note that Tower powers an API server, so you are free to create your own Tower client and create better GUIs - you're not limited to Survey and Template via the standard Tower interface you've seen in these screenshot. Build your own web-client, and let the client call the Templates!

Details on how to connect to the cluster are printed out, but of course you can have better notification mechanisms, for example, you could have Tower send an email or a Slack message.

Creating the cluster on VMs took about 2 minutes, check the elapsed time on the top right.
I can now access my cluster, and profit.

Extend Tower to create clusters on CockroachCloud

You have by now realized that you can pretty much run anything via Tower: it's not a solution solely for creating CockroachDB clusters on private/public cloud infrastructure.

To highlight this, I've created a new Template that uses a new Ansible playbook, cc_create.yaml, that allows me to create a CockroachCloud cluster (a CockroachDB cluster on Cockroach Labs DBaaS service).
How does that work? CockroachCloud can be managed entirely via APIs, see the OpenAPI spec. From the spec JSON file I have generated a Python SDK, cockroachdb-cloud-client.
The SDK has allowed me to create Ansible Modules (still a work-in-progress project) which I am keeping in the same cockroachdb-collection. You can view the module docs here.

The new Template has a new Survey GUI, making it very easy to create a CockroachCloud cluster directly from Tower, where you can audit and track who can do what, and setup things like billing, reporting, etc.

Below is the flow for this new template, and once the Job has run, you'll see the familiar message with details of the newly created cluster

And just to confirm, the cluster is also visible in the CockroachCloud console

Closing Thoughts

This was a small, simple example on how we can use a system such as Tower to quickly build a Control Plane. While there are many aspects that we haven't covered, I hope I've shown what Tower can bring in terms of enterprise features and easiness of adding new workflows and scripts to automate or facilitate common tasks.

The choice is not limited to Tower, however. There is an interesting project that I'm following called Ansible Semaphore. It's by far not as mature as Tower, but worth keeping an eye on it.