Creating A Terraform Provider - Part 1

Cross posting from Medium, this is a post I wrote for my employers' blog - Space Ape Tech

A fairly common task in Ops is to figure out how to manage a service programmatically; it’s often easy to get started with a service by using the UI but this won’t scale and eventually, we’ll want to have more control over the configuration of a service; a Terraform provider can be the tool that gives you that control.

To create a Terraform provider you just need to write the logic for managing the Creation, Reading, Updating and Deletion (CRUD) of a resource and Terraform will take care of the rest; state, locking, templating language and managing the lifecycle of the resources.

Just taking a look at the list of existing providers shows you how versatile Terraform can be. The list includes cloud providers, Kubernetes, DNS services, GitHub, monitoring tools and TLS certificate providers among others.

At Space Ape we decided that we wanted to start managing the configuration of our metrics service, Wavefront, programmatically. We were all set to write a new tool that would allow us to template Alerts and Dashboards in YAML and then create them in Wavefront via their API. We had already started writing a Go client for the Wavefront API when Hashicorp announced the release of Terraform v0.10.0, which split Providers out of Terraform core allowing each provider to be developed and released independently of Terraform and for custom providers to be created.

So we set about building the Wavefront Terraform Provider.

How to Build Your Own Provider

To avoid having to work against a real API for this blog post, so that you can follow along if you wish, I’ve created a small API that stores (imaginatively) “items”. The API allows you to create, read, update and delete items. An item has a name, description and a list of tags. An items’ name must be unique. I won’t go into much more detail about the working of the API, you can find a more detailed description here and the code in terraform-provider-items/api.

The client that the Terraform provider will use to interact with the API shouldn’t be implemented within the provider itself, as that would overcomplicate the provider and mean that the client can’t be used for other purposes. So I’ve also written a client in the API package.

The full source code of the example provider and API is available on GitHub.

Getting Started

The provider we’re going to build during this blog post will allow us to create an Item on our server using the following Terraform code:

Terraform Plugins are binaries that Terraform communicates with via RPC. It’s possible to write a provider in any language, but in reality, you’ll want to write it in Go; Terraform provide helper libraries in Go to aid in writing and testing providers.

The name of the repository (and therefore directory) that your provider lives in is important; all providers start with terraform-provider- anything after that represents the name of the provider. In this case, I’m going for the very imaginative terraform-provider-example.

Next, we’ll create ./main.go which will serve as the entry point to our provider. main.go is just used to invoke our provider, which we will implement in a separate package, in this case, called provider.

Now, we’ll create the provider package, within which will sit the implementation of your provider. Within the provider package, we’ll create provider.go and define the Provider function that our main.go called.

The Provider requires:

• A Schema which represents the various attributes we can provide to our provider via the provider block of a Terraform file. Note that if no value is provided we will check if environment variables are set. This is useful for making sure we don’t need to store secrets in the provider block of terraform files
• A ResourceMap defines the names of the resources the provider has and where to find the definition of those resources. In this case, you can see we have example_item resource, the definition of which is a *schema.Resource returned by the resourceItem() function, which we’ll define later
• A ConfigureFunc which can do any setup for us. In this case, we have providerConfigure which takes the address, port and token and returns a client that we’ll use to communicate with the API. Note the providerConfigure returns an interface{} so we can store anything we like here

Defining Our Item Resource

Before we start defining our resource there are some more naming conventions to cover:

• Resource files are named resource_[resourceName].go, eg resource_item.go
• Test files follow the usual Go naming standard of resource _[resourceName]_test, eg resource_item_test.go
• We can also write import tests which, by convention, are in files named import_[resourceName]_test.go, eg import_item_test.go. Although these tests could sit within the resource_[resourceName]_test.go file
• Lastly, there are also data source definitions, which we won’t be implementing here, but you may see in other providers. These are named data_source_[resourceName].go. Data Sources allow you to pull in information from resources that already exist, but that you don’t want to manage.

We start by creating resource_item.go and defining the resourceItem() function that we call from provider.go. This returns a *schema.Resource— the definition of our Item resource.

A schema.Resource needs us to set up a few things:

• A Schema— the attributes the Resources has
• A number of functions. Earlier I said you just need to set up the Create, Read, Update and Delete functions for the provider, there are also two more, Exists and Imports, I’ll cover these later on.

Let’s look at the Schema in more detail. The Schema element is a _map[string]*schema.Schema, where the string is the attribute name and the *schema.Schema defines what the attribute is. In this case, the attribute name is of a Type TypeString, it is required and has a short description. It also has ForceNew set to true, this is because the API doesn’t allow you to change the name of an item after it is created, therefore Terraform would have to destroy the resource and create it again for this to happen.

Lastly, there is a ValidateFunc which is a function with the signature (v interface{}, k string) (ws []string, es []error)

Where v is the value of the attribute, which you need to use type assertion on to retrieve the actual type, k is the name of the attribute (“name” in this case) and it returns a slice of warnings (strings) and a slice of errors.

The ValidationFunc used for the name attribute is the validateName function:

It returns an error if the name has any whitespace (The API doesn’t allow whitespace in names). This prevents us from making API calls we know will fail.

The description attribute is similar to name but doesn’t force a new resource and doesn’t have a validation func.

The tags attribute has a few differences. It is of type TypeSet, which is similar to a list, but where order doesn’t matter. You can use TypeList when you can be sure that the API will always return a list in the same order, otherwise, you’ll always have changes to apply. You’ll also notice that tags has a Elem field, which lets us define the type that is stored in the Set (or List), in this case, a string.

In our case, the API purposely returns tags in a random order. You can try changing the TypeSet to TypeList to see that there will (nearly) always be changes to apply due to the reordering of the tags.

It’s also worth noting that when you have a TypeSet or TypeList the Elem field can be of type &schema.Resource, which allows for a deeper resource structure. For instance, this is how the listener blocks of an aws_elb resource are defined.

Also notice that tags are not a required attribute, but rather than just setting Required to false (it’s default value), we set Optional to true. Either Required or Optional must be true

Functions

The Create, Read, Update and Delete functions are functions with the signature (d *schema.ResourceData, m interface{}) error. Where d is essentially the Schema we defined above, but with values added, e.g. the “name” attribute has a value we can send to our API.

m is the interface{} returned from the ConfigureFunc in provider.go, which in our case is our Client for talking to the server.

Each of the CRUD functions essentially just call their corresponding methods in the client, but there are a few things to note:

• When you create a resource you need to set the ID of the Terraform resource, this is done using the d.SetId method. The ID is generally the what the API uses to uniquely identify an item, in our case name is the ID, so we use that — d.SetId(item.Name)
• Setting the ID to an empty string indicates to Terraform the item no longer exists. So in the resourceDeleteItem() function we call d.SetId("") after deleting the Item. We also do this in the resourceReadItem function if we get an error and that error contains “not found”
• For each of these methods we obtain the Client from m, as it is an interface we have to assert the type as the client type — apiClient := m.(*client.Client)

As the underlying data structure for the d *schema.ResourceData ends up being an interface, rather than a solid type we end up doing a lot of type assertion. On a small resource like our Item it is easy enough to do it within the function but, for larger resources, it may be worth splitting these out into separate functions.

In the resourceCreateItem function, we can see that we use d.Get to retrieve the values from the resource that we wish to pass to the API via the client and that we must perform type assertion on the result d.Get("name").(string).

The Exists function (resourceExistsItem()) is slightly different in that it doesn't modify Terraform state, nor does it update any resource on the server; it’s just used to check if a resource exists. It has a slightly different function signature than the CRUD functions in that it also returns a bool to indicate if the resource exists — resourceExistsItem(d *schema.ResourceData, m interface{}) (bool, error)

Finally, there is an import function which is used for importing a resource into Terraform, an import function is an implementation of &schema.ResourceImporter. Terraform provides an implementation of this called schema.ImportStatePassthrough, which seems to work for the majority of use cases and you can always write your own implementation if you need to.

End of Part 1

At this point we have a functional Terraform provider, you could compile it and start creating Items with Terraform. However, there is a important piece missing —  tests! In part 2 we’ll add tests to our Provider and run through how we get Terraform to use the provider.

---