DEV Community

Mikhail Sliusarev
Mikhail Sliusarev

Posted on

Extending Terraform functionality with a custom data source - part 1

I want to talk about writing a Terraform provider, but instead of focusing on creating a resource, as in many articles on the Internet, I want to cover a data source.

There usually are two reasons to write a terraform provider:

  • To wrap a remote API to manage remote infrastructure as code
  • To create additional utility functions in Terraform

I will show you how to extend Terraform functionality and add an additional utility function using a custom provider.

This article will also give your a bare-bone setup to start with plugin development that you can use for your project. Sounds good? Let's dive into it.

Configure your dev environment

To start, you need to set up the Go environment. Depending on the OS you are using, instructions could be different. Take a look here to find instructions for your system. Examples in this article are for OSX because that's what I use.

brew install go
Enter fullscreen mode Exit fullscreen mode

You, of course, will need to install Terraform. If you haven't tried tfenv, it is a straightforward way to manage different versions of Terraform on your local machine.

brew install tfenv
tfenv install 1.1.9
Enter fullscreen mode Exit fullscreen mode

There are currently two standards for building Terraform plugins: the old one SDKv2 and the new one Plugin Framework. Though Plugin Framework is the future, it was still under technical review when I wrote this, so we will look into SDKv2 as it is a more stable way of creating a plugin currently.

Hashicorp has a scaffolding repo to help you start developing a plugin, and it makes sense to review it. But I would like us to start with an empty repo and add files as we go along. I think it will be easier to learn how to write a plugin this way.

First, we should choose the name for our provider and follow the pattern when the plugin name is always terraform-provider-<name>. Even though we don't connect to any external provider's API, we still call it a Terraform provider.

It's essential to pick a name in the beginning because later on, this name will be used all over the code, and it will be a hassle to change it. The name will be required as a prefix to any resource or data source you will define, so it's better if the name is concise and has no special symbols. I will call our provider test, and the project name is terramform-provider-test.

Let's create a new folder and fill it out with the primary files we will need for this project.

mkdir terraform-provider-test
cd terraform-provider-test
touch main.go provider.go main.tf
Enter fullscreen mode Exit fullscreen mode

That's the list of files we will start with. Typically you would follow folder structure as in the scaffolding repo,, placing Go files inside the internal/provider folder and Terraform files inside the examples folder. But let us first understand how it all works together before imposing this structure.

We will look into the content of the files in the next section. But before we do that, there is one last thing to set up. Terraform has a particular configuration in your user's Home folder called .terraformrc. You probably don't have it if you never developed for Terraform. This file will help us create so-called developer overrides, which are needed to specify the path to the plugin object file which Terraform can load in.

touch ~/.terraformrc
Enter fullscreen mode Exit fullscreen mode

Fill it out with the path to the project folder you have created.

provider_installation {
  dev_overrides {
    "test" = "/Users/~your~path~/terraform-provider-test/"
  }

  direct {}
}
Enter fullscreen mode Exit fullscreen mode

Plugin entry point

First thing, let's initialise Go dependency tracking.

go mod init terraform-provider-test
Enter fullscreen mode Exit fullscreen mode

And now, we are ready to create an entry point for our plugin, starting with the main.go file.

package main

import (
    "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
    "github.com/hashicorp/terraform-plugin-sdk/v2/plugin"
)

func main() {
    plugin.Serve(&plugin.ServeOpts{
        ProviderFunc: func() *schema.Provider {
            return Provider()
        },
    })
}
Enter fullscreen mode Exit fullscreen mode

In this file, we start the plugin and initialise a Provider object. Let's create that object in provider.go.

package main

import (
    "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func Provider() *schema.Provider {
    return &schema.Provider{
        Schema: nil,
        DataSourcesMap: nil,
        ProviderMetaSchema: map[string]*schema.Schema{},
        TerraformVersion: "",
    }
}
Enter fullscreen mode Exit fullscreen mode

An empty Provider that doesn't do anything just to make the project build successfully. After that, install SDKv2 dependencies.

go get github.com/hashicorp/terraform-plugin-sdk/v2/plugin
go get github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema
Enter fullscreen mode Exit fullscreen mode

And build the plugin.

go build -o terraform-provider-test
Enter fullscreen mode Exit fullscreen mode

If you see a terraform-provider-test executable in your folder, congratulations, you built your first Terraform plugin. Now we can try using it with some Terraform code.

Our Terraform code

We will now look at creating a simple data source that takes some information in and allows printing something out. It will be easier to write code when you know what kind of result you expect, so let's map out our idea in Terraform code before implementing it in Go. The following goes into main.tf.

terraform {
  required_providers {
    test = {
      version = "0"
      source  = "test"
    }
  }
}

data "test_msg" "my_msg" {
  message = "My output message"
}

Enter fullscreen mode Exit fullscreen mode

I thought of doing something very simple for starters. As you can see from the code, the data source takes in some text information in the message field. It doesn't have to do anything with this message yet. We will look at how to load that plugin in and make terraform plan pass on that example.

When you develop a plugin, you don't have to initialise your repository code with terraform init, just use plan.

tfenv use 1.1.9
terraform plan
Enter fullscreen mode Exit fullscreen mode

It should fail with something like this.

╷
│ Error: Invalid data source
│
│   on main.tf line 10, in data "test_msg" "my_msg":
│   10: data "test_msg" "my_msg" {
│
│ The provider hashicorp/test does not support the data source "test_msg".
Enter fullscreen mode Exit fullscreen mode

This error indicates that Terraform could load our provider in memory and communicate with it, meaning all previous steps were correct. And we are finally ready to add the data source code.

Define the input values schema

Terraform plugin SDK helps you define and validate schema for the input values for resources you create. When you make a resource, you start by defining its schema. Let's open a new file called data_source_msg.go and create our schema.

package main

import (
    "context"

    "github.com/hashicorp/terraform-plugin-log/tflog"
    "github.com/hashicorp/terraform-plugin-sdk/v2/diag"
    "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func dataSourceMsg() *schema.Resource {
    return &schema.Resource{
        Description: "Output a custom message",

        ReadContext: dataSourceMsgRead,

        Schema: map[string]*schema.Schema{
            "message": {
                Type:     schema.TypeString,
                Required: true,
            },
        },
    }
}

func dataSourceMsgRead(ctx context.Context, d *schema.ResourceData, meta any) diag.Diagnostics {
    tflog.Info(ctx, "test-msg data source has been loaded")
    return nil
}
Enter fullscreen mode Exit fullscreen mode

In this file, we define the data source schema, and the read function, which will be called by Terraform when it finds your data source in the infra configuration.

Now return to the provider.go file and add DataSourcesMap to point to the resource we have just defined.

DataSourcesMap: map[string]*schema.Resource{
  "test_msg":  dataSourceMsg(),
}
Enter fullscreen mode Exit fullscreen mode

So our plugin exposes a new data source at this point, and we can test that everything works by building it and running terraform plan.

go build -o terraform-provider-test
TF_LOG=INFO terraform plan
Enter fullscreen mode Exit fullscreen mode

tflog is the standard way to debug your provider code, and it comes in handy here to print a little message once the provider is loaded. We should be able to see the message in the output log.

2022-10-02T20:06:55.212+0200 [INFO]  provider.terraform-provider-test: configuring server automatic mTLS: timestamp=2022-10-02T20:06:55.211+0200
2022-10-02T20:06:55.250+0200 [INFO]  provider.terraform-provider-test: test-msg data source has been loaded: tf_provider_addr=provider tf_req_id=dd639fe4-d927-04fc-0d17-cf1c89fe8f9e @caller=/terraform-provider-test/data_source_msg.go:27 @module=provider tf_rpc=ReadDataSource tf_data_source_type=test_msg timestamp=2022-10-02T20:06:55.250+0200
2022-10-02T20:06:55.252+0200 [INFO]  backend/local: plan operation completed

No changes. Your infrastructure matches the configuration.
Enter fullscreen mode Exit fullscreen mode

Conclusion

If everything worked out well for you in this article, you should have set up the basis for Terraform plugin development. In the next part, I will show you how to add more functionality to your plugin.

Reference

Top comments (0)