DEV Community

Cover image for How to Wrap Your Terraform Provider for Pulumi
Speakeasy profile image Tatiana Caciur
Tatiana Caciur for Speakeasy

Posted on

How to Wrap Your Terraform Provider for Pulumi

#terraform #api #openapi #go

In this post, we'll show you how to make a Terraform provider available on Pulumi. The article assumes some prior knowledge of Terraform and Pulumi, as well as fluency in Go. If you are new to Terraform providers, please check out our Terraform documentation.

While we provide instructions for building a Pulumi provider here, the resultant provider is not intended for production use. If you want to maintain a Pulumi provider as part of your product offering, we recommend you get in touch with us. Without a partner, providers can become a significant ongoing cost.

Why Users Are Switching From Terraform to Pulumi

Following the recent HashiCorp license change, many users are exploring Pulumi as an alternative to Terraform.

The license change came as a surprise. In response, many companies are considering alternatives to manage their infrastructure-as-code setup. Given the scale of the Terraform ecosystem, it's unlikely that the license change will lead to Terraform disappearing. However, we can expect to see some fragmentation in the market.

If you're used to Terraform, you might notice that Pulumi has fewer providers available. Currently, Pulumi offers 125 providers in their registry, while Terraform boasts an incredible 3,511.

We know the comparison isn't completely fair, though – some users take the "everything-as-code" approach to the extreme, with providers for ordering pizza, building Factorio factories, and placing blocks in Minecraft. But even accounting for hobbyist providers, it's clear that Terraform has significantly more third-party support.

So, let's look at how we can shrink that provider gap...

How Pulumi Differs From Terraform

Pulumi and Terraform are both infrastructure-as-code tools, but they differ in many ways, most importantly in the languages they support.

Terraform programs are defined in the declarative HashiCorp Configuration Language (HCL), while Pulumi allows users to create imperative programs using familiar programming languages like Python, Go, JavaScript, TypeScript, C#, and Java.

This difference has some benefits and drawbacks. With Pulumi's imperative approach, users have more control over how their infrastructure is defined and can write complex logic that isn't easily expressed in Terraform's declarative language. However, this also means that Pulumi code can be less readable than Terraform code.

Bridging Terraform and Pulumi

Pulumi provides two tools to help maintainers build bridge providers. The first is Pulumi Terraform Bridge, which creates Pulumi SDKs based on a Terraform provider schema. The second repository, Terraform Bridge Provider Boilerplate, is a template for building a new Pulumi provider based on a Terraform provider.

While creating a new provider, we'll use the Terraform Bridge Provider Boilerplate, but we'll often call functions from the Pulumi Terraform Bridge.

Pulumi Terraform Bridge is actively maintained, so bear in mind that the requirements and steps below may change with time.

How the Pulumi Terraform Bridge Works

Pulumi Terraform Bridge plays an important role during two distinct phases: design time and runtime.

During design time, Pulumi Terraform Bridge inspects a Terraform provider schema, then generates Pulumi SDKs in multiple languages.

At runtime, the bridge connects Pulumi to the underlying resource via the Terraform provider schema. This way, the Terraform provider schema continues to perform validation and calculates differences between the state in Pulumi and the resource state.

Pulumi Terraform Bridge does not use the Terraform provider binaries. Instead, it creates a Pulumi provider based only on a Terraform provider's Go modules and provider schema.

Step by Step: Creating a Terraform Bridge Provider in Pulumi

The process of creating a Pulumi provider differs slightly depending on how the Terraform provider was created. In the past, most Terraform providers were based on Terraform Plugin SDK. More recent providers are usually based on Terraform Plugin Framework.

The steps you'll follow depend on your Terraform provider. Inspect the go.mod file in your provider to see whether it depends on github.com/hashicorp/terraform-plugin-sdk or github.com/hashicorp/terraform-plugin-framework.

Prerequisites

We manually installed the required tools before noticing that the Terraform Bridge Provider Boilerplate contains a Dockerfile to set up a development image. The manual process wasn't too painful, and either method should work.

The following must be available in your $PATH:

Terraform Plugin SDK to Pulumi

As an example of a Terraform provider based on Terraform Plugin SDK, we'll use this Spotify Terraform provider, a small provider for managing Spotify playlists with Terraform.

To create a new Pulumi provider, we'll start with the Terraform Bridge Provider Boilerplate.

Clone the Terraform Bridge Provider Boilerplate

Go to the Terraform Bridge Provider Boilerplate repository in GitHub and click on the green Use this template button. Select Create a new repository from the dropdown.

Select your organization as the owner of the new repository (we'll use speakeasy-api) and create a name for your Pulumi provider. In Pulumi, it is conventional to use pulumi- followed by the resource name in lowercase as the provider name. We'll use pulumi-spotify.

Click Create repository.

Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.

In the terminal, replace speakeasy-api with your GitHub organization name in the code below and run it:

git clone git@github.com:speakeasy-api/pulumi-spotify.git
cd pulumi-spotify
Enter fullscreen mode Exit fullscreen mode

Rename Pulumi-Specific Strings in the Boilerplate

The Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi. We need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.

The Makefile has a prepare command that handles some of the string replacement. In the terminal, replace speakeasy-api with your GitHub organization name in the command below and run it:

make prepare NAME=spotify REPOSITORY=github.com/speakeasy-api/pulumi-spotify
Enter fullscreen mode Exit fullscreen mode

The make command will print the sed commands it ran to replace the boilerplate strings in two files in the repo.

Next, we'll use sed to replace strings in the rest of the repo:

In the terminal, replace speakeasy-api with your GitHub organization name, then run:

find . -not \( -name '.git' -prune \) -not -name 'Makefile' -type f -exec sed -i '' 's|github.com/pulumi/pulumi-spotify|github.com/speakeasy-api/pulumi-spotify|g' {} \;
Enter fullscreen mode Exit fullscreen mode

In the Makefile, replace the ORG variable with the name of your GitHub organization.

Finally, in the provider/resources.go file, manually replace the values in the tfbridge.ProviderInfo struct. Many of these values define names and other fields in the resulting Pulumi SDK packages. Set the GitHubOrg to conradludgate.

Import Your Terraform Provider

Back in the provider/resources.go file, replace the github.com/terraform-providers/terraform-provider-spotify/spotify import with github.com/conradludgate/terraform-provider-spotify/spotify.

In a terminal run the following to change into the provider directory and install requirements.

cd provider
go mod tidy
Enter fullscreen mode Exit fullscreen mode

Fix a Dependency Version

This temporary step is a workaround related to the version of terraform-plugin-sdk imported in the boilerplate.

During our testing, we encountered a bug in terraform-plugin-sdk that was fixed in v2.0.0-20230710100801-03a71d0fca3d.

In provider/go.mod, replace replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20220824175045-450992f2f5b9 with replace github.com/hashicorp/terraform-plugin-sdk/v2 => github.com/pulumi/terraform-plugin-sdk/v2 v2.0.0-20230710100801-03a71d0fca3d. Note the difference in the version strings.

Remove Outdated make Step

This temporary step removes a single line from the Makefile that copies a nonexistent scripts directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a scripts folder containing install-pulumi-plugin.js, but Pulumi no longer generates these files.

In the Makefile, remove the cp -R scripts/ bin && \ line from build_nodejs:

makefile
build_nodejs:: VERSION := $(shell pulumictl get version --language javascript)
build_nodejs:: install_plugins tfgen # build the node sdk
    $(WORKING_DIR)/bin/$(TFGEN) nodejs --overlays provider/overlays/nodejs --out sdk/nodejs/
    cd sdk/nodejs/ && \
        yarn install && \
        yarn run tsc && \
        cp -R scripts/ bin && \ # remove this line
        cp ../../README.md ../../LICENSE package.json yarn.lock ./bin/ && \
        sed -i.bak -e "s/\$${VERSION}/$(VERSION)/g" ./bin/package.json
Enter fullscreen mode Exit fullscreen mode

Remove the Nonexistent prov.MustApplyAutoAliasing() Function

In provider/resources.go, remove the line prov.MustApplyAutoAliasing() from the Provider() function.

Build the Generator

In the terminal, run:

make tfgen
Enter fullscreen mode Exit fullscreen mode

Go will build the pulumi-tfgen-spotify binary. You can safely ignore any warnings about missing documentation. This can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide because the Pulumi boilerplate code does not include this step yet.

Build the Provider

In the terminal, run:

make provider
Enter fullscreen mode Exit fullscreen mode

Go now builds the pulumi-resource-spotify binary and outputs the same warnings as before.

Build the SDKs

In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.

In the terminal, run:

make build_sdks
Enter fullscreen mode Exit fullscreen mode

You can find the generated SDKs in the new sdk directory in your repository.

Terraform Plugin Framework to Pulumi

As we mentioned earlier, more recent Terraform plugins are based on Terraform Plugin Framework instead of Terraform Plugin SDK. The way Terraform Plugin Framework structures Go code adds a few extra steps when bridging a plugin to Pulumi.

The difference is significant enough that Pulumi Terraform Bridge includes a dedicated tool called Pulumi Bridge for Terraform Plugin Framework in its main repository.

As with providers created using Terraform Plugin SDK, Pulumi Bridge for Terraform Plugin Framework needs to create a new Go binary that calls the Terraform plugin's new provider function. Plugins created with Terraform Plugin Framework define their new provider functions in an internal package, which means we can't import the package directly.

To work around this, we'll create a shim that imports the internal package and exposes a function to our bridge.

But we're getting ahead of ourselves. Let's look at a step-by-step example.

Airbyte Terraform Provider

For this example, we'll bridge the Airbyte Terraform provider to Pulumi. While not the focus of this guide, it is worth mentioning that this provider was entirely generated by Speakeasy.

Clone the Terraform Bridge Provider Boilerplate

Go to the Terraform Bridge Provider Boilerplate repository in GitHub and click on the green Use this template button. Select Create a new repository from the dropdown.

Select your organization as the owner of the new repository (we'll use speakeasy-api again) and create a name for your Pulumi provider. Let's use pulumi-airbyte.

Click Create repository.

Use your Git client of choice to clone your new Pulumi provider repository on your local machine. Our examples below will use the command line on macOS.

In the terminal, replace speakeasy-api with your GitHub organization name in the code below and run it:

git clone git@github.com:speakeasy-api/pulumi-airbyte.git
cd pulumi-airbyte
Enter fullscreen mode Exit fullscreen mode

Rename Pulumi-Specific Strings in the Boilerplate

You'll remember that the Pulumi Terraform Bridge Provider Boilerplate in its current state is primarily an internal tool used by the Pulumi team to bring Terraform providers into Pulumi, so we need to replace a few instances where the boilerplate assumes Pulumi will publish the provider in their GitHub organization.

The Makefile has a prepare command that handles some of the string replacement. In the terminal, replace speakeasy-api with your GitHub organization name in the command below and run it:

make prepare NAME=airbyte REPOSITORY=github.com/speakeasy-api/pulumi-airbyte
Enter fullscreen mode Exit fullscreen mode

The make command will print the sed commands it ran to replace the boilerplate strings in two files in the repo.

Next, we'll use sed to replace strings in the rest of the repo.

In the terminal, replace speakeasy-api with your GitHub organization name, then run:

find . -not \( -name '.git' -prune \) -not -name 'Makefile' -type f -exec sed -i '' 's|github.com/pulumi/pulumi-airbyte|github.com/speakeasy-api/pulumi-airbyte|g' {} \;
Enter fullscreen mode Exit fullscreen mode

In the Makefile, replace the ORG variable with the name of your GitHub organization.

In the provider/resources.go file, manually replace the values in the tfbridge.ProviderInfo struct. Many of these values define names and other fields in the resulting Pulumi SDK packages.

Most importantly, in provider/resources.go, replace fmt.Sprintf("github.com/pulumi/pulumi-%[1]s/sdk/", mainPkg), with fmt.Sprintf("github.com/speakeasy-api/pulumi-%[1]s/sdk/", mainPkg),:

  ImportBasePath: filepath.Join(
-   fmt.Sprintf("github.com/pulumi/pulumi-%[1]s/sdk/", mainPkg),
+   fmt.Sprintf("github.com/speakeasy-api/pulumi-%[1]s/sdk/", mainPkg),
    tfbridge.GetModuleMajorVersion(version.Version),
Enter fullscreen mode Exit fullscreen mode

Remember to replace speakeasy-api with your organization name.

Remove Outdated make Step

This temporary step removes a single line from the Makefile that copies a nonexistent scripts directory while building the Node.js SDK. In earlier versions of Pulumi, the Node.js SDK included a scripts folder containing install-pulumi-plugin.js, but Pulumi no longer generates these files.

In the Makefile, remove the cp -R scripts/ bin && \ line from build_nodejs:

build_nodejs:: VERSION := $(shell pulumictl get version --language javascript)
build_nodejs:: install_plugins tfgen # build the node sdk
    $(WORKING_DIR)/bin/$(TFGEN) nodejs --overlays provider/overlays/nodejs --out sdk/nodejs/
    cd sdk/nodejs/ && \
        yarn install && \
        yarn run tsc && \
        cp -R scripts/ bin && \ # remove this line
        cp ../../README.md ../../LICENSE package.json yarn.lock ./bin/ && \
        sed -i.bak -e "s/\$${VERSION}/$(VERSION)/g" ./bin/package.json
Enter fullscreen mode Exit fullscreen mode

Remove the Nonexistent ‘prov.MustApplyAutoAliasing()’ Function

In provider/resources.go, remove the line prov.MustApplyAutoAliasing() from the Provider() function.

Clone the terraform-provider-airbyte repository (replace the destination with your destination):

git clone git@github.com:airbytehq/terraform-provider-airbyte.git /Users/speakeasy/terraform-provider-airbyte
cd /Users/speakeasy/terraform-provider-airbyte
Enter fullscreen mode Exit fullscreen mode

Replace the module name:

sed -i '' '1s|.*|module github.com/airbytehq/terraform-provider-airbyte|' go.mod
Enter fullscreen mode Exit fullscreen mode

Replace the module name in imports:

find . -name '*.go' -print0 | xargs -0 sed -i '' 's|airbyte/internal|github.com/airbytehq/terraform-provider-airbyte/internal|g'
Enter fullscreen mode Exit fullscreen mode

In the steps below, replace /Users/speakeasy/terraform-provider-airbyte with your local terraform-provider-airbyte repository.

Create a Shim To Import the Internal New Provider Function

Start with a new directory called provider/shim in your pulumi-airbyte project:

mkdir provider/shim
Enter fullscreen mode Exit fullscreen mode

Add a go.mod file to this directory with the following contents:

module github.com/airbytehq/terraform-provider-airbyte/shim

go 1.18

// TODO: This replacement is necessary only because airbytehq/terraform-provider-airbyte is not in https://pkg.go.dev/ and the module name is not an FQDN
// So to work around this we clone the terraform-provider-airbyte code and rewrite references to it to the local version.
replace github.com/airbytehq/terraform-provider-airbyte => /Users/speakeasy/terraform-provider-airbyte

require (
  github.com/airbytehq/terraform-provider-airbyte v0.0.0
  github.com/hashicorp/terraform-plugin-framework v1.3.5
)
Enter fullscreen mode Exit fullscreen mode

Now we'll add shim.go to this directory:

package shim

import (
    tfpf "github.com/hashicorp/terraform-plugin-framework/provider"
    "github.com/airbytehq/terraform-provider-airbyte/internal/provider"
)

func NewProvider() tfpf.Provider {
    return provider.New("dev")()
}
Enter fullscreen mode Exit fullscreen mode

Add Shim Requirements

To have Go gather the requirements for our shim module, run the following from the root of the project:

cd provider/shim
go mod tidy
cd ../..
Enter fullscreen mode Exit fullscreen mode

Import the New Shim Provider and the Terraform Package Framework Bridge

In provider/resources.go, edit your imports to look like this (replace speakeasy-api with your organization name):

import (
    "fmt"
    "path/filepath"

    "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge"
    "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge/tokens"
    shim "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim"
    // Import the Pulumi Terraform Framework Bridge:
    pf "github.com/pulumi/pulumi-terraform-bridge/pf/tfbridge"
    "github.com/pulumi/pulumi/sdk/v3/go/common/resource"
    "github.com/speakeasy-api/pulumi-airbyte/provider/pkg/version"
    // Import our shim:
    airbyteshim "github.com/airbytehq/terraform-provider-airbyte/shim"
)
Enter fullscreen mode Exit fullscreen mode

Instantiate the Shimmed Provider

In provider/resources.go, replace shimv2.NewProvider(airbyte.Provider()) with pf.ShimProvider(airbyteshim.NewProvider()):

func Provider() tfbridge.ProviderInfo {
    // Instantiate the Terraform provider
-   p := shimv2.NewProvider(airbyte.Provider())
+   p := pf.ShimProvider(airbyteshim.NewProvider())
Enter fullscreen mode Exit fullscreen mode

Add the Shim Module as a Requirement

Edit provider/go.mod, and change the requirements to the following:

require (
    github.com/airbytehq/terraform-provider-airbyte/shim v0.0.0
    github.com/pulumi/pulumi-terraform-bridge/pf v0.17.0
    github.com/pulumi/pulumi-terraform-bridge/v3 v3.61.0
)
Enter fullscreen mode Exit fullscreen mode

Also in provider/go.mod, replace github.com/airbytehq/terraform-provider-airbyte/shim with ./shim as shown below, to let the Go compiler look for the shim in our local repository:

replace (
    github.com/airbytehq/terraform-provider-airbyte/shim => ./shim
  // TODO: The following replacement is only necessary because the terraform-provider-airbyte Go module is not named github.com/airbytehq/terraform-provider-airbyte and it does not appear in the Go package registry
  // This replaces references to github.com/airbytehq/terraform-provider-airbyte with a reference to our local terraform-provider-airbyte repository
  github.com/airbytehq/terraform-provider-airbyte => /Users/speakeasy/terraform-provider-airbyte
)
Enter fullscreen mode Exit fullscreen mode

Install Go Requirements

From the root of the project, run:

cd provider
go mod tidy
cd ..
Enter fullscreen mode Exit fullscreen mode

Build the Generator

In the terminal, run:

make tfgen
Enter fullscreen mode Exit fullscreen mode

Go will build the pulumi-tfgen-airbyte binary. You can safely ignore any warnings about missing documentation. The missing documentation warnings can be resolved by mapping documentation from Terraform to Pulumi, but we won't cover that in this guide as the Pulumi boilerplate code does not include this step yet.

Build the Provider

In the terminal, run:

make provider
Enter fullscreen mode Exit fullscreen mode

Go now builds the pulumi-resource-airbyte binary and outputs the same warnings as before.

Build the SDKs

In the final step, Pulumi generates SDK packages for .NET, Go, Node.js, and Python.

In the terminal, run:

make build_sdks
Enter fullscreen mode Exit fullscreen mode

You can find the generated SDKs in the new sdk directory in your repository.

Summary

We hope this comparison of Terraform and Pulumi and our step-by-step guide to bridging a Terraform provider into Pulumi has been useful to you. You should now be able to create a Pulumi provider based on your Terraform provider.

Speakeasy can help you generate a Terraform provider based on your OpenAPI specifications. Follow our documentation to enter this exciting ecosystem.

Speakeasy also has a Pulumi target in beta. Join our Slack community to discuss our upcoming Pulumi release or for expert advice on Terraform providers.

Top comments (0)

Read next

eduardoarndt profile image

Go Design Patterns #4: Abstract Factory

Eduardo -

evolvedev profile image

Building a Google Drive Downloader in Golang (Part 1)

EvolveDev -

marcos_placona profile image

How to add Retrieval-Augmented Generation (RAG) to your app using generated SDKs

Marcos Placona -

binoy_59380e698d318 profile image

Terraform - Kong API Gateway deployment in Kubernetes

Binoy -