How to deploy to multiple providers with dynamic credentials at once in Terraform CDK for Python

My organization primarily uses AWS, so this post's code predominantly targets AWS providers but the principles for supporting multiple configurations for dynamic provider credentials in the Terraform CDK for Python should apply to all providers, just consult their specific documentation for details.

Table of contents

• Terraform Cloud dynamic provider credentials
• Configuring dynamic provider credentials for multiple providers
• Using dynamic credentials for multiple providers in the Terraform CDK for Python

Terraform Cloud dynamic provider credentials

If you use Hashicorp's Terraform and Terraform Cloud products, you may be familiar with a feature they recently added known as dynamic provider credentials. If not, it's a mechanism which allows runners in Terraform Cloud (TFC) which execute your infrastructure changes to authenticate to your target cloud platform with scoped, temporary credentials. It uses the OpenID Connect (OIDC) protocol which defines a trust relationship, and in the context of AWS it allows the TFC runners to assume a configured role. This post assumes a familiarity with this authentication method. More information can be found here:

• Dynamic provider credential documentation
• Tutorial about setting up dynamic provider credentials

For those unfamiliar, the gist of this setup for AWS is:

1. Create an identity provider in the IAM service of each of your target AWS accounts.

2. Create a role with sufficient permissions that Terraform Cloud runners can assume within your AWS account and specify this role in the trust relationship.

3. Declare these two environment variables in the remote configuration for your Terraform Cloud workspace:

   • TFC_AWS_PROVIDER_AUTH = true
   • TFC_AWS_RUN_ROLE_ARN = ARN of the IAM role you created in the above step, e.g. arn:aws:iam::123456789010:role/TerraformCloudDynamicProviderCredentialsRole

Here's a diagram I made showing the high-level process for how the TFC runner assumes the given role to execute infrastructure changes:

Configuring dynamic provider credentials for multiple providers

An early limitation with this authentication setup was that it only supported a single instance of a given provider, which meant Terraform workspaces could only target a single AWS account at a time using this authentication method. Hashicorp eventually implemented support for configuring dynamic provider credentials for multiple providers simultaneously¹. This means it's possible to now design workspaces that manage infrastructure state in multiple AWS accounts for example, instead of having to split these deployments into separate workspaces.

This is a useful feature in my organization namely because we use the AWS IAM Identity Center (formerly AWS Single Sign-On) to allow our customers to authenticate to AWS with their organizational identity using SAML. As part of our workflow, we preferred to provision a client's infrastructure in a new dedicated account in our AWS organization, and then create corresponding Single Sign-On (SSO) roles that they use to login to that account. These SSO roles must be managed in the AWS organizational root account², meaning we can now design workspaces that:

1. Create the dedicated AWS organizational account
2. Create infrastructure in this new account
3. Generate policies to access this infrastructure in this account
4. And now create an SSO permission set in the root account and associate the aforementioned customer managed IAM policies in the new AWS account to this permission set

And all this in a single invocation of apply!
This greatly simplified our workspace design and coordinating state for a single customer project.

TFC workspace configuration for multiple dynamic provider credentials

Here are Hashicorp's official docs regarding this setup: https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/specifying-multiple-configurations

Configuring dynamic credentials for multiple providers still uses the workspace environment variables mentioned above, however you will now create multiple pairs of these environment variables that share a common underscore (_) delimited suffix called a "tag" in the documentation. In my testing I found it was necessary to make these "tags" match identically to the provider aliases I specified in my code.

For the following example, let's say I have two target AWS accounts that are pre-configured to support dynamic provider credentials³. The account with ID 012345678910 will be labelled "A" and the account with ID 109876543210 will be labelled "B".

I would likely use the following AWS provider blocks in my Terraform HCL:

provider "aws" {
  alias = "A"
  allowed_account_ids = ["012345678910"]
  ...
}

provider "aws" {
  alias = "B"
  allowed_account_ids = ["109876543210"]
  ...
}

Corresponding to these provider blocks, you would extend the TFC workspace environment variables accordingly:

Another important configuration change is also needed:

You must also add a specific Terraform variable to your workspace code to reference which set of provider credentials your resources should use.

Hashicorp calls this out in their docs here, and below I copy-paste verbatim the variable configuration for AWS providers on this page: https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/aws-configuration#required-terraform-variable

variable "tfc_aws_dynamic_credentials" {
  description = "Object containing AWS dynamic credentials configuration"
  type = object({
    default = object({
      shared_config_file = string
    })
    aliases = map(object({
      shared_config_file = string
    }))
  })
}

And then by adapting their sample AWS provider definitions using this variable, we would edit the above AWS provider codeblock like so:

provider "aws" {
  alias = "A"
  allowed_account_ids = ["012345678910"]
+ shared_config_files = [var.tfc_aws_dynamic_credentials.aliases["A"].shared_config_file]
  ...
}

provider "aws" {
  alias = "B"
  allowed_account_ids = ["109876543210"]
+ shared_config_files = [var.tfc_aws_dynamic_credentials.aliases["B"].shared_config_file]
  ...
}

And voilà! With this configuration, we can define resources which specify provider = aws.A or provider = aws.B and can coordinate infrastructure changes in both accounts with the same call to terraform plan and terraform apply.

Here's a diagram I made which illustrates the high-level process for how this new configuration coordinates multiple dynamic provider credentials:

Using dynamic credentials for multiple providers in the Terraform CDK for Python

Backgound

My organization was previously using the AWS CDK for Python so we ended up building many supporting modules with battle-hardened business logic also in Python. Naturally we were hesitant to discard these mature libraries during our migration to using Terraform, so we had an interest in using the Terraform CDK for Python (also called cdktf) so that we could still use these Python libraries for common infrastructure patterns we support among our customers.

In the process of adapting the configuration for dynamic credentials for multiple providers to the Python cdktf, I personally found Hashicorp's documentation on this subject to be somewhat underdeveloped, at least as of writing. There are many examples in HCL, but the Python cdktf docs seem to still mostly be auto-generated and don't currently have a lot of examples to reference.

Given this, to hopefully spare others the trial and error I endured, I wrote this post to share a sample of how to configure dynamic credentials for multiple providers in the Python cdktf for anyone else hoping to take advantage of these technologies.

Python cdktf Code

First, let's translate the above AWS providers for our 2 accounts into Python cdktf code since the translation is relatively straighforward:

import cdktf
from constructs import Construct
from cdktf_cdktf_provider_aws.provider import AwsProvider

class AppStack(cdktf.TerraformStack):
  def __init__(self, scope: Construct, stack_id: str):
    super().__init__(scope, stack_id)

    aws_account_A_id    = '012345678910'
    aws_account_A_alias = 'A'

    aws_account_B_id    = '109876543210'
    aws_account_B_alias = 'B'

    aws_provider_A = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_A_alias}",
                                       alias = aws_account_A_alias,
                                       allowed_account_ids=[aws_account_A_id])

    aws_provider_B = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_B_alias}",
                                       alias = aws_account_B_alias,
                                       allowed_account_ids=[aws_account_B_id])

And now for the trickiest part, we will need to translate the above HCL definition for the required Terraform variable tfc_aws_dynamic_credentials into cdktf code. Since the variable is only defined at runtime on a TFC runner, it's not possible to use Python-native value-accessing syntax. Instead, you must use the cdktf built-in remote state management functions.

Here's the fully translated variable declaration:

tfc_aws_dynamic_credentials =(
  cdktf.TerraformVariable(
    self,
    "tfc_aws_dynamic_credentials",
    type = cdktf.VariableType.object(
      {
        "default": cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING}),
        "aliases": (
          cdktf.VariableType.map(
            cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING})
          )
        )
      }
    )
  )
)

If you look closely this translation is still mostly straightforward, with the main difference being the use of cdktf classes to represent keywords in HCL like object, map, etc.

Next we will need to use the cdktf.Fn submodule to get execution-delayed accessors for the variable to be applied once it's defined at runtime.

For example, for the above providers in HCL above the syntax to access the shared_config_file attribute of the tfc_aws_dynamic_credentials variable is:

var.tfc_aws_dynamic_credentials.aliases["B"].shared_config_file

In Python we can't directly access the aliases attribute of this variable (including any of the remaining nested attributes) since the Python interpreter is only ever running during "synthesis" time, meaning the variable's contents are not populated for Python to manage. In other words, tfc_aws_dynamic_credentials.aliases will just yield an AttributeError.

Instead we need the cdktf.Fn.lookup function, or even better the cdktf.Fn.lookup_nested function which will delay the alias attribute lookup until runtime on the TFC runner. Using that function and adapting the syntax yields:

cdktf.Fn.lookup_nested(tfc_aws_dynamic_credentials.value,
                       ["aliases", "B", "shared_config_file"])

This syntax is a bit of a "line-full" so I chose to make a helper function which accepts the AWS provider alias as the only argument:

def get_aws_dynamic_config_file_from_variable(provider_alias: str):
  return cdktf.Fn.lookup_nested(tfc_aws_dynamic_credentials.value,
                                ["aliases", provider_alias, "shared_config_file"])

_{Note that the shared_config_files property of AWS providers expects a list object so you will need to emplace the return value of this function as a list element in the provider declaration.}

Putting this all together in one cdktf Stack, we get the following code:

import cdktf
from constructs import Construct
from cdktf_cdktf_provider_aws.provider import AwsProvider

class AppStack(cdktf.TerraformStack):
  def __init__(self, scope: Construct, stack_id: str):
    super().__init__(scope, stack_id)

    tfc_aws_dynamic_credentials = (
      cdktf.TerraformVariable(
        self,
        "tfc_aws_dynamic_credentials",
        type = cdktf.VariableType.object(
          {
            "default": cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING}),
            "aliases": (
              cdktf.VariableType.map(
                cdktf.VariableType.object({"shared_config_file": cdktf.VariableType.STRING})
              )
            )
          }
        )
      )
    )

    def get_aws_dynamic_config_file_from_variable(provider_alias: str):
      return cdktf.Fn.lookup_nested(tfc_aws_dynamic_credentials.value,
                                    ["aliases", provider_alias, "shared_config_file"])


    aws_account_A_id    = '012345678910'
    aws_account_A_alias = 'A'
    aws_account_A_shared_config_file = get_aws_dynamic_config_file_from_variable(aws_account_A_alias)

    aws_provider_A = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_A_alias}",
                                       alias               = aws_account_A_alias,
                                       allowed_account_ids = [aws_account_A_id],
                                       shared_config_files = [aws_account_A_shared_config_file])

    aws_account_B_id    = '109876543210'
    aws_account_B_alias = 'B'
    aws_account_B_shared_config_file = get_aws_dynamic_config_file_from_variable(aws_account_A_alias)

    aws_provider_B = cdktf.AwsProvider(self,
                                       f"aws-provider-{aws_account_B_alias}",
                                       alias               = aws_account_B_alias,
                                       allowed_account_ids = [aws_account_B_id],
                                       shared_config_files = [aws_account_B_shared_config_file])

    ...

And with this, you may now freely declare more infrastructure targeting either of these 2 AWS accounts specifically by simply adding the keyword argument provider = aws_provider_A or provider = aws_provider_B.

Happy Python-wrapped Terraforming! Feel free to post any questions below and I'll be happy to answer the best I can.

1. Hashicorp's announcement for multiple dynamic provider credentials configurations: https://www.hashicorp.com/blog/terraform-cloud-now-supports-multiple-configurations-for-dynamic-provider-credent ↩

2. Technically it's possible to delegate SSO administration to a different organizational account besides the root account (a.k.a. "management" account), making it possible to manage IAM Identity Center principals and role assignments from that account. However, this doesn't remove the requirement for deploying to multiple AWS accounts. ↩

3. In my AWS organization I have written a CloudFormation StackSet which automatically creates a role named TerraformCloudDynamicProviderCredentialsRole with sufficient permissions, along with an OIDC trust connection to our Terraform Cloud organization which grants assume-role access to that role for any new AWS organizational accounts we create. This means I can safely assume that this role already exists in these accounts and that Terraform Cloud can use OIDC to assume it for TFC runs, but you should take care to verify your AWS account(s) have this pre-existing infrastructure. ↩