Terraform Ephemeral Resources

Storing sensitive data like database passwords and API keys in Terraform state files poses significant security risks. Terraform 1.10 introduced the ephemeral resource to address this issue, allowing dynamic retrieval of sensitive values during runtime without persisting them in the state file.

Resources

Ephemeral resources are defined using a new block type in HCL:

ephemeral "<resource type>" "<resource name>" {
  # attributes and nested blocks
}

These resources are referenced similarly to data sources but are not stored in the state file. Attributes and nested blocks will vary per provider implementation; however, they are referenced similarly to data sources.

Example: Using AWS Secrets Manager with Postgres

In this example, the database credentials are retrieved from AWS Secrets Manager and used to configure the PostgreSQL provider without being stored in the state file.

data "aws_secretsmanager_secret" "example" {
    name = "myprodsecret"
}

ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = data.aws_secretsmanager_secret.example.arn
}

locals {
  credentials = jsondecode(ephemeral.aws_secretsmanager_secret_version.db.secret_string)
}

provider "postgresql" {
  host     = "1.2.3.4"
  port     = "5432"
  username = local.credentials["username"]
  password = local.credentials["password"]
}

data "postgresql_tables" "my_tables" {
  database = "postgres"
}

output "tables" {
  value = data.postgresql_tables.my_tables.tables.*.object_name
}

If we examine the items in the state, there is no reference to aws_secretsmanager_secret_version.

# terraform state list
data.aws_secretsmanager_secret.example
data.postgresql_tables.my_tables

Supported Providers

At the time of writing, the current supported ephemeral blocks are:

• kubernetes_certificate_signing_request
• kubernetes_token_request
• aws_secretsmanager_secret_version
• aws_lambda_invocation
• aws_kms_secrets
• azurerm_key_vault_secret
• azurerm_key_vault_certificate
• google_service_account_access_token
• google_service_account_id_token
• google_service_account_jwt
• google_service_account_key

Variables and Outputs

Variables can be declared as ephemeral with ephemeral = true.

variable "api_key" {
  type        = string
  ephemeral   = true
  description = "an api key not stored in state"
}

Outputs in non-root modules can also be declared as ephemeral:

Example: Using Ephemeral Outputs in Modules

output "database_secret" {
  ephemeral = true
  value     = aws_secretsmanager_secret_version.db.secret_string
}

module "database" {
  source = "./database"
}

locals {
  credentials = jsondecode(module.database.database_secret)
}

provider "postgresql" {
  host     = data.aws_db_instance.example.address
  port     = data.aws_db_instance.example.port
  username = local.credentials["username"]
  password = local.credentials["password"]
}

Looking at the state again, there is still no reference to aws_secretsmanager_secret_version:

# terraform state list
data.postgresql_tables.my_tables
module.database.data.aws_secretsmanager_secret.example

Benefits of Ephemeral Resources

• Enhanced Security: Sensitive data is not stored in state files, reducing the risk of unauthorized access.
• Dynamic Retrieval: Values are fetched at runtime, ensuring up-to-date information.
• Compliance: Helps meet regulatory requirements by keeping secrets out of long-term storage.
• Simplified Secret Management: Integrates with secret management solutions like AWS Secrets Manager.

Terraform's ephemeral feature significantly improves security posture by keeping sensitive data out of state files. By leveraging this feature, teams can maintain the benefits of infrastructure as code while adhering to security best practices. For more detailed information, refer to the Hashicorp documentation on ephemeral resources.