DEV Community

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

AWS: Kubernetes and External Secrets Operator for AWS Secrets Manager

#aws #devops #security #kubernetes

We have a new EKS cluster 1.30 on our project, where we want to completely remove the old IRSA with OIDC and start using EKS Pod Identities — see AWS: EKS Pod Identities — a replacement for IRSA? Simplifying IAM access management.

And everything seems to work fine, but when I started deploying our Backend API, its Pods did not start and hung in the ContainerCreating status.

The problem: Kubernetes Secrets Store CSI Driver and “An IAM role must be associated with service account”

The backend secrets are stored in the AWS Secrets Manager, from where they are synchronized to Kubernetes Secrets with the Kubernetes Secrets Store CSI Driver, and then Backend API Pods create their environment variables using these Kubernetes Secrets. See AWS: Kubernetes — Integrating AWS Secrets Manager and Parameter Store.

When checking the status of a Pod, the is wonderful message ”An IAM role must be associated with service account ”:

Warning FailedMount 43s (x2 over 2m45s) kubelet MountVolume.SetUp failed for volume "backend-api-secret-class-volume" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod dev-backend-api-ns/backend-api-deployment-65c559d47-bb4dz, err: rpc error: code = Unknown desc = us-east-1: An IAM role must be associated with service account backend-api-sa (namespace: dev-backend-api-ns)
Enter fullscreen mode Exit fullscreen mode

Google leads us to the GitHub issue Pod Identity Association not recognized by secrets store CSI driver, which was opened in December 2023.

It also has a pool-request Add support for Pod Identity Association, which is supposed to fix this issue — but it’s still in the Open status, although a few days ago they added a comment that “We are conducting an initial investigation on this feature request and will share updates soon”.

And the driver is now the latest version — 0.3.9:

$ helm list -n kube-system | grep secret
secrets-store-csi-driver kube-system 1 2024-07-18 12:46:20.945937022 +0300 EEST deployed secrets-store-csi-driver-1.4.4 1.4.4      
secrets-store-csi-driver-provider-aws kube-system 1 2024-07-18 12:46:23.287242734 +0300 EEST deployed secrets-store-csi-driver-provider-aws-0.3.9
Enter fullscreen mode Exit fullscreen mode

So what to do?

The first option is to add OIDC and the old scheme again. But I really don’t want to do this, because I would like to have a completely new authentication on the new Kubernetes cluster, and not to make crutches that will need to be cut out later.

The second option is to try to switch from the Kubernetes Secrets Store CSI Driver to the External Secrets Operator, which can work with a bunch of different providers — AWS Secrets Manager, Hashicorp Vault, Google Secrets Manager, etc.

In addition, I don’t really like the fact that to create a Kubernetes Secret using the Kubernetes Secrets Store CSI Driver, you need to separately describe objects in its SecretProviderClass, and then actually duplicate them in secretObjects.

External Secrets Operator: an overview

So, the External Secrets Operator (ESO) is able to retrieve secrets from external resources and create regular Kubernetes Secrets.

To access AWS, it uses a standard scheme with ServiceAccount, which means we can create an EKS Pod Identity Association on the AWS IAM Role that will provide access to AWS Secrets Manager.

External Secrets Operator uses two main resources:

  • SecretStore: describes how to access the secrets - which provider (AWS, Google, Vault, etc.) and authentication, and is created at the level of a separate Kubernetes Namespace for access distribution
  • there is also a ClusterSecretStore that can be created globally and is accessible from any Namespace
  • ExternalSecret: describes which data to receive from the provider and, if necessary, changes to make

External Secrets Operator has a whole bunch of external providers — see Provider.

In AWS, it can work with both the SecretsManager itself and the ParameterStore.

In addition, the External Secrets Operator can even make changes in the AWS Secrets Manager — but we will only use it to create Kubernetes Secrets, because the secrets in the AWS Secrets Manager are created from each project’s Terraform.

So, our task is:

  • install External Secrets Operator
  • configure it to access AWS with AWS IAM Role and Kubernetes ServiceAccount using EKS Pod Identities
  • and create a Kubernetes Secret that we can connect to the Kubernetes Pod to set the necessary environment variables for our service

Installing External Secrets Operator with Helm

Add a repository:

$ helm repo add external-secrets https://charts.external-secrets.io
"external-secrets" has been added to your repositories
Enter fullscreen mode Exit fullscreen mode

The Helm chart itself and the available values are here — external-secrets.

Install the chart with the operator in the ops-external-secrets-ns Namespace:

$ helm install -n ops-external-secrets-ns --create-namespace external-secrets external-secrets/external-secrets
Enter fullscreen mode Exit fullscreen mode

Check the Pods:

$ kk -n ops-external-secrets-ns get pod
NAME READY STATUS RESTARTS AGE
external-secrets-5859d8dc69-vxhjb 1/1 Running 0 33s
external-secrets-cert-controller-5bbb8c4bb8-nmjn9 1/1 Running 0 33s
external-secrets-webhook-564cd5b69-r5mmb 1/1 Running 0 33s
Enter fullscreen mode Exit fullscreen mode

And ServiceAccounts:

$ kk -n ops-external-secrets-ns get sa 
NAME SECRETS AGE
default 0 10m
external-secrets 0 9m59s
external-secrets-cert-controller 0 9m59s
external-secrets-webhook 0 9m59s
Enter fullscreen mode Exit fullscreen mode

Here we are interested in the ServiceAccount external-secrets - the operator will use it to access AWS Secrets Manager and Parameter Store, and we will connect it to an EKS cluster with a Pod Identity.

Authentication with AWS IAM

What do we need:

  • an IAM Policy with permissions to the Secrets Manager and Parameter Store
  • an IAM Role with a Trust Policy for the EKS Pod Identity
  • connect IAM Policy to this role

Then the External Secrets Operator Pod will assume this Role through the Kubernetes ServiceAccount and will get access to the secrets.

For now, we’ll stick to the simplest scheme, and then we’ll see how you can further separate access through separate IAM Roles for each SecretStore in different Namespaces.

Creating an IAM Policy

Go to the AWS IAM, create a new IAM Policy, allow only read operations on Secrets Manager and Parameter Store resources:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAccessToSecretsManager",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:<AWS_REGION>:<AWS_ACCOUNT_ID>:secret:*"
            ]
        },
        {
            "Sid": "AllowAccessToParameterStore",
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:GetParametersByPath"
            ],
            "Resource": [
                "arn:aws:ssm:<AWS_REGION>:<AWS_ACCOUNT_ID>:parameter/*"
            ]
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Save it with the external-secrets-operator-test-policy name:

Creating an IAM Role for EKS Pod Identity

Go to the IAM Roles, create a new Role, select EKS — Pod Identity in the Use case:

Connect the IAM Policy external-secrets-operator-test-policy created above:

Save the new Role as external-secrets-operator-test-role, and in the Trust policy we have "Service": "pods.eks.amazonaws.com":

Creating an EKS Pod Identity Association

Now connect this Role to our EKS cluster atlas-eks-ops-1–30-cluster in the ops-external-secrets-ns Namespace to the Kubernetes ServiceAccount named external-secrets:

$ aws --profile work eks create-pod-identity-association --cluster-name atlas-eks-ops-1-30-cluster \
> --role-arn arn:aws:iam::492***148:role/external-secrets-operator-test-role \
> --namespace ops-external-secrets-ns \
> --service-account external-secrets
Enter fullscreen mode Exit fullscreen mode

Check in the EKS > Access:

Okay — everything is ready here. Now the ESO may access the Secrets and Parameters.

Next, we need a SecretStore, which will describe how to access the AWS Secrets Manager or Parameter Store, and an ExternalSecret, which will actually be responsible for Kubernetes Secrets.

Creating a Kubernetes Secrets from AWS Secrets Manager

Documentation for all values for Operator resources is in the API specification.

Creating a SecretStore

We will test in a separate Namespace ops-test-ns.

Let’s write a manifest for the SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: test-secret-store
  namespace: ops-test-ns
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
Enter fullscreen mode Exit fullscreen mode

Create the resource:

$ kk apply -f test-secretstore.yml 
secretstore.external-secrets.io/test-secret-store created
Enter fullscreen mode Exit fullscreen mode

Check it and its STATUS:

$ kk get secretstore
NAME AGE STATUS CAPABILITIES READY
test-secret-store 29s Valid ReadWrite True
Enter fullscreen mode Exit fullscreen mode

Creating an ExternalSecret

Now we need an ExternalSecret that will use the SecretStore created above.

Let’s create a test secret in AWS Secrets Manager:

Save it as test-aws-secret:

And we have a secret with a string {"secret_key_1":"secret_value_1","secret_key_2":"secret_value_2"}:

First, let’s get the entire string and then see how it can be added to a Kubernetes Secret.

Define a manifest for the ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: test-external-secret
  namespace: ops-test-ns
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: test-secret-store
    kind: SecretStore
  target:
    name: test-kubernetes-secret
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      metadata:
        labels:
          app: test
  data:
    - secretKey: key_in_the_kubernetes_secret
      remoteRef:
        # AWS Secrets Manager secret's name
        key: test-aws-secret
Enter fullscreen mode Exit fullscreen mode

Here in the spec:

  • refreshInterval: How often to check for changes in AWS Secrets Manager
  • secretStoreRef: Which SecretStore to use for access to AWS Secrets Manager (can be specified separately in data.sourceRef.storeRef)
  • target:
  • name: a Kubernetes Secret name to be created
  • creationPolicy: "owner" of Kubernetes Secret:
  • Owner: deletes the Kubernetes Secret if the corresponding ExternalSecret is deleted
  • Merge: does not create a Kubernetes Secret, but changes entries in an existing one
  • Orphan: leaves the Kubernetes Secret if the corresponding ExternalSecret is deleted
  • deletionPolicy:
  • Retain: leaves Kubernetes Secret if all fields are deleted in the corresponding AWS Secrets Manager Secret
  • Delete: deletes the Kubernetes Secret if all fields are deleted in the corresponding AWS Secrets Manager Secret
  • Merge: deletes all entries in the Kubernetes Secret if all fields are deleted in the corresponding AWS Secrets Manager Secret, but leaves the Kubernetes Secret itself
  • template: describes the structure of the Kubernetes Secret to be created - its type, labels, annotations, etc.
  • data: describes the relationship between a secret in AWS Secrets Manager and a Kubernetes Secret:
  • secretKey: the key name in the Kubernetes Secret
  • remoteRef:
  • key: a name of a secret in AWS Secrets Manager

Create the resource:

$ kk apply -f test-externalsecret.yml 
externalsecret.external-secrets.io/test-external-secret created
Enter fullscreen mode Exit fullscreen mode

Check its status:

$ kk get externalsecret
NAME STORE REFRESH INTERVAL STATUS READY
test-external-secret test-secret-store 1h SecretSynced True
Enter fullscreen mode Exit fullscreen mode

STATUS == SecretSynced - that is, the External Secrets Operator was able to get the value from AWS Secrets Manager and create a Kubernetes Secret.

In case of problems, you can look at ESO logs:

$ ktail -n ops-external-secrets-ns -l app.kubernetes.io/instance=external-secrets
Enter fullscreen mode Exit fullscreen mode

Check the Kubernetes Secret itself:

$ kk get secret test-kubernetes-secret -o yaml
apiVersion: v1
data:
  key_in_the_kubernetes_secret: eyJzZWNyZXRfa2V5XzEiOiJzZWNyZXRfdmFsdWVfMSIsInNlY3JldF9rZXlfMiI6InNlY3JldF92YWx1ZV8yIn0=
...
Enter fullscreen mode Exit fullscreen mode

And the “eyJzZWNyZXRfa2V5XzEiOiJzZWNyZXRfdmFsdWVfMSIsInNlY3JldF9rZXlfMiI6InNlY3JldF92YWx1ZV8yIn0=” string gives us the whole secret from AWS Secrets Manager:

$ echo eyJzZWNyZXRfa2V5XzEiOiJzZWNyZXRfdmFsdWVfMSIsInNlY3JldF9rZXlfMiI6InNlY3JldF92YWx1ZV8yIn0= | base64 -d
{"secret_key_1":"secret_value_1","secret_key_2":"secret_value_2"}
Enter fullscreen mode Exit fullscreen mode

But in this form, it’s not very useful to us, so we can do it in another way with the property parameter to the data, in which we specify a specific key from the secret:

data:
  - secretKey: secret_key_1_value
    remoteRef:
      # AWS Secrets Manager secret's name
      key: test-aws-secret
      property: secret_key_1
Enter fullscreen mode Exit fullscreen mode

Then our Kubernetes Secret will look like this:

$ kk get secret test-kubernetes-secret -o yaml
apiVersion: v1
data:
  secret_key_1_value: c2VjcmV0X3ZhbHVlXzE=
...
Enter fullscreen mode Exit fullscreen mode

Where “c2VjcmV0X3ZhbHVlXzE=” is the value of_”secret_value_1_".

Or, instead of describing each key from AWS Secrets Manager, we can use spec.dataFrom instead of spec.data in our ExternalSecret:

...
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: test-secret-store
    kind: SecretStore
  target:
    name: test-kubernetes-secret
    creationPolicy: Owner
  ...     
  dataFrom:
  - extract:
      key: test-aws-secret
...
Enter fullscreen mode Exit fullscreen mode

And then our Kubernetes Secret will look like this:

$ kk get secret test-kubernetes-secret -o yaml
apiVersion: v1
data:
  secret_key_1: c2VjcmV0X3ZhbHVlXzE=
  secret_key_2: c2VjcmV0X3ZhbHVlXzI=
...
Enter fullscreen mode Exit fullscreen mode

Where “c2VjcmV0X3ZhbHVlXzE=” is the value of_”secret_value_1", and _“c2VjcmV0X3ZhbHVlXzI=” is the value of “secret_value_2”.

Or we can rename the keys, for example:

...
  dataFrom:
  - extract:
      key: test-aws-secret
    rewrite:
    - regexp:
        source: "secret_key_([0-9])"
        target: "SECRET_KEY_${1}"
Enter fullscreen mode Exit fullscreen mode

And the result will be:

$ kk get secret test-kubernetes-secret -o yaml
apiVersion: v1
data:
  SECRET_KEY_1: c2VjcmV0X3ZhbHVlXzE=
  SECRET_KEY_2: c2VjcmV0X3ZhbHVlXzI=
...
Enter fullscreen mode Exit fullscreen mode

Advanced IAM Permissions per SecretStore

Now let’s see how we can use an IAM Role with a separate IAM Policy at the SecretStore level.

That is, instead of having a single IAM Role with an IAM Policy that provides access to all the secrets in AWS Secrets Manager, and which is used by our External Secrets Operator, we can create a separate IAM Role, connect it to a specific SecretStore in a specific Kubernetes Namespace, and then this SecretStore will have access only to those secrets that are described in the corresponding IAM Policy.

Schematically, this can be represented as follows:

So, what we will do:

  • in the external-secrets-operator-test-role, which is connected to the external-secrets ServiceAccount via EKS Pod Identity, disable the IAM Policy that provides access to all AWS Secrets
  • create a new IAM Policy external-secrets-operator-test-application-policy, which will grant permission only to a specific secret
  • create a new IAM Role external-secrets-operator-test-application-role:
  • in the Trust Policy, allow it to perform Assume on behalf of the external-secrets-operator-test-role role and
  • connect the IAM Policy external-secrets-operator-test-application-policy to this role
  • and add a parameter to the SecretStore with the role: external-secrets-operator-test-application-role

Then External Secrets will work like this:

  • the Kubernetes Pod with External Secrets via EKS Pod Identity executes AssumeRole external-secrets-operator-test-role
  • when creating an ExternalSecret, it will use a SecretStore, in which the external-secrets-operator-test-application-role is set
  • the ExternalSecret with the external-secrets-operator-test-role will perform the second AssumeRole — “assume” the role of external-secrets-operator-test-application-role with its IAM Policy external-secrets-operator-test-application-policy
  • and with this role, it will get access to the secret in AWS Secrets Manager

Let’s go.

In the IAM Role external-secrets-operator-test-role, remove the connected IAM Policy that gave full access to AWS Secrets Manager:

Create a new IAM Policy external-secrets-operator-test-application-policy with access to one specific secret test-aws-secret:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAccessToSecretsManager",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-1:492***148:secret:test-aws-secret*"
            ]
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Save it:

Create a new IAM Role with the Custom trust policy, where we allow to Assume from the external-secrets-operator-test-role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Action": [
          "sts:AssumeRole",
           "sts:TagSession"
         ],
            "Principal": {
                "AWS": "arn:aws:iam::492***148:role/external-secrets-operator-test-role"
            }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Connect a new Policy to this Role:

And save the new Role as external-secrets-operator-test-application-role:

Now let’s return to our SecretStore and add the role parameter to the spec.provider.aws:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: test-secret-store
  namespace: ops-test-ns
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      role: arn:aws:iam::492***148:role/external-secrets-operator-test-application-role
Enter fullscreen mode Exit fullscreen mode

Update the SecretStore :

$ kk apply -f test-secretstore.yml 
secretstore.external-secrets.io/test-secret-store configured
Enter fullscreen mode Exit fullscreen mode

To check, let’s delete the old one ExternalSecret:

$ kk delete externalsecret test-external-secret
externalsecret.external-secrets.io "test-external-secret" deleted
Enter fullscreen mode Exit fullscreen mode

Create it once again:

$ kk apply -f test-externalsecret.yml 
externalsecret.external-secrets.io/test-external-secret created
Enter fullscreen mode Exit fullscreen mode

And check its status:

$ kk get externalsecret
NAME STORE REFRESH INTERVAL STATUS READY
test-external-secret test-secret-store 1h SecretSynced True
Enter fullscreen mode Exit fullscreen mode

Okay, that works.

Now let’s try to use another secret from AWS Secrets Manager -”test/rds/kraken”, to which we did not grant permission in the IAM Policy external-secrets-operator-test-application-policy:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: test-external-secret
  namespace: ops-test-ns
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: test-secret-store
    kind: SecretStore
  target:
    name: test-kubernetes-secret
  ...
  dataFrom:
  - extract:
      key: test/rds/kraken
Enter fullscreen mode Exit fullscreen mode

Deploy, check:

$ kk get externalsecret
NAME STORE REFRESH INTERVAL STATUS READY
test-external-secret test-secret-store 1h SecretSyncedError False
Enter fullscreen mode Exit fullscreen mode

And now, it has STATUS == SecretSyncedError.

This also can be see in logs — the “ external-secrets-operator-test-application-role is not authorized to perform: secretsmanager:GetSecretValue on resource: test/rds/kraken ” record:

external-secrets-5859d8dc69-2fgc8:external-secrets {... "msg":"could not get secret data from provider","ExternalSecret":{"name":"test-external-secret","namespace":"ops-test-ns"},"error":"AccessDeniedException: User: arn:aws:sts::492***148:assumed-role/external-secrets-operator-test-application-role/1724248118674412261 is not authorized to perform: secretsmanager:GetSecretValue on resource: test/rds/kraken because no identity-based policy allows the secretsmanager:GetSecretValue action\n\tstatus code: 400 ...}
Enter fullscreen mode Exit fullscreen mode

Conclusions

So far, I’ve really liked External Secrets Operator.

First of all, there is really much less manifest code to create resources.

Second, it works well with EKS Pod Identity.

Third, it is a very flexible system for distributing access to secrets.

The fourth is that you can create Kubernetes Secrets from different providers with a single operator.

And in general, it’s a simpler system, because you don’t need to have a DaemonSet with views on each WorkerNode, as implemented in secrets-store-csi-driver-provider-aws, and you don't need to mount Volumes to Pods to create Kubernetes Secrets.

It looks really cool, so we will migrate to it.

Originally published at RTFM: Linux, DevOps, and system administration.

Top comments (0)

Read next

xetnopnawab profile image

🚀 Week 1 Recap: Learning in Public – Software Engineering with DevOps 🚀

Md Nawab ✔️ -

er_dward profile image

Karpenter to EKS Auto Mode, worth it?

Internet Explorer -

ruivalim profile image

5 trends to watch in the World of DevOps in 2025

Rui Valim -

imsushant12 profile image

Conclusion of My Node.js Journey and a Sneak Peek into My Upcoming AWS Series

Sushant Gaurav -