Using 1Password with External Secrets Operator in a GitOps way

I recently created a free Kubernetes cluster on Oracles Always Free Tier. But how to handle secrets in a secure way, especially in a GitOps scenario?

This post will describe the full journey how to integrate the External Secrets Operator with 1Password to automatically inject secrets from 1Password to your cluster. For managing the different components, I will use argocd, but you can use also just helm install or FluxCD.

My Goal was to have a enterprise-known solution which integrates with different secret-managment tools, in my case it's 1Password.

External Secrets Operator

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.
-external-secrets.io

I like the fact that i can theoretically change my secret-management-tool without touching my secrets. Its also possible to have the same Secret-File in all my deployments over different stages. Because the secret-value gets filled regarding to the current stage. These are all things which were often a pain for me with sealed-secrets.

1Password

Really comfy Password-manager.
I have it so why not also using it for K8s?

Architecture

Let us have a look on the architecture especially the mechanics between external secret operator with Kubernetes and 1Password with the external secret operator.

External Secret Operator

The External Secret Operator uses in general two Custom Resources. The first is the SecretStore which holds information, how to connect to the "Secret Backend". The "Secret Backend" can be something like AWS Secrets Manager, Scaleway or like in our case 1Password. It holds for example the URL and the authentication for it. You can have different SecretStores if you have more than one provider. The other resource is the ExternalSecret which is a blueprint for the secret which should be created. So in practice if you create an ExternalSecret a normal K8s-Secret follows. The ExternalSecret holds information about which entry in 1Password to fetch and in which way to populate the values in a Secret. (More later)

1Password Connect

1Password-Connect is a kind-of Proxy which caches Secrets from the 1Password Cloud-Service, so you are able to retrieve secrets even 1Password is down, it also reduces the amount of requests to 1Password.
Its needed for the External Secrets Operator to work.
In my approach, it is right next to the External Secret Operator in the same namespace.

All together the workflow looks similar to that.

1. External Secret is created
2. External Secret Operator uses the information's in SecretStore to fetch the needed password/login from 1Password Connect
3. 1Password connect itself fetch the needed password/login from 1Password directly
4. External Secret Operator gets the password/login & creates the normal Kubernetes Secret with the requested values

Don't worry if you don't understand the whole process, we'll do a full hands-on walk-through at the end.

Install needed Helm-Charts

For the Installation of the helm-charts i will use argocd-apps, because i find this approach really portable. If i have a new cluster i just install argocd and then a kubectl apply . and i have all my needed apps installed. Beyond that i can use kustomize or helm and i have a auto-update feature of the apps - if i want to. Just set the targetRevision field to something like this 1.x.x.

But you can also use helm or fluxcd.

Install 1Password-Connect

Install the helm-chart

I modified the default values of the official Chart to the following:

# values.yaml
connect:
  serviceType: ClusterIP

The default value is NodePort which is IMO dangerous if you publish your Secret-management-api to the local network or even the Internet. ClusterIP is suffient because the External Secrets Operator is in the cluster - not outside.

To install the Chart with the predefined values you can just apply this argocd-app. (kubectl apply -f app-1password-connect.yaml)

# app-1password-connect.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: 1password-connect
  namespace: argocd 
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://1password.github.io/connect-helm-charts
    targetRevision: 1.x.x
    chart: connect
    helm:
      values: |
        connect:
          serviceType: ClusterIP
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

After that you should have a failed Pod in the external-secrets namespace.

$ kubectl get pods -n external-secrets
onepassword-connect-5fcbd4c68b-fgx68                         0/2     CreateContainerConfigError   0             4d20h

The reason is Error: secret "op-credentials" not found (kubectl get event -n external-secrets). This Secret contains the token which is used for the connect-server to connect to 1Password.

To create the Secret install the 1Password CLI and execute following commands.

# Create a new Vault for the Secrets
$ op vault create "K8s"
# Create a connect server in 1Password
op connect server create "Kubernetes" --vaults "K8s"

The location of the credentials file can be seen in the output.

With the 1password-credentials.json in place we can create the needed secret.

# The connect service expects the 1password-credentials.json in base64
$ kubectl create secret generic op-credentials -n external-secrets --from-literal=1password-credentials.json="$(cat /path/to/1password-credentials.json | base64)"

After a while the pod should be running

$ kubectl get pod -n external-secrets
onepassword-connect-5fcbd4c68b-chmfg                         2/2     Running   0
  21m

Congratulations you completed the first step - installed the 1Password-Connect Server in Kubernetes!

Install the External Secret Operator

For installation you can use again an argocd-app.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets-operator
  namespace: argocd 
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.x.x
    chart: external-secrets
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

After that your namespace should look like this.

$ kubectl get pods -n external-secrets
NAME                                                         READY   STATUS    RESTARTS      AGE
external-secrets-operator-5db95c68b8-j6pzj                   1/1     Running   1 (18d ago)   18d
external-secrets-operator-cert-controller-65f74dc7bf-784hg   1/1     Running   1 (18d ago)   18d
external-secrets-operator-webhook-5475bddd67-p9vq9           1/1     Running   0             18d
onepassword-connect-5fcbd4c68b-chmfg                         2/2     Running   0             31m

Then we can establish the connection to the connect server (höhö).

# Create the token and save it in the OP_CONNECT_TOKEN environment variable 
export OP_CONNECT_TOKEN=$(op connect token create "external-secret-operator" --server "Kubernetes" --vault "K8s")
# Create secret with the token which is used by the External-Secret-Operator ClusterSecretStore
kubectl create secret -n external-secrets generic onepassword-connect-token --from-literal=token=$OP_CONNECT_TOKEN

Create the ClusterSecretStore with kubectl apply -f

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: k8s
spec:
  provider:
    onepassword:
      connectHost: http://onepassword-connect:8080
      vaults:
        K8s: 1  # look in this vault first
      auth:
        secretRef:
          connectTokenSecretRef:
            name: onepassword-connect-token
            key: token
            namespace: external-secrets

To see if it the connection was successful you can check the status of the ClusterSecretStore.

kubectl get -n external-secrets clustersecretstores.external-secrets.io k8s
NAME   AGE    STATUS   CAPABILITIES   READY
k8s    103m   Valid    ReadOnly       True

Congratulations you also installed the external secret operator and established a connection with 1password connect.

End-to-End-Test the Solution

Theoretically you should be able to retrieve secrets from 1Password - but let us test it with a pragmatical example.

First let us create a secret in 1Password with the 1Password CLI.

# This creates a new entry named "Scaleway Credentails" with two properties/fields. 
op item create --vault="K8s" --title="Scaleway Credentials" --category="login" accessKeyId="token-xyz" secretKey="xyz"

In the next step we can create our ExternalSecret which then creates our normal Kubernetes Secret.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  # name of the ExternalSecret & Secret which gets created
  name: scaleway-credentials
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: dev
  target:
    creationPolicy: Owner
  data:
  - secretKey: accessKeyId
    remoteRef:
      # 1password-entry-name
      key: "Scaleway Credentials"
      # 1password-field
      property: accessKeyId
  - secretKey: secretKey
    remoteRef:
      # 1password-entry-name
      key: "Scaleway Credentials"
      # 1password-field
      property: secretKey

I tried my best to explain the references with comments.
To summarize the name of the ExternalSecret do not count. But its important that the secretKey is and the property field has the same value. The secretKey and property field represent the "Field" in 1Password e.g. password or user. The key field represents the entry in 1Password e.g. dev.to.

You can check the status of the external secret and also if the normal K8s secret was created.

$ kubectl get externalsecret -n default
NAME                   STORE   REFRESH INTERVAL   STATUS         READY
scaleway-credentials   k8s     1h                 SecretSynced   True
$kubectl get secret -n default
NAME                   TYPE     DATA   AGE
scaleway-credentials   Opaque   2      7m13s

You can see the secret is created. You can also have a deeper look into the values.

$ kubectl get secret -n default -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    accessKeyId: dG9rZW4teHl6
    secretKey: eHl6
  immutable: false
  kind: Secret
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"external-secrets.io/v1beta1","kind":"ExternalSecret","metadata":{"annotations":{},"name":"scaleway-credentials","namespace":"default"},"spec":{"data":[{"remoteRef":{"key":"Scaleway Credentials","property":"accessKeyId"},"secretKey":"accessKeyId"},{"remoteRef":{"key":"Scaleway Credentials","property":"secretKey"},"secretKey":"secretKey"}],"secretStoreRef":{"kind":"ClusterSecretStore","name":"k8s"},"target":{"creationPolicy":"Owner"}}}
      reconcile.external-secrets.io/data-hash: f3d79dd2ad3055723d0bebe3481f2372
    creationTimestamp: "2024-07-29T15:36:04Z"
    labels:
      reconcile.external-secrets.io/created-by: c66895ad2c04350c73c512c0175cee24
    name: scaleway-credentials
    namespace: default
    ownerReferences:
    - apiVersion: external-secrets.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: ExternalSecret
      name: scaleway-credentials
      uid: 1df5f0f5-ec44-4cb7-859a-3d3ef60b1d45
    resourceVersion: "689603"
    uid: 9543092a-7bdc-4695-9c83-bc35c36188c0
  type: Opaque
kind: List
metadata:
  resourceVersion: ""

Under data you can see the both keys and the corresponding values encoded in base64.

Congratulations you successfully created a secret in 1Password and pulled it into a normal Kubernetes Secret which can now be used as usual in deployments and other resources. 🥳🥳🥳

Thanks for reading my first post!
I hope you had fun and that it helped you. I'd love to hear your thoughts or feedback, so feel free to leave a comment below.