Serverless Mysteries with Secret Manager Libraries on Google Cloud

Secrets are private information used by an application. They're often obfuscated passphrases used to authenticate with APIs or connect to databases: not something that should ever be in your code! Until Google Cloud's Secret Manager, many tutorials suggested using secrets on platforms such as App Engine or Cloud Run with the magic of plaintext (in-the-clear) environment variables.

Unencrypted environment variables aren't very secret, but at least they don't need to be committed to your codebase. Using one to carry a secret is simple but it's not very secure:

YOU_KNOW_WHO="Lord Voldemort" npm start

Environment variables are a gossipy approach to injecting data, some examples of weak security include:

• developers type them into terminal history
• configuration manifests reveal them
• exfiltrating them from production is as easy as a stray bit of code like console.log(process.env) and access to view the logs

Secret Manager provides a better approach for managing your secrets:

1. Create a named secret
2. Grant your code access to it
3. Watch the records of everything that tries to access it

Secret manager works smoothly. My first experience was a bit magical (work with me, we're on a theme). And what secrets are there in magic? In myth and fiction, names are significant mysteries doubling as a magic spell or a homing beacon for evil ears. How many times in the Harry Potter by J.K. Rowling series did we hear the phrase "You-Know-Who" or "He-Who-Must-Not-Be-Named" as an alias for the big bad?

In this post we'll walk through creating a "Ministry of Magic Loyalty Checker" service on Cloud Run, using Secret Manager to supply the dark wizard's true name.

Prepare to follow along

If you'd like to follow along, you'll need a Google Cloud project, a working installation of the gcloud CLI, and enabled APIs for Cloud Run and Secret Manager.

Assuming you already have a project, the fastest way to get started is a quick 2-step:

• Open the Cloud Shell to get a VM workspace provisioned with all the tools you'll need.
• Enable the APIs with a CLI command:

gcloud services enable run.googleapis.com secretmanager.googleapis.com

Step 1: Keeping Secrets Close

Secret Manager is a Key-Value Store, one with encryption, versioning, access control, and audit logging around individual key-values. Each one is initialized before it can be assigned a value.

Create a Secret

Use gcloud to create a new secret. This is like a variable declaration: it's a placeholder for something to come.

gcloud secrets create you-know-who --replication-policy="automatic"

The replication policy flag instructs Google Cloud to manage storage locations of the key.

Create a Version

A version is the secret data itself: every secret value is going to change eventually. Versioning as a first-class concept makes rotation much easier.

• Open secret.txt in your favorite editor
• Type "Lord Voldemort"
• Save the file

This keeps the secret out of your terminal history.

gcloud secrets versions add "you-know-who" --data-file secret.txt
Created version [1] of the secret [you-know-who].

Step 2: Configure the Cloud Run service identity

Cloud Run allows you to specify the "service identity" of each service. This allows you to attach IAM permissions to your service to limit it's access to other Cloud resources. I prefer to create one service identity for each Cloud Run service so my services run with minimal privileges.

This feature ensures only the loyalty-check service is able to access the secret because only it (and Project Owners) have read permission.

Create a service account for the loyalty-checker service:

gcloud iam service-accounts create loyalty-identity

Grant the service account access to the you-know-who secret:

gcloud secrets add-iam-policy-binding you-know-who \
 --member serviceAccount:loyalty-identity@example-project-id.iam.gserviceaccount.com \
 --role roles/secretmanager.secretAccessor

Step 3: Implement the "Loyalty Check" service

The code of this service is mostly interesting as an example of how to use the Secret Manager client libraries. We'll walk through it step by step.

Create a new npm package

The only dependencies are Express v4 and the Secret Manager v3 client library.

npm init loyalty
npm i express@4 @google-cloud/secret-manager@3

Create a Dockerfile

A Dockerfile is used to define how to create a container image for deployment to Cloud Run.

This Dockerfile uses a Node.js 14 base image, copies in the package.json manifest to install npm dependencies, and adds the custom source code. When the container is started npm start is executed to run the service.

Create the application code

Both code blocks are placed in a single index.js file for use.

The code uses the Secret Manager client library to dig up the secret:

The code above does the following:

• Ensures the SECRET environment variable is present to name the secret.
• Defines a getSecret() function that uses global state to keep the Secret Manager client library in memory for reuse.
• Provides an escape valve if the code is running locally: just call the service with SECRET=dev npm start
• Logs an error if the API interaction fails but allows the caller to decide what to do.

Next define a web server that takes requests to perform a loyalty check. The secret is only retrieved when first requested. No need to have it decrypted in active memory until it's needed!

By storing the secret value in global state, changes to the secret value or access revocation will not affect this instance of the service. On Cloud Run, this means this value will be used by a given container instance until it scales down.

Who checks the loyalty checker?

Everyone is loyal to the Ministry of Magic unless the Loyalty Checker knows the name of You-Know-Who. Seems like this secret has turned the service into a traitor!

Maybe this could be made a little smarter: according to the quota reference Secret Manager supports payloads up to 64 KiB per key, so we could encode JSON as a string and stash a lookup table:

{
  "Harry Potter": "Ministry of Magic",
  "Voldemort": "Lord Voldemort",
  "Severus Snape": "unknown",
  "Lucius Malfoy": "Lord Vodemort"
}

This works for several characters, but quickly reaches a point where using a database is more sensible. The secret's role would change from directly holding the mystery to holding the database credentials to look up a trove of PII.

Step 4: Ship the Cloud Run service

Now that we're code complete, let's get the Loyalty Checker into production.

Build the container

This could be done with docker, but today we'll use Cloud Build. This step builds the service into a container image and pushes it to Google Container Registry. From there we can deploy to Cloud Run.

gcloud builds submit --tag gcr.io/$GOOGLE_CLOUD_PROJECT/loyalty

Deploy the service

This step is a bit more complicated than the typical Cloud Run deployment, because we need to specify the service account and configure the full name of the secret:

gcloud run deploy loyalty \
  --image gcr.io/$GOOGLE_CLOUD_PROJECT/loyalty \
  --update-env-vars SECRET_NAME=projects/$GOOGLE_CLOUD_PROJECT/secrets/you-know-who/versions/1 \
  --service-account loyalty-identity@adamross-svls-kibble.iam.gserviceaccount.com \
  --allow-unauthenticated \
  --platform managed \
  --region us-central1

The 1 at the end of the SECRET_NAME value specifies the version of the secret to use. When new versions are created, the number increments. You can double-check what the most recent enabled version is by running gcloud secrets versions list you-know-who.

Step 5: Try out the "Loyalty Checker"

Use curl to send an HTTP request to the URL of your Cloud Run service. You'll see that URL on screen after you deploy.

curl https://loyalty-[HASH]-uc.a.run.app/loyalty
You serve the Lord Voldemort!

We've created a new Cloud Run service which pulls essential configuration from Secret Manager. Access has been carefully managed to limit it to a single service account, which is only used by a single Cloud Run service. That's the traditional way to keep a secret: by not sharing it with anyone. Unfortunately once *this service* unlocks the true name it tells anyone who asks.

Down with Environment Variables!?

What about the "Boy Who Lived," is the name "Harry Potter" a secret too? No, it was a newspaper headline. If folks didn't know the name, he wouldn't have been their hero.

Don't overuse secrets: they're not only more expensive than environment variables, but the more you hide production configuration the more mysteries there are when it's time to troubleshoot production failures.

Next Steps

I'm looking forward to how Secret Manager helps improve security practices on Google Cloud. What will your first secret be?

Learn how to leave the libraries behind with Cloud Run's built-in Secret Manager integration, perhaps by reading the follow-up blog post?

Generalize your product knowledge by reading the Secret Manager documentation.

All code © Google w/ Apache 2 license