Migrating from Secret Manager API to built-in secrets

Secret Manager is directly integrated into a number of products, including Cloud Run and Cloud Build.

Many samples I've authored use the Secret Manager API by calling the API from within the code. This can be more secure, but variable leakage is something that needs to be considered in any deployment. Removing the direct API calls can help with portability, and reduce the amount of dependencies in the deployment, and the complexity of the code itself.

So let's look at how we can migrate to built-in secrets.

Integrating Secret Manager API

Screenshot: "Secret value: Input your secret value or import it directly from a file.", showing a file upload dialog, or a text area input field.

When you create secrets, you can create them from a file or from direct input. When you retrieve these secrets, you may presume they're a single value, or a file of content you have to process.

Once you create the secret, you can access it with the Secret Manager API using a client library for Secret Manager available in many different languages.

This post will use Python examples and packages, but the same patterns can be applied to other languages.

For example, this code snippet will retrieve the latest version of the secret mySecret in the current project:

import google.auth
from google.cloud import secretmanager as sm

# trick to detect current project
_, project = google.auth.default() 
client = sm.SecretManagerServiceClient()

secret = "mySecret"

name = f"projects/{project}/secrets/{secret}/versions/latest"
secret_value = client.access_secret_version(path).payload.data.decode("UTF-8")

The secret_value can then be used as required. This presumes that mySecret is a single value, for example an API key.

One way developers can include multiple secrets in one format is with .env files. These files contain one or more key/value pairs which mean you can store multiple secrets in one file, then use helper packages to parse these values into your code. This prevents you from having to define multiple separate secret values.

If mySecret was a .env file, I could load it using python-dotenv:

## pip install python-dotenv
import io
from dotenv import load_dotenv

load_dotenv(stream=io.StringIO(secret_value))

Or I could to similar with django-environ, for example:

## pip install django_environ
import io
import environ
env = environ.Env()
env.read_env(io.StringIO(secret_value))

In both of these examples, these packages presume that the values are in a .env file within the same directory as the running process. Using StringIO means that a string variable can be sent to these methods in a file-like object, so the values can be read as though they were a file.

When using the Secret Manager API, you can run the application in any place where Google Cloud authentication works. That is, any Google Cloud service, or your local machine if you have set up gcloud.

Consuming built-in secrets

If you have your secret configured as an environment variable, then you only have to retrieve it as an environment variable:

import os
secret_value = os.environ.get("MYSECRET", None)

This method prevents KeyErrors if the key is not found.

For products with built-in secrets, they may be made available in additional ways.

In Cloud Build you can connect secrets directly to environment variables. But Cloud Run additionally allows mounting the secret as a volume, which means it can be read as a file. Any time the file is read, if you specify latest, the current latest will be received! However, you must mount the file in a new volume so you can't mount it directly in the default .env.

If you want to adapt your code to handle a local .env file, or a separately mounted file, or an environment variable, you will have to ensure all methods are possible in your code.

You also need to consider which configuration takes priority, as by default both python-dotenv and django-environ accept the first declared value as the value they use. You can override this by using the --override or --overwrite respectively.

When developing applications, I might choose to say my local .env file takes priority, then any mounted secrets, and then any declared variables.

Another point to mention is that using python-dotenv, if a file is not found or a value is empty, it silently continues. This means you can include the method calls without having to explicitly handle errors:

import io
import os

from dotenv import load_dotenv

load_dotenv()
load_dotenv("/secrets/.env")
load_dotenv(stream=io.StringIO(os.environ.get("MYSECRET", None)))

The same code works for django-environ, where you can just import in the order of priority without having to worry about missing files.

import io
import os

import environ
env = environ.Env()

env.read_env()
env.read_env("/secrets/.env")
env.read_env(io.StringIO(os.environ.get("MYSECRET", None)))

Note that in these examples, I'm choosing /secrets/as my volume, and keeping the path the same name as the original file. You can choose any volume and path, as long as the volume is not already used by the application (for example, if you choose /app/ as your working directory, you cannot mount secrets there.)

Deploying built-in secrets

To run this code locally, you'd create a .env file with the contents MYSECRET=serkitValue.

If you're committing this code to git, ensure you're not committing the secret file! Make sure you add .env to your .gitignore file!

You can also choose to ignore any contents of your .gitignore file in your Google Cloud commands by adding .gitignore's contents to .gcloudignore:

.git
#!include:.gitignore

You can then create the secret from this file with gcloud:

gcloud secrets create mySecret --data-file .env

For Cloud Build, you will need to ensure the secret is available in the environment (which your script can then use):

steps: 
 - name: python:slim
   entrypoint: pip
   args: ['install', '-r', 'requirements.txt', '--user']

 - name: python:slim
   secretEnv: ['MYSECRET']
   entrypoint: python
   args: ['main.py']

availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/mySecret/versions/latest
    env: 'MYSECRET'

You can also reference secrets in the args call itself, using bash variables.

For Cloud Run, you'll have to deploy the service specifying either a mount or an environment variable:

# for mounted volume
gcloud run deploy myservice --source .  \
  --update-secrets /secrets/.env=mySecret:latest

# for environment variable
gcloud run deploy myservice --source .  \
  --update-secrets MYSECRET=mySecret:latest

And never forget IAM!

Don't forget: you'll also need to make sure the service account you're using has permissions to access your secret!

Katie is a Developer Advocate for Google Cloud, focusing on Python and Serverless. She tweets @glasnt about clouds, crafts, and cats.