DEV Community

Using Secrets in Google Cloud Functions

Dustin Ingram on April 25, 2019

Google Cloud Functions makes it easy to build serverless Python programs. This post will show you how you can use the Google Secret Manager to safe...

Read full post

smurfolan • Feb 28 '20

Thank you for the well structured content.

I faced the same issues related to unit tests as Sergio Sánchez Ramírez but I am also getting the value of 'None' at this line:

project_id = os.environ["GCP_PROJECT"]

I am not really sure what's the reason.

Dustin Ingram • Feb 28 '20

Are you saying you're getting this in your unit test? You'll need to monkeypatch project_id as well, as GCP_PROJECT won't exist as an environment variable in your testing environment.

smurfolan • Feb 29 '20

No, this is in the context of Google Cloud Build. I have a trigger for my master branch and once a new commit is pushed a build gets fired. I just did:

project_id = os.environ["GCP_PROJECT"]
print(project_id)

to see what is the result. And, it returns 'None' in the cloud execution environment.

Dustin Ingram • Feb 29 '20

Hmm, not sure I totally understand your setup. Why is your build executing your function?

If you could include any more details like your Cloud Build configuration I might be able to help.

Otherwise, an alternative would be checking whether this variable is set or not and not continuing if it is:

project_id = os.environ.get("GCP_PROJECT")

# If project_id exists, this is being executed on Cloud Functions
# Otherwise, it's being executed somewhere else and the following
# will fail.
if project_id:
    client = secretmanager.SecretManagerServiceClient()
    secret_name = "my-secret"
    resource_name = f"projects/{project_id}/secrets/{secret_name}/versions/latest"
    response = client.access_secret_version(resource_name)
    secret_string = response.payload.data.decode('UTF-8')

smurfolan • Feb 29 '20

I am using Google's Cloud Build (cloud.google.com/cloud-build) as a CI/CD tool for my cloud function written in Python (cloud.google.com/functions/docs/ca...)

So, I have a trigger defined on Cloud Build linked to my master branch. The .yaml file in my project looks like this: dev-to-uploads.s3.amazonaws.com/i/...

And, when executed, the build step creates a new container and I am not sure if inside of it:

os.environ.get("GCP_PROJECT")

is relevant.

Dustin Ingram • Feb 29 '20

Which step is failing here, the test step or the deploy step?

Can you include the test that is testing the function in question?

smurfolan • Mar 1 '20 • Edited

The step which fails is the one which executes pytest: dev-to-uploads.s3.amazonaws.com/i/...

I also tested without this step and it gets successfully deployed and the project id is available in a "production" situation. Maybe I will have to mock/stub the code for my tests.

Dustin Ingram • Mar 1 '20

Yes, so this is happening in your tests? You'll need to monkeypatch project_id like I mentioned in my original reply.

If you can include the test that's failing I can try to show you how to do that.

Also it's a lot easier for me to help if you share actual text and not screenshots!

smurfolan • Mar 2 '20

My main.py looks like this:

import os
from google.cloud import secretmanager
import logging

client = secretmanager.SecretManagerServiceClient()
secret_name = "my-secret"
project_id = os.environ.get('GCP_PROJECT')
resource_name = "projects/{}/secrets/{}/versions/latest".format(project_id, secret_name)
response = client.access_secret_version(resource_name)
secret_string = response.payload.data.decode('UTF-8')

def new_measures_handler(data, context):
    """Background Cloud Function to be triggered by Cloud Storage.
    Args:
         event (dict): The dictionary with data specific to this type of event.
         context (google.cloud.functions.Context): The Cloud Functions
         event metadata.
    """
    logging.info(secret_string)
    print('File: {}.'.format(data['name']))

and if I deploy it like this on GCP it works as expected. Google Cloud Build builds the function and deploys it. Project and respectively project secret can be accessed. But, when I uncomment my test step in .yaml and it gets executed on Google Cloud Build

steps:
- name: 'docker.io/library/python'
  args: ['pip3','install', '-r', 'requirements.txt', '--user']
#- name: 'docker.io/library/python'
#  args: ['python3','/builder/home/.local/bin/pytest', '.']
- name: 'gcr.io/cloud-builders/gcloud'
  args: ['functions', 'deploy', 'new_measures_handler', '--runtime', 'python37', '--trigger-resource', 'gcp-etl-prod-bucket', '--trigger-event', 'google.storage.object.finalize']

I start getting the error. As you say, I need to mock it somehow. This is how my current test looks like:


def test_print(capsys):
    # arrange
    name = 'test'
    data = {'name': name}

    # act
    main.new_measures_handler(data, None)
    out, err = capsys.readouterr()

    #assert
    assert out == 'File: {}.\n'.format(name)

Dustin Ingram • Mar 2 '20

OK, so your test should monkeypatch the environment like this:

def test_print(capsys, monkeypatch):
    monkeypatch.setenv('GCP_PROJECT', 'some-project-id')
    ....

You'll probably need to monkeypatch secretmanager.SecretManagerServiceClient as well.

Alejandro Barone • May 3 '19 • Edited

Hi Dustin, thanks for the post, is awesome!. Just a little fix. On the cloud shell command it needs to be pip3 as the python3 version that is used afterwards.

And I think the resource_name doesnt need the plus sign at the end

Thanks again!
Cheers.

Dustin Ingram • May 3 '19

Thank you, nice catch!

Sergio Sánchez Ramírez • Feb 12 '20

Thanks for the post! It has been very useful!

The only problem I'm facing is how can test the function locally or even on my CI pipeline on the repo, as secretmanager.SecretManagerServiceClient() is trying to connect to Secret Manager service as soon as I import my main.py file on main_test.py file, and I don't have any GCP auth credentials on the environment.

Not sure if there is a way to mock up the client without changing the whole structure.

Thanks again!

Dustin Ingram • Feb 12 '20

Hi Sergio, I'd advise monkey-patching the SecretManagerServiceClient to something you can use in your tests.

For example, if you use the pretend for stubbing, it could be something like:

import pretend
stub_secretmanagerserviceclient = lambda: pretend.stub(
    access_secret_version=lambda resource_name: pretend.stub(
        payload=pretend.stub(
            data=b'my_secret_string'
        )
    )
)

Anthony Gualandri • Mar 5 '20

This may be a stupid question, but I was wondering if you could explain more the part above about "the function is doing this work outside of the Python function itself. This will speed up our function by only accessing the secret once when the Cloud Function is instantiated, and not once for every single request." Does this keep the secret being retrieved secure or is it the same thing as storing it as an environment variable then, if the call to the Secrets Manager isn't made every time the function runs?

Also, if willing and able, can you explain the difference wrt to the cloud functions framework how it runs code that is included in the main.py file outside of any of the specific functions within the main.py file? I'm thinking in terms of Google's pricing structure here for frequency of calls to functions and duration. Thanks!

Dustin Ingram • Mar 5 '20

Yep, great questions actually. Doing it this way keeps the secret as secure as doing it inline on each request -- either way, the secret will only be stored as a variable in the memory of the execution environment, it's just a matter of whether that variable is scoped to the function or not, which doesn't make a difference here.

With regards to the Functions Framework, it behaves exactly like the Cloud Functions runtime, so you shouldn't see any difference in behavior if you use the framework vs. the vanilla runtime.

With regards to pricing: anything done outside the function itself happens once per instance per cold start, so moving as much execution outside the function itself should reduce the overall compute time across all invocations of your function.

Anthony Gualandri • Mar 6 '20

Thanks Dustin, this helps clarify things a lot more. I'm having trouble translating this to node.js though (which is what I'm needing to use due to other API's we use having only js helper libraries) given the async structure of js. So I'm wondering whether this kind of global scoping so that certain functions run only at instance cold start is not possible with js. If you have any info on that please let me know. Thanks!

Dustin Ingram • Mar 6 '20

Hey Anthony, one of my colleagues just published a similar guide for Node, hopefully this helps! dev.to/googlecloud/serverless-myst...

Anthony Gualandri • Mar 6 '20

Sweet! Wow, great timing hahaha. You weren't kidding with the just published part (today). This totally helped clarify the way things should be setup in the node environment. Thanks for sharing and taking the time to comment back!

Natalya Kosenko • Apr 27 '20

Thanks Dustin, very helpful article, I was able to set it up following your steps, it works!
I was just wondering if there a way to share the same code for accessing the secret between a few different google cloud functions. It's just does not feel right to copy-paste the secret-related code in each google cloud function. I was trying to find the answer from the Google documentation, but so far it looks like there is no easy way.

Harnit Singh • Sep 4 '19

This is fantastic, thank you v. much Dustin.
Not sure if others will run into the same, but I had to explicitly grant permissions to enable KMS.decrypt for the service account email used by my cloud functions.

BalaGanesh • Feb 21 '20

Harnit Singh ,

We are in the process of implementing new cloud function.

Am very new to that can you please help me how effectively we can use KMS in cloud function ,our setup is mobile application http request and respond back

Dustin Ingram • Dec 6 '19

Thank you, nice catch!

guysegal • Dec 6 '19

Hi Dustin, Great article!

There's one thing I didn't understand - why can't I commit the .env.yaml file to the repository?
The secret is encrypted and the only if you have access to the encryption key you can decrypt it.

Assuming you don't have access to the encryption key, what is the risk here?
Because the benefit of committing to the repo is that you have a full and ready to deploy code on your master at any given point.

I understand that exposing the encrypted secret is some kind of a threat, but I guess that if your encryption key was compromised you have bigger problems...

Dustin Ingram • Dec 6 '19

You're right, it can be included, as long as you're sure all secrets are properly encrypted. (This is also what CI services like Travis do: docs.travis-ci.com/user/encryption...)

Marc de Ruyter • Nov 2 '19

These kind of long and probably error prone procedures are exactly what Google should make simple, fast and easy.
Back to Netlify for me.