Google Cloud App Engine Environment Variables

One of my recent projects involved an application that had to be hosted on Google Cloud App Engine. I didn't use App Engine before, but I've managed apps on Heroku and OpenShift and was interested to see what Google Cloud PaaS had to offer.

It was a fairly standard Node.js application with most of the configuration done with environment variables. Soon it became clear that this could be a problem – App Engine does not support configurable environment variables.

Intentional or not, App Engine has only one way of defining those variables – in app.yaml configuration file. This file describes App Engine settings (runtime, url mappings etc.), including env_variables section that instructs App Engine to set environment variables on deployment.

Example app.yaml file content:

runtime: nodejs10

handlers:
- url: /api/.*
  script: auto
  secure: always

- url: /.*
  static_files: index.html
  upload: index.html
  secure: always
  http_headers:
    X-Frame-Options: deny
    X-DNS-Prefetch-Control: off
    X-XSS-Protection: 1; mode=block
    X-Permitted-Cross-Domain-Policies: none

env_variables:
    VAR1: 'VALUE1'
    VAR2: 'VALUE2'

Our deployment pipeline was already fully automated, so I needed to store app.yaml file somewhere to supply it to build server before pushing code to Google Cloud. Ideally, it would be application's code repository. However, having environment variables in app.yaml file caused an issue: we either needed to commit application configuration to the repository, or leave the whole file untracked. Neither of these options was suitable, so I started looking for any other ways of dealing with this App Engine limitation.

As a side note, Heroku and OpenShift (at least its previous incarnation) have an option to set environment variables from the web/command line interface, which simplified application configuration management.

My search brought me some disappointing results:

1. Store application configuration in the Google Cloud Datastore and read from it on application startup (link).
2. Encrypt configuration values with Cloud KMS and commit together with the rest of the app.yaml configuration (link).
3. Use separate app.yaml files for different environments (and, I guess, commit them all to the repository?) (same link).

Option #1 assumed additional component in our infrastructure and vendor lock-in to Google Cloud Datastore database, which was far from ideal.

Option #2 solved the security part of the problem, but would mean hardcoding encrypted environment-specific values to the codebase. It would also require to update the code with each new environment added or any changes to the existing environment variables. Not ideal.

Option #3 didn't solve he problem at all – code would still store information about its environments and application secrets will be available right in the code repository...

Extra step in deployment pipeline

Eventually, I've came up with an approach that involved compiling app.yaml file from template file during the build process. At that moment we used Google Cloud Build as a build server for CI/CD, but quickly moved to GitLab CI since Cloud Build does not support environment variables either.

To be fair, I should mention that Cloud Build supports "substitutions", which are remotely similar to environment variables. Unfortunately, the only way to pass substitutions to the build job is through command line arguments, which means managing environment variables somewhere outside. And this brings us back to the original problem...

The application was already using EJS library, so I used it to compile the template:

# app.tpl.yaml
runtime: nodejs10
env_variables:
  SSO_CLIENT_ID: <% SSO_CLIENT_ID %>
  SSO_SECRET: <% SSO_SECRET %>

with a script like that:

// bin/config-compile.js
const fs = require('fs');
const ejs = require('ejs');

const template = fs.readFileSync('app.tpl.yaml').toString();
const content = ejs.render(template, process.env);

fs.writeFileSync('app.yaml', content);

and GitLab CI step similar to this:

# .gitlab-ci.yml
config-compile:
  stage: build
  image: node:10
  script:
    - node bin/config-compile.js
  artifacts:
    paths:
      - app.yaml
    expire_in: 1 days
    when: always

This enabled us to manage application configuration in GitLab environment variables.

The approach can easily be adapted to any other templating library, programming language or build server, the code above is just an example.

A note on security

I found it interesting that one of the oldest Google Cloud products doesn't support such a common functionality. However, I accept there could be valid security reasons for doing so, e.g. exposure to dependency vulnerability similar to the one discovered in rest-client or typo-squatting like in npm registry. This is especially relevant as App Engine does not provide options to limit/manage outgoing connections from the environment.

Alternative solutions not covered in this post

• "Secrets in Google App Engine" by Stuart Leitch
• "How to use Environment Variables in GCloud App Engine" by Gunar Gessner