GCP Serverless deployments with Terraform: Cloud Run vs App Engine

As part of my job, I have been recently tasked to build the infrastructure hosting a NodeJs application on Google App Engine Flexible platform.
One of the security guardrails on the project forbade the use of external IP addresses, therefore there was a specific requirement to deploy App Engine instances with an internal IP address only, and route traffic to the app through an external HTTP Cloud Load Balancer, with App Engine serving as its serverless NEGs backend.
This solution allows to terminate SSL encryption on the Load Balancer and to have multi-region load balancing—App Engine is indeed a regional service, so it doesn’t come with global load balancing by default.

While building the infrastructure with Terraform, I came across a bunch of limitations and issues, which made me start wondering if it was the case to opt for another Google serverless product, namely Cloud Run.

GCP Cloud Run and App Engine are fully managed serverless services that let you deploy your app without worrying too much about the underlying infrastructure or such requirements as capacity provisioning, load balancing, or scaling.

While being both serverless, scalable services, they present some major differences:

Terraform is an open-source, cloud-agnostic tool that allows to achieve reliable, repeatable deployments. Terraform deploys your infrastructure taking care of all its dependencies and sorting them out.

In this article I will compare Cloud Run and App Engine, going through the limitations of deploying their infrastructure with Terraform, and I will show you why Terraform works fine with Cloud Run but it is not suited for building the App Engine service as Infrastructure as code.

Use case: we want to deploy a simple nodejs app on GCP. We do not want to expose the app publicly to the internet, but we want to access it through an external load balancer.
Cloud Run and App Engine will serve both as serverless network endpoint groups (NEGs) backend.

Prerequisites

• Terraform installed locally. You can download it here.
• Google Cloud SDK.
• A Google Cloud project.

Deploying on App Engine

First thing to know is that App Engine is a regional service and you cannot have more than one App Engine app in your project.
Once created, App Engine cannot be deleted and you cannot change the deployment region, therefore, choose carefully where you want to deploy it.
The App Engine Flexible service provides a bunch of additional features that are not included in the Standard environment, among these, the possibility to deploy your app in a private subnet with an internal IP only.

In order to deploy your app on App Engine, you need to provide a configuration file, called app.yaml, that must reside in the root of your application folder. This configuration file will be used by Cloud Build, which—under the hood—builds and deploys your app.
When deploying with Terraform, we zip our application code folder, upload it to a Cloud Storage bucket, and give Cloud Build Service Account permissions to read from that bucket and build the application. You can access the full Terraform code at this link.

Therefore, when we run terraform apply, two processes happen simultaneously:

• Terraform makes a call to google API and build the resources according to its state file
• Cloud Build retrieves the Cloud Storage object, unzips it and builds the app

This can lead to conflicts and mismatching during the deployment, which most of the times cause our terraform apply to fail. Furthermore, although Terraform gives you the option of specifying the network field in your google_app_engine_flexible_app_version resource, unfortunately it hasn’t integrated the internal_ip argument in the network block yet.

# app-engine.tf

resource "google_app_engine_flexible_app_version" "myapp_v1" {
...

 network  {
   name             = google_compute_network.vpc.name
   subnetwork       = google_compute_subnetwork.private.name
   forwarded_ports  = null
   instance_tag     = "app-engine-vm"
   session_affinity = true
 }
...
}

Hence, running your Terraform code will most likely result in an error probably because of the mismatch between Terraform resource and the app.yaml file. This means that you will not be able to build your infrastructure exposed through the external load balancer.
At the moment of writing this article, the only way to avoid having an ephemeral external IP address assigned to your App Engine instances is to deploy the service through the gcloud CLI.

App Engine services work with versions. Every time you change something in your application code, you can deploy a new version of your service.
Therefore, the ideal way of managing App Engine deployments would be through a CI/CD pipeline, while Terraform wouldn’t be suited for this service, because it would require manual changes into the google_app_engine_flexible_app_version resource, every time a change in the source code is made.

Deploying on Cloud Run

Cloud Run is a managed serverless platform that lets you run request-serving containers, directly on Google's scalable infrastructure. It scales in or out automatically depending on requests, and can scale to zero if there are no incoming requests, allowing you to completely cut your costs.

Terraform supports Cloud Run starting from Google provider version 3.3. In this example we built our Cloud Run service to be able to receive requests only from the internal Google network and the load balancer. This means that our container will be accessible from the internet only through the external load balancer.

# cloud-run.tf

resource "google_cloud_run_service" "run_service" {
 provider = google-beta
 project  = var.project_id
 name     = var.service_name
 location = var.project_region
 template {
   spec {
     containers {
       image = local.app_image
     }
   }
 }
 metadata {
   annotations = {
     "run.googleapis.com/ingress" = "internal-and-cloud-load-balancing"
   }
 }
 autogenerate_revision_name = true
 traffic {
   percent         = 100
   latest_revision = true
 }
}

By default, Cloud Run services are secured by IAM, and to access them you need the Cloud Run Invoker permission set. For simplicity, in this example we allow unauthenticated access to our service, by granting the permission set to allUsers.

# cloud-run.tf

resource "google_cloud_run_service_iam_member" "allUsers" {
 service  = google_cloud_run_service.run_service.name
 location = google_cloud_run_service.run_service.location
 role     = "roles/run.invoker"
 member   = "allUsers"
}

Here we built our image directly with Terraform, using a null_resource with a local-exec provisioner, but you can automate the process with Cloud Build or any other CI/CD platform.
Once your apply is done, navigate to your Google Load Balancing console and click on the load balancer address: you’ll be able to see your Hello World app running. Full code here.

Cloud Run works with revisions. Each revision corresponds to a change made in the configuration or a new image uploaded. Although Terraform is ideal to launch your initial infrastructure to Cloud Run, it is not the best choice to update it, because this would mean manually updating the template block in your resource every time you change something in your application code.
Hence, it is advisable to use Terraform to provision the infrastructure, and another CI step to build and deploy the container. A workaround to separate the two workflows is to add a lifecycle block in your google_cloud_run_service resource to ignore the changes in the deployed image, so that Terraform will not complain that the image is different from the one it manages.

# cloud-run.tf

lifecycle {
   ignore_changes = [template[0].spec[0].containers[0].image]
 }

Conclusions

As Google state in their documentation:

Cloud Run is designed to improve upon the App Engine experience [...]. Cloud Run services can handle the same workloads as App Engine services, but Cloud Run offers customers much more flexibility [...]. This flexibility, along with improved integrations with both Google Cloud and third-party services, also enables Cloud Run to handle workloads that cannot run on App Engine.

From a developer perspective, we can add to this statement that, when dealing with the deployment of these two resources as Infrastructure as Code, due to the nature of both platforms, Terraform can be a good choice to build the initial infrastructure, while for the actual deployments of the code hosted on them, it is better to opt for a CI/CD tool (e.g. Google Cloud Build).
On top of that, considering that the current version of Terraform has not yet fully integrated all App Engine API features and its API calls often result in errors and deployment failures, when it comes to building the infrastructure with a IaC tool, Cloud Run is definitely the service that can be better implemented with Terraform.
With that said, let's keep an eye open for better integrations of App Engine in future versions of Terraform and Google provider.