Today, we will learn to enable HTTPS and wildcard custom domain mapping on Google App Engine.

But why ?

With my client, we wanted to use AppEngine for our frontend applications.

Google AppEngine deploys static web applications easily with performant and fully managed servers. It also offers the possibility to have different versions of the same app running and being available at the same time.

Access specific application's version

For this to work, each version of each service has a dedicated URI in this form: https://version-dot-service-dot-project.region.r.appspot.com. This is very powerful and handy to manage multiple environments. In our project, we have a production version, a development version and we plan to have one version for each pull request opened on Github.

The only drawback is the readability and the difficulty to remember these URIs. Google enables you to configure your own custom domain and let you dispatch this domain to a given service. But a more flexible approach is to define a wildcard custom domain for the project. With a wildcard custom domain, you can enjoy flexible routing, juste like with default generated URI, but with your own custom domain: http://version.service.example.com.

Here comes trouble

Fine! We have a clean URI for each version of each service. But here comes the biggest issue with this system: You can't enable HTTPS for your wild card custom domain since Google's managed SSL certificates are not available for wildcard domains.

"But why?" you will probably ask yourself. To issue a wildcard SSL certificate with LetsEncrypt, you need to achieve a DNS-01 challenge instead of HTTP-01 challenge. To automate certificate renewal (which is required given their TTL), you need to modify your DNS zone via API, which is different for each DNS provider. That is why Google does not (for now) implement managed wildcard SSL certificates.

Enable wildcard SSL certificate with automatic renewal

After this far too long introduction, we will finally learn how to add HTTPS for our beautiful URI with our beautiful custom domain.

At the end, we will have the following workflow:

Prerequisite

The main prerequisite is to use a compatible DNS provider. Let's encrypt automatic renewal of SSL certificates is only compatible with these providers:

• google
• cloudflare
• digitalocean
• dnsimple
• linode
• ovh
• rfc2136
• route53

To follow this guide, you will need a running App Engine project with at least 1 service. Of course it's more impressive if you have more than one !

If you don't have one (or prefer not to mess with your real app), you can deploy this example with a beautiful cat and dog !

Step 1: Setup a service account

For our Cloud Run service to manage SSL certificates and access Cloud Storage bucket, it will need a service account with all the authorizations needed.

You can also edit the default Cloud Run service account, but it is not recommended since it opens a potential security breach for existing or future Cloud Run services.

Create a new dedicated service account named certbot (certbot@<project-id>.iam.gserviceaccount.com) with the following role App Engine Administrator (needed to upload SSL certificates) and Cloud Run Invoker (needed by Cloud Scheduler to call the Cloud Run service).

If your DNS zone is managed by google, you should also give the permissions to manage DNS records (DNS Administrator). This is needed by certbot to resolve the DNS challenge.

Add yourself as a user and administrator of this service account.

Step 2: Add custom wildcard domain

Go to App Engine custom domain setting page to create and add the custom domain mappings.

Add a custom domain

To add a custom domain, you have to first register it and give to the service account the permissions to manage it's certificates.

Go to the domain registration page.

If you haven't registered any domain yet, you will have to follow the instructions and resolve a challenge to ensure that you are the owner of the domain.
During this step, make sure to enter your domain without the wildcard. For example, if you want to use *.my.domain.com, make sure to input my.domain.com, without * nor http:// (the placeholder is misleading here).

WARNING: Not following this rule will cause issues later in this guide when we will try to upload an SSL certificate, the domain is recognized only if it matches exactly the one specified in the SSL certificate.

Open the details page for your domain, and add the previously created service account's email (certbot@<project-id>.iam.gserviceaccount.com) as an owner.

Create the mappings

You should now be prompted for domain mappings. Add your wildcard domain (*.my.domain.com for example).

You should have 3 domain mapping :

• my.domain.com
• www.my.domain.com
• *.my.domain.com

With this configuration, the default service will be served on both https://my.domain.com and https://www.my.domain.com. If this is not desired, you can just remove them and only keep the wildcard mapping.

Edit your DNS zone

You should now see a list of DNS records. You have to edit your DNS zone with these new records.

If, in the previous step, you only kept the wildcard mapping, you can just add the CNAME entry

Step 3: Create a Cloud Storage bucket

To manage automatic certificate renewal, certbot needs to keep a state on the filesystem. Since Fully Managed Cloud Run doesn't provide any persistent FS, we will have to backup and restore it from a Cloud Storage bucket.

Go to the bucket creation page and give it a unique name (certbot-sate_my-domain-com for example).

You can choose whatever region you want (but try to put it at least in the same region as your Cloud Run and App Engine services).

Choose a storage class. Nearline is fine, or you can even choose Codline if you want to save the planet 🌳.

You can use Uniform access control since the entire bucket will only be used by our renewal service.

When created, go to AUTHORIZATION tab and add a new member. There, put your service account email (certbot@<project-id>.iam.gserviceaccount.com) and give it the Old Storage Bucket Owner) role.

Step 4: Create the Cloud Run service

Go to Cloud Run service creation page, choose a region and a name (certbot for example).

Use the eu.gcr.io/gcloud-certbot/gcloud-certbot image.

You can also build and publish the image yourself in your project. This is better for reliability and security.

Under Advanced Settings, change this configurations:

• Service Account: use the service account created during step 2 (certbot@<project-id>.iam.gserviceaccount.com).
• Delay before request timeout: 600 (an SSL renewal is not a high speed process... so 10 minutes should be ok)
• Max instances number: 1
• Max number of requests by container: 1 (we don't want to be able to renew multiple time in parallel)

Go to Variables tab and add the following variables:

• LETSENCRYPT_BUCKET: The bucket you have created (example: gs://certbot-sate_my-domain-com)
• CERTIFICATE_NAME: The display name of the certificate. This will be visible in the App Engine Console, it should be unique for your application.
• CUSTOM_DOMAIN: Your naked custom domain (example: my.domain.com if your wildcard is *.my.domain.com)
• DNS_PROVIDER: The name of your DNS provider. It should be one of:
  • google
  • cloudflare
  • digitalocean
  • dnsimple
  • linode
  • ovh
  • rfc2136
  • route53
• DNS_PROVIDER_CREDENTIALS: The credential file content. It should contain the access token for your DNS provider. You can find documentation depending on your provider here > If you are using Google Domains for your DNS, omit this variable. You should instead add authorization to your service account to modify your DNS record.
• LETSENCRYPT_CONTACT_EMAIL: Your contact email. This will be sent to Let's Encrypt for record and will be publicly visible, so be sure to create an email for this specific use.

Next, let the default publicly opened traffic (Cloud Scheduler uses external HTTP requests), but enable Require Authentication to avoid some malicious people from being able to trigger SSL certificate renewal.

Validate, you should be redirected to the service administration page. There, go to the Authorization tab and add a member. Add the service account (certbot@<project-id>.iam.gserviceaccount.com)

Enable App Engine Admin API

If your project is new or you have never automated anything related to App Engine, you probably need to enable App Engine Admin API.

This will be required by the Cloud Run service to upload the generated SSL certificate.

Create a Cloud Scheduler task

To renew our certificate every 2 months, we will create a Cloud Scheduler task.

Go to the creation page and use this configuration:

• Name: appengine_widlcard_certificate_renewal
• Frequency: 0 1 1 */2 *
• Time Zone: Your time zone :-)
• Target: HTTP
• URL: /renew
• HTTP Method: GET
• Authentication Header: OIDC token (click on Show More button)
• Service Account: the create service account (certbot@gcloud-certbot.iam.gserviceaccount.com)
• Target: /renew

Validate, and you should now see your task.

You try it by clicking on Execute. Click on Display in the Logs column to see it working !

If the task has finished without errors, you should see your custom SSL certificate in the App Engine console.

Troubleshooting

If there is a problem, and there will probably be at least one, you can disable the required authentication from your Cloud Run service. This will allow you to call the renewal endpoint manually and see the output logs.

gcloud run deploy certbot --image gcloud run deploy certbot --image eu.gcr.io/gcloud-certbot/gcloud-certbot --allow-unauthenticated

Don't forget to enable it when you have fixed all issues to avoid malicious calls:

gcloud run deploy certbot --image eu.gcr.io/gcloud-certbot/gcloud-certbot:latest --no-allow-unauthenticated

Congratulations ! You can now access every version of every service with your beautiful custom domain https://version.service.my.domain.com \o/

Conclusion

This guide has been made during my work at Zenika for my client. The final goal of all this was to being able to publish our features branches with ease. And I think this goal is now achieved, we can now deploy a new version an app for a given PR with a simple command:

gcloud app deploy . my-service --version pr-1234 --no-stop-previous-version --no-promote

I hope this will help you do the same for your own app !

That being said, since Google Cloud decided to put less efforts on GAE, it can be a good idea for new project to look for a different solution. Firebase is currently working on this and we have made another guide (longer of course) guide to do the same thing on GKE