Today, we will learn to enable HTTPS and wildcard custom domain mapping on Google App Engine.
But why ?
With my client, we wanted to use AppEngine for our frontend applications.
Google AppEngine deploys static web applications easily with performant and fully managed servers. It also offers the possibility to have different versions of the same app running and being available at the same time.
Access specific application's version
For this to work, each version of each service has a dedicated URI in this form:
https://version-dot-service-dot-project.region.r.appspot.com. This is very powerful and handy to manage multiple environments. In our project, we have a production version, a development version and we plan to have one version for each pull request opened on Github.
The only drawback is the readability and the difficulty to remember these URIs. Google enables you to configure your own custom domain and let you dispatch this domain to a given service. But a more flexible approach is to define a wildcard custom domain for the project. With a wildcard custom domain, you can enjoy flexible routing, juste like with default generated URI, but with your own custom domain: http://version.service.example.com.
Here comes trouble
Fine! We have a clean URI for each version of each service. But here comes the biggest issue with this system: You can't enable HTTPS for your wild card custom domain since Google's managed SSL certificates are not available for wildcard domains.
"But why?" you will probably ask yourself. To issue a wildcard SSL certificate with LetsEncrypt, you need to achieve a DNS-01 challenge instead of HTTP-01 challenge. To automate certificate renewal (which is required given their TTL), you need to modify your DNS zone via API, which is different for each DNS provider. That is why Google does not (for now) implement managed wildcard SSL certificates.
Enable wildcard SSL certificate with automatic renewal
After this far too long introduction, we will finally learn how to add HTTPS for our beautiful URI with our beautiful custom domain.
At the end, we will have the following workflow:
The main prerequisite is to use a compatible DNS provider. Let's encrypt automatic renewal of SSL certificates is only compatible with these providers:
To follow this guide, you will need a running App Engine project with at least 1 service. Of course it's more impressive if you have more than one !
If you don't have one (or prefer not to mess with your real app), you can deploy this example with a beautiful cat and dog !
Step 1: Setup a service account
For our Cloud Run service to manage SSL certificates and access Cloud Storage bucket, it will need a service account with all the authorizations needed.
You can also edit the default Cloud Run service account, but it is not recommended since it opens a potential security breach for existing or future Cloud Run services.
Create a new dedicated service account named
certbot@<project-id>.iam.gserviceaccount.com) with the following role
App Engine Administrator (needed to upload SSL certificates) and
Cloud Run Invoker (needed by Cloud Scheduler to call the Cloud Run service).
If your DNS zone is managed by google, you should also give the permissions to manage DNS records (
DNS Administrator). This is needed by
certbotto resolve the DNS challenge.
Add yourself as a user and administrator of this service account.
Step 2: Add custom wildcard domain
Go to App Engine custom domain setting page to create and add the custom domain mappings.
Add a custom domain
To add a custom domain, you have to first register it and give to the service account the permissions to manage it's certificates.
Go to the domain registration page.
If you haven't registered any domain yet, you will have to follow the instructions and resolve a challenge to ensure that you are the owner of the domain.
During this step, make sure to enter your domain without the wildcard. For example, if you want to use
*.my.domain.com, make sure to input
http://(the placeholder is misleading here).
WARNING: Not following this rule will cause issues later in this guide when we will try to upload an SSL certificate, the domain is recognized only if it matches exactly the one specified in the SSL certificate.
Open the details page for your domain, and add the previously created service account's email (
certbot@<project-id>.iam.gserviceaccount.com) as an owner.
Create the mappings
You should now be prompted for domain mappings. Add your wildcard domain (
*.my.domain.com for example).
You should have 3 domain mapping :
With this configuration, the default service will be served on both
https://www.my.domain.com. If this is not desired, you can just remove them and only keep the wildcard mapping.
Edit your DNS zone
You should now see a list of DNS records. You have to edit your DNS zone with these new records.
If, in the previous step, you only kept the wildcard mapping, you can just add the CNAME entry
Step 3: Create a Cloud Storage bucket
To manage automatic certificate renewal,
certbot needs to keep a state on the filesystem. Since Fully Managed Cloud Run doesn't provide any persistent FS, we will have to backup and restore it from a Cloud Storage bucket.
Go to the bucket creation page and give it a unique name (
certbot-sate_my-domain-com for example).
You can choose whatever region you want (but try to put it at least in the same region as your Cloud Run and App Engine services).
Choose a storage class. Nearline is fine, or you can even choose Codline if you want to save the planet 🌳.
You can use Uniform access control since the entire bucket will only be used by our renewal service.
When created, go to
AUTHORIZATION tab and add a new member. There, put your service account email (
certbot@<project-id>.iam.gserviceaccount.com) and give it the
Old Storage Bucket Owner) role.
Step 4: Create the Cloud Run service
Go to Cloud Run service creation page, choose a region and a name (
certbot for example).
You can also build and publish the image yourself in your project. This is better for reliability and security.
Advanced Settings, change this configurations:
- Service Account: use the service account created during step 2 (
- Delay before request timeout: 600 (an SSL renewal is not a high speed process... so 10 minutes should be ok)
- Max instances number: 1
- Max number of requests by container: 1 (we don't want to be able to renew multiple time in parallel)
Variables tab and add the following variables:
- LETSENCRYPT_BUCKET: The bucket you have created (example:
- CERTIFICATE_NAME: The display name of the certificate. This will be visible in the App Engine Console, it should be unique for your application.
- CUSTOM_DOMAIN: Your naked custom domain (example:
my.domain.comif your wildcard is
- DNS_PROVIDER: The name of your DNS provider. It should be one of:
- DNS_PROVIDER_CREDENTIALS: The credential file content. It should contain the access token for your DNS provider. You can find documentation depending on your provider here > If you are using Google Domains for your DNS, omit this variable. You should instead add authorization to your service account to modify your DNS record.
- LETSENCRYPT_CONTACT_EMAIL: Your contact email. This will be sent to Let's Encrypt for record and will be publicly visible, so be sure to create an email for this specific use.
Next, let the default publicly opened traffic (Cloud Scheduler uses external HTTP requests), but enable
Require Authentication to avoid some malicious people from being able to trigger SSL certificate renewal.
Validate, you should be redirected to the service administration page. There, go to the
Authorization tab and add a member. Add the service account (
Enable App Engine Admin API
If your project is new or you have never automated anything related to App Engine, you probably need to enable App Engine Admin API.
This will be required by the Cloud Run service to upload the generated SSL certificate.
Create a Cloud Scheduler task
To renew our certificate every 2 months, we will create a Cloud Scheduler task.
Go to the creation page and use this configuration:
- Name: appengine_widlcard_certificate_renewal
- Frequency: 0 1 1 */2 *
- Time Zone: Your time zone :-)
- Target: HTTP
- URL: /renew
- HTTP Method: GET
- Authentication Header: OIDC token (click on
- Service Account: the create service account (
- Target: /renew
Validate, and you should now see your task.
You try it by clicking on
Execute. Click on
Display in the
Logs column to see it working !
If the task has finished without errors, you should see your custom SSL certificate in the App Engine console.
If there is a problem, and there will probably be at least one, you can disable the required authentication from your Cloud Run service. This will allow you to call the renewal endpoint manually and see the output logs.
gcloud run deploy certbot --image gcloud run deploy certbot --image eu.gcr.io/gcloud-certbot/gcloud-certbot --allow-unauthenticated
Don't forget to enable it when you have fixed all issues to avoid malicious calls:
gcloud run deploy certbot --image eu.gcr.io/gcloud-certbot/gcloud-certbot:latest --no-allow-unauthenticated
Congratulations ! You can now access every version of every service with your beautiful custom domain
This guide has been made during my work at Zenika for my client. The final goal of all this was to being able to publish our features branches with ease. And I think this goal is now achieved, we can now deploy a new version an app for a given PR with a simple command:
gcloud app deploy . my-service --version pr-1234 --no-stop-previous-version --no-promote
I hope this will help you do the same for your own app !
That being said, since Google Cloud decided to put less efforts on GAE, it can be a good idea for new project to look for a different solution. Firebase is currently working on this and we have made another guide (longer of course) guide to do the same thing on GKE
Top comments (4)
Hello, is there any updated version of this. i got stuck at the cloud run eu.gcr.io/gcloud-certbot/gcloud-certbot, and the repo too seems not to be working
I tried compiling the docker image. Boy what a mistake. I ended up getting a docker image by cloning the repo here: github.com/Zenika/gcloud-certbot
I then ran ./build.sh in that repo which pulled the certbot-repo. Unfortunately things have changed in that repo and the gcloud-certbot repo is super outdated.
Instead of trying to figure out what was wrong I checked out a commit from Nov 2020 for the certbot repo and used that. The image is currently compiling but no idea if it will work.
Didn't end up working :(
Couldn't get this to work for me either