Table of contents
- Recap
- Living document intention
-
Serving an HTTP app using Docker, GKE and Cloudflare
- 1. Write an HTTP app
- 2. Build the app into an image
- 3. Register and onboard with Google Cloud
- 4. Setup a Google Artifacts Registry project
- 5. Tag the Docker image based on Google's Artifact Registry
- 6. Install the
gcloud
command on your machine - 7. Configure and authenticate gcloud
- 8. Push the Docker image to Artifact Registry
- 9. Set up a Shared VPC project
- 10. Set up a shared GKE cluster project
- 11. Set up Cloud SQL instances
- 12. Create a Kubernetes cluster
- 13. Prepare secrets for staging and production
- 14. Create your first GKE workload
- 15. Apply secrets to GKE workloads
- 16. Prepare Cloudflare for Total SSL
- 17. Expose your application using Cloudflare Tunnel
- 18. See your application live 🎉
- Where does our trip take us next? 🚀
- Changelog
- Stay connected 💬
Recap
Software Engineering Entrepreneurship » Issue 5 » Up and running with Google Cloud
Morgan Wowk ・ Feb 18 '23
In issue 4-5 of my series Software Engineering Entrepreneurship we covered the technology stack serving the foundation of our future apps as a Software Engineer-by-day turned Entrepreneur.
Living document intention
At the start of this series I disclosed that I would not give a step-by-step tutorial at any point. This document, however, will serve as the most important log and perhaps the one exception for myself and others to avoid hurdles building this stack in the future.
Find below an ordered, roughly detailed guide to setting up this technology stack. I will update this document as new discoveries are made or as new steps are introduced. See the bottom of this document for a changelog documenting such edits. Use this document only as an assistant to setting up infrastructure based on your distinct use case and requirements.
Serving an HTTP app using Docker, GKE and Cloudflare
Subject to change (see changelog)
1. Write an HTTP app
In your language of choice write an HTTP app that when ran, serves on a specified port.
In an ideal world, use Docker for both your development and your production build. You may have a different Dockerfile for development and production. Your production Dockerfile should run an executable (using the CMD
keyword) that serves a long running process on a port you specify. For example, in Go you might have a Dockerfile.dev
that runs an air command and a separate production Dockerfile
that runs a go build
and then CMD ["yourapp"]
.
2. Build the app into an image
docker build -t example-api-build-<build-version> -f ./deployment/docker/api/Dockerfile .
Replacing the name appropriately and <build-version>
with a versioning mechanism you have decided on (or test
if you wish to defer that task for later). See Container Image Versioning by Rahul Sharma.
3. Register and onboard with Google Cloud
This step is likely the most time consuming.
Budget decision
First, make sure you're in a position with enough available budget to commit to using Google Cloud. Deploying Google Cloud VMs and services could easily exceed $100 / month and this may come as a surprise to some. You should be careful with the resources you create as you can unintentionally rack up costs. Google will strive for minimum costs in many cases and provide estimates as much as possible. I would still recommend getting into the habit of looking up "[Insert GCP Product] Pricing" and at least doing a rough estimate of your own.
Check out Google's free tier as well.
Registering with Google Cloud
If you're comfortable moving forward you can roughly follow the steps below:
- If you haven't already, register a domain and setup an email with the domain you're going to use to administrate Google Cloud.
- Register with Google Cloud.
- Warning: Be cautious following Google's on-screen onboarding flow. Its purpose is to scaffold Google products for you based on your needs and answers to prompts. While I went through it and learned from it I ended up re-doing all the work it did on my own and actually removed most of the scaffolding it had done on my behalf. The reason was that over time I learned more about what "projects" are and how I wanted resources to be organized within GCP. I also discovered that some of the resources Google scaffolded for me actually immediately started incurring costs which consumed all of my initial GCP credits. After speaking with Google's support I was able to confirm there is no cost or credit forgiveness policy at Google either.
- As part of onboarding, set up a billing account you will attach to all the projects you later create.
Example project structure
For your inspiration, below is the project structure I have implemented:
Some notes on this structure:
Element | Description |
---|---|
Common folder | Folder containing projects that will hold resources pertaining to both staging and production environments. |
Production folder | Folder containing projects that will hold resources only pertaining to production. |
Staging folder | Folder containing projects that will hold resources only pertaining to staging. |
apps-artifacts | Project which will contain a resource of Google's Artifact Registry. The registry will be available to staging and production. |
apps-shared-cluster | Project which will contain a Google Kubernetes Cluster (GKE) that is shared between staging and production. This is only to be used prior to launch and will be deprecated in favor of a more fault tolerant system closer to launch. |
apps-shared-sql | Project which will contain a Cloud SQL instance that will be shared between staging and production. This is only to be used prior to launch and will be deprecated in favor of a more fault tolerant system closer to launch. |
apps-shared-vpc | This important project should be setup first. It contains a Shared VPC that will allow resources (Cloud SQL, GKE, etc.) to communicate with each other across projects via private IP. |
production-resources | Project that is empty to start off. It will later be used to contain a production-specific GKE cluster, replacing the existing shared cluster. |
staging-resources | Project that is empty to start off. It will later be used to contain a staging-specific GKE cluster, replacing the existing shared cluster. |
4. Setup a Google Artifacts Registry project
- Create a project
apps-artifacts
orapps-docker-artifacts
. - Open the project.
- In the left navigation select Artifact Registry.
- Enable the Artifact Registry API.
- Create a registry to hold Docker images following the prompts in the UI to your liking.
5. Tag the Docker image based on Google's Artifact Registry
The format to use for tagging Artifact Registry images is LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE
.
Command synopsis:
docker tag <image-to-tag-from-earlier-step> <tag>
Example command:
docker tag example-api-build-test northamerica-northeast2-docker.pkg.dev/apps-artifacts/docker-images/example-api-build-test:latest
6. Install the gcloud
command on your machine
On your local machine you're going to be using the gcloud
CLI every day soon enough. Now's a good time to install it. Follow Google's instructions to install gcloud CLI.
7. Configure and authenticate gcloud
The first exciting thing to do with your new gcloud
command is to setup your configuration.
See your existing configurations:
gcloud config configurations list
Create a new configuration:
gcloud config configurations create config-example-company
Set the google account on your currently active configuration:
gcloud config set account yourname@exampleapps.net
Authenticate gcloud with your Google account:
gcloud auth login
Other useful commands:
gcloud projects list
gcloud config set project <project-name>
8. Push the Docker image to Artifact Registry
Configure Docker for pushing to Artifact Registry
gcloud auth configure-docker LOCATION-docker.pkg.dev
# Example: gcloud auth configure-docker northamerica-northeast2-docker.pkg.dev
Push the Docker image to Artifact Registry
docker push <tag-from-earlier-step>
# Example: docker push northamerica-northeast2-docker.pkg.dev/apps-artifacts/docker-images/example-api-build-test:latest
9. Set up a Shared VPC project
If you're following the same structure of projects as I have then you're going to want to setup a Shared VPC project; A project that will serve as your networking "host" for resources spread across multiple projects. This will allow your resources across projects to communicate over private IP, reducing latency and improving security compared to public IP without the Shared VPC.
- Create a project
apps-shared-vpc
in the directoryApps/Common
. - Within the project navigate to VPC Network -> Shared VPC.
- Create a Shared VPC. Reference the section Create a network and two subnets from Google's Setting up clusters with Shared VPC article.
- Navigate to Kubernetes Engine -> Clusters.
- Enable the Kubernetes Engine API. It will remain unused but is required for the Shared VPC to later work with GKE.
10. Set up a shared GKE cluster project
From issue 5 of my series Software Engineering Entrepreneurship you would have heard my recommendation to use a shared GKE cluster while you are in a pre-seed / development phase of building your own apps. You can repeat the steps below for one or many GKE projects you create depending on your use case.
Create the project
Create a project apps-shared-cluster
in the directory Apps/Common
.
Grant permissions to Artifact Registry
Because the Artifact Registry is in a separate project you need to explicitly allow the apps-shared-cluster
project to read images from the apps-artifacts
project. Without doing so, your Kubernetes deployments will fail with an image pull error.
- Within the
apps-shared-cluster
project navigate to IAM & Admin. - Copy the Principal email for the name
Google APIs Service Agent
,Compute Engine default service account
- as well as the email that resembles<id>@cloudbuild.gserviceaccount.com
. - For each Principal email, navigate to the
apps-artifacts
project. - Navigate to IAM & Admin for the artifacts project.
- Select "Grant access" and assign the role
Artifact Registry Reader
andCompute Image User
to each Principal copied from the previous step.
Grant permissions to networking within the Shared VPC
In later steps you will deploy a Kubernetes cluster and relevant workloads. As part of this process GKE will attempt to automatically create firewall rules for you based on what services are available to the outside world. In order for GKE to do this it needs access in the apps-shared-cluster
project to control networking in the apps-shared-vpc
project that is the networking host. Follow the steps below to configure necessary permissions:
- Open the project
apps-shared-cluster
. - Navigate to IAM & Admin.
- For both the Principals named
Google APIs Service Agent
andCompute Engine default service account
select "Grant access" and assign the principals (by email) the following roles:- Compute Network Admin
- Service Networking Service Agent
11. Set up Cloud SQL instances
If your app is using Cloud SQL similar to my use case then you may follow the steps below to get setup.
- Create a project
apps-shared-sql
if you plan on using a single shared VM for both your staging and production databases during initial development. Otherwise, create or select the project of your choosing to host your Cloud SQL instance (e.g.staging-resources
). - Within the chosen project navigate to SQL.
- Continue to create a Cloud SQL instance.
- Ensure the instance is created on the Shared VPC you put together earlier.
- If prompted and you do are not familiar enough with writing your own IP ranges, choose to let Google automatically allocate an IP range (still within the Shared VPC network).
- With the instance created you can use Google's UI to create databases if you like. I recommend creating databases such as
example_app_staging
,example_app_production
. Including the environment in the database name is helpful when you are using a shared instance or migrate to a shared instance in the future - you will avoid conflicts with database names. - You can also use Google's UI to create users. I recommend creating users such as
<lastname><firstinitial>_RW
,code_example_app_staging
andcode_example_app_production
. Following best practices of ensuring app environment have separate users from those that your team members connect to on their own clients. - Securely store the username, passwords and connection details for each user created.
12. Create a Kubernetes cluster
A kubernetes cluster represents an instruction to Google to reserve a node pool (physical VMs) that will later host GKE workflows. When creating GKE clusters you will have the option to self manage nodes or use Google's recommended GKE Autopilot cluster - Autopilot will automatically allocate the minimum resources (CPU, RAM and storage) to meet the needs of your workloads.
- Open your previously created GKE project (e.g.
apps-shared-cluster
). - Navigate to Kubernetes Engine -> Clusters.
- Continue to create an Autopilot cluster (recommended) or manually manage a cluster. See GKE pricing.
- If you are going to be using a shared cluster for staging and production to save costs then use a name such as
apps-shared
. - Ensure the GKE cluster is created within the Shared VPC network created earlier.
- Here is an example configuration for my own cluster based on the Shared VPC we created earlier.
- If you're unsure, choose to make the cluster public instead of private for now.
- If you are going to be using a shared cluster for staging and production to save costs then use a name such as
13. Prepare secrets for staging and production
One task you will need to endeavour is having a secrets strategy for your local, staging and production environments. In this section of the document I will share with you the secrets strategy I am personally using. This is before setting up any sort of continuous delivery approach.
- Setup a Gitlab/Github organization to version control applications and secrets.
- Setup an isolated group and projects to store secrets for applications. For example:
- Have a folder for each environment within your secrets project. E.g
staging
andproduction
. -
Store files within those folders that will be pushed to Kubernetes as
secrets
later on. Here is an example secrets yml file:
DB: Connections: Primary: Host: 11.22.70.4 Port: 3306 Username: code_example_app_production Password: A5odaPLjBo[o#f}A$TBQ Database: example_app_production
Commit these files to your repository. We will come back to this later on when we are ready to push secrets to your Kubernetes namespaces.
14. Create your first GKE workload
You're now getting really close to having a running, accessible application. Now's the time to create your GKE workload. Here you will deploy your previously created artifact (HTTP application). Disclaimer though, if your application relies on secrets to establish a database connection then your workload will fail to spin up pods until those secrets are applied. Regardless though, Google's UI will only let you deploy the workload first and then apply secrets after. Follow the steps below:
Create your workload
- Open your GKE project in Google's console.
- Navigate to Kubernetes Engine -> Workloads.
- Select "Deploy".
- Select "Existing container image".
- Change the artifact registry source project to the project you created earlier to host your Artifact Registry (e.g.
apps-artifacts
). - Select the image tag/sha you wish you deploy (e.g.
example-api-build-test:latest
.
- Continue.
- Configure your workload
- For the deployment name, think about what you are serving. For example, if it's an HTTP API use
api
orbackend
. Note: I recommend having separate namespaces for each environment. E.g. The namespaceexample-app-staging
. If you do this, you don't need to include the environmentstaging
orproduction
in your deployment name. You can simply useapi
instead ofapi-staging
. There are other scoping benefits later on such as the name of Kubernetes secrets. - For the labels I suggest the format
app: example-app-staging
to control workload scheduling based on app and environment. - For the GKE cluster select the one you created in the previous steps.
- For the deployment name, think about what you are serving. For example, if it's an HTTP API use
- Select "Continue".
-
Select "Expose deployment as a new service" to allow Google to expose a public endpoint (IP) for your created service / HTTP app.
- "Port 1" should be the port that will be publicly accessible.
- "Target port" should be the port your HTTP app is told to run on within the codebase and/or Dockerfile.
- "Load balancer" is the appropriate service type if you wish to expose the application to the internet directly from the Kubernetes workload.
Important: Exposing your services at this level circumvents future layers of your infrastructure such as rate limiting, caching and API gateways. For the remainder of this article we will choose "Cluster IP" instead and expose the service through private IP and Cloudflare instead.
Select "Deploy"
Heads up
Your Workload will attempt to start running based on the Docker image pushed to Artifact Registry. However, if that application depends on a database connection or other secret information then it is likely going to fail to start and that is completely normal for now.
15. Apply secrets to GKE workloads
This step covers how you can apply secrets to your GKE namespaces in order for your dependent workloads to successfully run.
Connect to your cluster
- Navigate to Kubernetes Engine -> Clusters.
- Next to the relevant cluster select "Connect".
- Choose to connect through your own command line. Tip: I recommend having the
kubectx
command on your machine to always have visibility on which K8 context you are in. - Once you are connected to the correct context you can now run through the following steps to apply secrets.
Navigate to your secret directory
Based on the environment you are applying secrets, navigate in terminal to the directory of your CI resources / secrets repository holding the relevant file(s).
Check your namespaces
kubectl get namespace
# Work within the relevant namespace for all the following commands using the '-n <namespace>' flag
Create the secret
Here is an example of applying a yml file as a secret to a staging application.
kubectl -n <namespace> create secret generic <secret-name>-<secret-environment> --from-file=./<secret-file>
# Example: kubectl -n example-app-staging create secret generic app-config-example-app-staging --from-file=./config.yml
Take notice that even though it's applied within the specific staging
namespace we're still including the environment staging
in the name of the secret. This is a future-proof incase our team decided to move multiple environments into a single namespace at any point (not recommended.
Update your deployment to mount the secret
Even though the secret is created, it's only holding a place in Kubernetes data store without actually being used in any particular workload / deployment. We now need to edit your failing workload to (1) register a volume derived from a secret and then (2) mount a volume within the created pods.
- Open your GKE workload in Google Cloud.
- Select "Edit".
-
Make the following additions to the deployment yml. Use this as a structural and naming reference.
spec: template: spec: containers: - image: <do-not-edit> volumeMounts: - mountPath: "/secrets" name: app-config-example-app-staging readOnly: true volumes: - name: app-config-example-app-staging secret: secretName: app-config-example-app-staging
Note that correct indentation / placement is important here.
Select "Save". Heads up: Because Kubernetes and your Autopilot cluster is always managing your deployment and making changes, you may be asked to "Forcefully apply" the new deployment. This is okay to select.
With the secret now mounted into the workload at the location /secrets/config.yml
the workload will now refresh, generate new pods (allow some time) and should succeed. If you continue to encounter errors you can always look at the "Logs" section of your Google Cloud workload or the "Logs Explorer" product itself in your navigation.
16. Prepare Cloudflare for Total SSL
As part of our technology stack we are going to use Cloudflare as an edge DNS, SSL, Cacheing and Tunnel provider. Below are my recommendations for setting up Cloudflare:
- Add your domain to Cloudflare.
- Open your Cloudflare Dashboard.
- Navigate to SSL/TLS -> Edge Certificates.
- Enable "Total TLS" and opt-in to the paid add-on "Advanced Certificate Manager" ($10 / month at time of writing).
By enabling Total TLS, Cloudflare will automatically issue SSL certificates for new DNS records whether they're created manually or through a Tunnel. This serves as a huge convenience for live applications and local development.
17. Expose your application using Cloudflare Tunnel
Earlier we made the decision to expose our GKE Workload using "Cluster IP". This means the application itself is only served internally so we can add additional layers before it is exposed to the internet. In this case we are interested in placing Cloudflare between our application and the internet.
We will deploy a sibling GKE Workload that leverages Cloudflare Tunnel to expose our Cluster IP to the internet via custom domain.
- Install the
cloudflared
CLI on your local machine. - Run
cloudflared tunnel login
to authenticate your CLI with Cloudflare. Select the appropriate domain you will be creating tunnels for. - Run
cloudflared tunnel create example-app-staging
replacingexample-app-staging
with your desired tunnel name. - Run
cloudflared tunnel route dns example-app-staging tunnel.example.com
again replacing the tunnel name appropriately. - Copy the created credential json file into your secrets storage (i.e. Gitlab repository) for tracking. Use a name such as
cloudflare-tunnel-credential-example-app-staging.json
. - Run
kubectl -n example-app-staging create secret generic cloudflare-tunnel-credential-example-app-staging --from-file=credentials.json=/<path-from-earlier-output>/<tunnel-id>.json
replacing the namespace, secret name and tunnel credential path appropriately. -
Now create your Cloudflare Tunnel config using a name such as
cloudflare-tunnel-config-example-app-staging.yml
. Reference the following template or see the bottom of Cloudflare's K8 example yaml:
tunnel: example-app-staging credentials-file: /etc/cloudflared/creds/credentials.json # Serves the metrics server under /metrics and the readiness server under /ready metrics: 0.0.0.0:2000 # Autoupdates applied in a k8s pod will be lost when the pod is removed or restarted, so # autoupdate doesn't make sense in Kubernetes. However, outside of Kubernetes, we strongly # recommend using autoupdate. no-autoupdate: true # The `ingress` block tells cloudflared which local service to route incoming # requests to. For more about ingress rules, see # https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress # # Remember, these rules route traffic from cloudflared to a local service. To route traffic # from the internet to cloudflared, run `cloudflared tunnel route dns <tunnel> <hostname>`. # E.g. `cloudflared tunnel route dns example-tunnel tunnel.example.com`. ingress: # The first rule proxies traffic to the httpbin sample Service defined in app.yaml - hostname: example-app.staging.exampleapps.net service: http://api-service:80 # This rule matches any traffic which didn't match a previous rule, and responds with HTTP 404. - service: http_status:404
-
Create a secret for the config within the same namespace as the service you're exposing.
kubectl -n example-app-staging create secret generic cloudflare-tunnel-config-example-app-staging --from-file=config.yaml=./cloudflare-tunnel-config-example-app-staging.yml
-
Construct a Kubernetes deployment (i.e.
api-cloudflare-tunnel.yml
) based on Cloudflare's example yml here.- Use a deployment name of your choosing. One suggestion is
cloudflare-tunnel-<deployment-to-expose>
(e.g.cloudflare-tunnel-api
).
- Use a deployment name of your choosing. One suggestion is
-
Make the following edits to the deployment yml. Use this as a structural and naming reference.
spec: template: spec: containers: - image: <do-not-edit> volumeMounts: - mountPath: "/etc/cloudflared/creds" name: cloudflare-tunnel-credential-example-app-staging readOnly: true - mountPath: "/etc/cloudflared/config" name: cloudflare-tunnel-config-example-app-staging readOnly: true volumes: - name: cloudflare-tunnel-credential-example-app-staging secret: secretName: cloudflare-tunnel-credential-example-app-staging - name: cloudflare-tunnel-config-example-app-staging secret: secretName: cloudflare-tunnel-config-example-app-staging
Note that correct indentation / placement is important here.
-
Apply the new deployment under the same Kubernetes context and namespace as the service you're trying to expose. With our staging example app as an example, that would be:
kubectl -n example-app-staging apply -f api-cloudflare-tunnel.yml
Your GKE Workload pods will recognize the new changes and should now succeed in starting the tunnel.
Important: Be sure to set the maximum number of replicas for your Cloudflare GKE Workload to 1. This is to support only having 1 tunnel running as duplicates will fail.
18. See your application live 🎉
This is where all of your hard work pays off. While there are still many improvements to make to the current infrastructure you should see that your application is now available through your custom domain using private IP and a secure Tunnel to Cloudflare!
You now have an application available to the internet with a database and auto-scaling capabilities.
Where does our trip take us next? 🚀
The fun doesn't stop here. Continue visiting this page and following the Software Engineering Entrepreneurship series. I will take you along the journey as we work on improved security measures, building out CI/CD and more.
Changelog
February 24, 2023
- Removed the previous use of exposing our GKE Workload using a public IP load balancer. This document now covers securing your application behind a private IP and establishing a Tunnel directly to Cloudflare and your custom domain.
Stay connected 💬
I would love to connect with you and follow journeys of your own in life. Connect with me on LinkedIn, Twitter or our home at DEV.
Top comments (4)
Discussion point
In the end you will end up with the following workloads:
Would it be a cost optimization to have the Cloudflare Tunnel deployed as a sibling container to the
api
in the same pod?Would Cloudflare behave appropriately with parrarel connections to the same tunnel when multiple replicas exist for the
api
deployment in such a scenario?This question is important as you're paying for every new vCPU and GB of RAM. Potentially taking a multi-container approach would provide a more effective use of resources.
Billing breakdown after 1 month
This is expected to rise slightly in the second month with the all the services continually running.
Is it motivation to keep grinding and start bringing in revenue? Sure is! 😀 Later this year you can hear about more steps I've taken in that direction.
This is very good! Weldone Morgan!
Woo! Great to hear from you on here.
Thanks a bunch, Sunday.