This tutorial shows you how to deploy and automate a Docker container using Pulumi in Google Cloud Platform's Cloud Run with minimum permissions.
Here's the GitHub Repository for reference:
https://github.com/superjose/deploy-to-cloud-run-go
Get the CLIs:
1) Create a Google Cloud Account.
4) Install Docker - We need it to build the Docker container.
Bootstrap the project:
5) Create a pulumi
directory.
6) Run pulumi new go
to initialize a pulumi project with Go (It can be your language of choice).
7) Navigate to the pulumi directory.
8) Navigate to cloud.google.com and create a new project.
Take note of the project-id
(Usually the project's name).
Generate the necessary permissions using gcloud CLI
9) Login with the Google Auth CLI:
gcloud auth login
Open the link that will show up and finish logging in.
Enable the service usage API
gcloud services enable serviceusage.googleapis.com
10) (Optional) Set the project in google cloud CLI (Can be changed anytime). This saves you from passing --project [PROJECT-ID]
into every gcloud
command.
If your machine has multiple GCP projects, skip this step and pass the --project
flag into every gcloud
command.
11) Create a service account (The account that Pulumi will connect to):
gcloud iam service-accounts create pulumi-gcp --description="Pulumi GCP"
12) Download the credentials for the service accounts and store them locally (Remember to replace [PROJECT-ID]
with your GCP Project Id):
gcloud iam service-accounts keys create ~/keys/gcp/pulumi-service-account-key-file.json --iam-account=pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com
13) Set Pulumi's gcp credentials config path:
(This will connect the service account with Pulumi)
pulumi config set gcp:credentials ~/keys/gcp/pulumi-service-account-key-file.json
14) Set the GCP Project by doing:
pulumi config set gcp:project [PROJECT-ID]
15) Create a roles.gcp.yml
file (Inside the pulumi
dir) and add the required permissions in includedPermissions
:
# https://cloud.google.com/iam/docs/creating-custom-roles#creating
# Yaml to define the Pulumi GCP Roles that need to be created with gcloud CLI
title: Pulumi GCP Roles
description: |
This policy ensures that all GCP roles are created using Pulumi.
stage: GA
# https://cloud.google.com/iam/docs/permissions-reference
includedPermissions:
- serviceusage.services.list
- serviceusage.services.enable
- serviceusage.services.disable
- serviceusage.services.get
- serviceusage.services.use
# Permissions for GCR
- storage.objects.create
- storage.objects.delete # Optional: only include if you need to delete images
- storage.objects.get
# Permissions for Google Artifact Registry
- artifactregistry.repositories.create
- artifactregistry.repositories.delete
- artifactregistry.repositories.get
- artifactregistry.repositories.list
- artifactregistry.repositories.update
- artifactregistry.repositories.downloadArtifacts
- artifactregistry.repositories.uploadArtifacts
- artifactregistry.repositories.deleteArtifacts
# Permissions
# Permissions
- run.services.create
- run.services.get
- run.services.list
- run.services.update
- run.services.delete
- run.services.getIamPolicy
- run.services.setIamPolicy
- iam.serviceAccounts.actAs
# NOTE: This should be removed the first time you're creating a role.
# This etag is to update the current active role (As GCP lets you manage multiple roles)
# I'm commenting it out so I can always replace the role
# etag: BwYS74Xx5y4=
16) Create the pulumi_admin_role
with the file above:
(We assume we're running this code from the pulumi
directory)
gcloud iam roles create pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml'
17) In case you need to make edits, change the file and use:
gcloud iam roles update pulumi_admin_role --project=[PROJECT-ID] --file='./roles.gcp.yml'
18) We're also adding the serviceAccountAdmin
role (I haven't found a better way) (Otherwise we'd get 403 errors when refreshing and updating in Pulumi)1
gcloud projects add-iam-policy-binding [PROJECT-ID] --role roles/iam.serviceAccountAdmin --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com
gcloud projects add-iam-policy-binding [PROJECT-ID] --role projects/[PROJECT-ID]/roles/pulumi_admin_role --member serviceAccount:pulumi-gcp@[PROJECT-ID].iam.gserviceaccount.com
19) Our main.go
in the pulumi
directory:
(Check the code for comments!)
- We enable the required services (Artifact Registry, and Cloud Run).
- Artifact Registry is used to host the docker container image.
- Cloud Runner will launch the Docker image from Artifact Registry.
- We build the docker image locally (We specify the platform in case you're using an ARM chip like M1, M2, Snapdragon SQ, X Elite, etc.)
- We create a chain of "DependsOn" to notify Pulumi: 5.1 Services need to be enabled first 5.2 We create the Artifact Repository 5.3 We build the docker image and push it to Artifact Registry. 5.4 We pull the Docker Image from Artifact Registry and run it. 5.5 We add IAM permissions so it can be accessed from anywhere.
package main
import (
"errors"
"log"
"os"
"github.com/joho/godotenv"
"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/artifactregistry"
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/cloudrun"
"github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/projects"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
// This is the name you created in Google Cloud Platform (GCP).
const gcpProjectId = "deploy-to-cloud-run-go"
// The Docker Image Name
const dockerImageName = "my-app-docker"
const artifactRegistryServiceName = "artifact-registry-api"
const artifactRegistryRepoName = "my-app-artifact-repo"
const artifactRegistryRepoLocation = "us-east1"
const cloudRunAdminServiceName = "cloud-run-admin-service"
const cloudRunServiceName = "cloud-run-service"
// For more info: https://cloud.google.com/run/docs/locations
const cloudRunLocation = "us-east1"
// The tag for the Docker image
const imageTag = "latest"
// This is a url like: us-east1-docker.pkg.dev
// It is used to push the Docker image to Google Container Registry
// For more info: https://cloud.google.com/container-registry/docs/pushing-and-pulling
// The format is: <region>-docker.pkg.dev
var dockerGCPServer = cloudRunLocation + "-docker.pkg.dev"
// The full path to the Docker image
// It is used to deploy the Docker image to Google Cloud Run
// The format is: <region>-docker.pkg.dev/<project-id>/<repo-name>/<image-name>:<tag>
// For more info: https://cloud.google.com/run/docs/deploying
// Example: us-east1-docker.pkg.dev/deploy-to-cloud-run-go/my-app--artifact-repo/my-app-docker:latest
var dockerImageWithPath = dockerGCPServer + "/" + gcpProjectId + "/" + artifactRegistryRepoName + "/" + dockerImageName + ":" + imageTag
func main() {
// Load the .env file
err := godotenv.Load()
if err != nil {
log.Fatal("Error loading .env file")
}
pulumi.Run(func(ctx *pulumi.Context) error {
enabledServices, serviceResultErr := enableServices(ctx)
if serviceResultErr != nil {
return serviceResultErr
}
artifactRegistryRepo, createArtifactErr := createArtifactRegistryNewRepository(ctx, &enabledServices)
if createArtifactErr != nil {
return createArtifactErr
}
dockerImage, buildAndPushErr := buildAndPushToContainerRegistry(ctx, &enabledServices, artifactRegistryRepo)
if buildAndPushErr != nil {
return buildAndPushErr
}
deployContainerErr := deployContainerToCloudRun(ctx, &enabledServices, dockerImage)
if deployContainerErr != nil {
return deployContainerErr
}
return nil
})
}
type EnabledServices struct {
CloudRunService *projects.Service `pulumi:"cloudRunService"`
ArtifactRegistryService *projects.Service `pulumi:"artifactRegistryService"`
}
func enableServices(ctx *pulumi.Context) (EnabledServices, error) {
cloudResourceManager, cloudResourceErr := projects.NewService(ctx, "cloud-resource-manager", &projects.ServiceArgs{
Service: pulumi.String("cloudresourcemanager.googleapis.com"),
Project: pulumi.String(gcpProjectId),
})
if cloudResourceErr != nil {
return EnabledServices{}, cloudResourceErr
}
cloudRunService, cloudRunAdminErr := projects.NewService(ctx, cloudRunAdminServiceName, &projects.ServiceArgs{
Service: pulumi.String("run.googleapis.com"),
Project: pulumi.String(gcpProjectId),
}, pulumi.DependsOn([]pulumi.Resource{cloudResourceManager}))
if cloudRunAdminErr != nil {
return EnabledServices{}, cloudRunAdminErr
}
artifactRegistryService, err := projects.NewService(ctx, artifactRegistryServiceName, &projects.ServiceArgs{
Service: pulumi.String("artifactregistry.googleapis.com"),
}, pulumi.DependsOn([]pulumi.Resource{cloudResourceManager}))
if err != nil {
return EnabledServices{}, err
}
return EnabledServices{
CloudRunService: cloudRunService,
ArtifactRegistryService: artifactRegistryService,
}, nil
}
func createArtifactRegistryNewRepository(ctx *pulumi.Context, enabledServices *EnabledServices) (*artifactregistry.Repository, error) {
if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
return nil, errors.New("enabledServices cannot be nil")
}
dependingResources := []pulumi.Resource{
enabledServices.ArtifactRegistryService,
}
repo, err := artifactregistry.NewRepository(ctx, artifactRegistryRepoName, &artifactregistry.RepositoryArgs{
Location: pulumi.String(artifactRegistryRepoLocation),
RepositoryId: pulumi.String(artifactRegistryRepoName),
Format: pulumi.String("DOCKER"),
Description: pulumi.String("The repository that will hold social-log Docker images."),
}, pulumi.DependsOn(dependingResources))
if err != nil {
return nil, err
}
return repo, nil
}
func buildAndPushToContainerRegistry(ctx *pulumi.Context, enabledServices *EnabledServices, artifactRegistryRepo *artifactregistry.Repository) (*docker.Image, error) {
if enabledServices == nil || enabledServices.ArtifactRegistryService == nil {
return nil, errors.New("enabledServices cannot be nil")
}
if artifactRegistryRepo == nil {
return nil, errors.New("artifactRegistryRepo cannot be nil")
}
// Lookup GOOGLE_CREDENTIALS environment variable which should hold the path to the JSON key file
jsonKeyPath, present := os.LookupEnv("GOOGLE_CREDENTIALS_FILE_PATH")
if !present {
return nil, errors.New("GOOGLE_CREDENTIALS_FILE_PATH environment variable is not set")
}
// Read the JSON key file
jsonKey, err := os.ReadFile(jsonKeyPath)
if err != nil {
return nil, err
}
dependingSources := []pulumi.Resource{
enabledServices.ArtifactRegistryService,
artifactRegistryRepo,
}
// Build and push Docker image to Google Container Registry using the JSON key
image, err := docker.NewImage(ctx, dockerImageName, &docker.ImageArgs{
Build: &docker.DockerBuildArgs{
Context: pulumi.String("../"), // Adjust the context according to your project structure
ExtraOptions: pulumi.StringArray{
// This option is needed for devices running on ARM architecture, such as Apple M1/M2/MX CPUs
pulumi.String("--platform=linux/amd64"),
},
},
ImageName: pulumi.String(dockerImageWithPath),
Registry: &docker.ImageRegistryArgs{
Server: pulumi.String(dockerGCPServer),
Username: pulumi.String("_json_key"), // Special username for GCP
Password: pulumi.String(string(jsonKey)), // Provide the contents of the key file
},
}, pulumi.DependsOn(dependingSources))
if err != nil {
return nil, err
}
return image, nil
}
func deployContainerToCloudRun(ctx *pulumi.Context, enabledServices *EnabledServices, dockerImage *docker.Image) error {
if enabledServices == nil || enabledServices.CloudRunService == nil {
return errors.New("enabledServices cannot be nil")
}
if dockerImage == nil {
return errors.New("dockerImage cannot be nil")
}
dependingSources := []pulumi.Resource{
enabledServices.CloudRunService,
dockerImage,
}
appService, err := cloudrun.NewService(ctx, cloudRunServiceName, &cloudrun.ServiceArgs{
Location: pulumi.String(cloudRunLocation), // Choose the appropriate region for your service
Template: &cloudrun.ServiceTemplateArgs{
Spec: &cloudrun.ServiceTemplateSpecArgs{
Containers: cloudrun.ServiceTemplateSpecContainerArray{
&cloudrun.ServiceTemplateSpecContainerArgs{
Image: dockerImage.ImageName,
Resources: &cloudrun.ServiceTemplateSpecContainerResourcesArgs{
Limits: pulumi.StringMap{
"memory": pulumi.String("256Mi"), // Adjust the memory limit as needed
},
},
},
},
},
},
Traffics: cloudrun.ServiceTrafficArray{
&cloudrun.ServiceTrafficArgs{
Percent: pulumi.Int(100),
LatestRevision: pulumi.Bool(true),
},
},
}, pulumi.DependsOn(dependingSources))
if err != nil {
return err
}
_, iamErr := cloudrun.NewIamMember(ctx, "invoker", &cloudrun.IamMemberArgs{
Service: appService.Name,
Location: appService.Location,
Role: pulumi.String("roles/run.invoker"),
Member: pulumi.String("allUsers"),
})
if iamErr != nil {
return iamErr
}
ctx.Export("containerUrl", appService.Statuses.Index(pulumi.Int(0)).Url().ToOutput(ctx.Context()))
return nil
}
18) Update the ENV
in your Dockerfile:
There's a known issue, and here in which you need to export your HOME
environment variable to /root
;
# Set the ENV HOME before your ENTRYPOINT.
ENV HOME=/root
# This is specific to your project.
ENTRYPOINT ["/whatever-is-your-entrypoint"]
20) Create a .env file in the pulumi directory:
Set the **full path* to the one you saved on 11)
GOOGLE_CREDENTIALS_FILE_PATH="/Users/myusername/keys/gcp/pulumi-service-account-key-file.json"
21) Run pulumi up
And you should be up and going!
22) The go.mod
module social-log-go
go 1.21
toolchain go1.22.0
require (
github.com/pulumi/pulumi-gcp/sdk/v7 v7.11.2
github.com/pulumi/pulumi/sdk/v3 v3.108.1
)
require (
dario.cat/mergo v1.0.0 // indirect
github.com/Masterminds/semver v1.5.0 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
github.com/agext/levenshtein v1.2.3 // indirect
github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/blang/semver v3.5.1+incompatible // indirect
github.com/charmbracelet/bubbles v0.16.1 // indirect
github.com/charmbracelet/bubbletea v0.24.2 // indirect
github.com/charmbracelet/lipgloss v0.7.1 // indirect
github.com/cheggaaa/pb v1.0.29 // indirect
github.com/cloudflare/circl v1.3.7 // indirect
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/djherbis/times v1.5.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.5.0 // indirect
github.com/go-git/go-git/v5 v5.11.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/glog v1.1.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/hcl/v2 v2.17.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/mitchellh/go-ps v1.0.0 // indirect
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.15.2 // indirect
github.com/opentracing/basictracer-go v1.1.0 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pgavlin/fx v0.1.6 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pkg/term v1.1.0 // indirect
github.com/pulumi/appdash v0.0.0-20231130102222-75f619a67231 // indirect
github.com/pulumi/esc v0.6.2 // indirect
github.com/pulumi/pulumi-docker/sdk/v3 v3.6.1 // indirect
github.com/rivo/uniseg v0.4.4 // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
github.com/sergi/go-diff v1.3.1 // indirect
github.com/skeema/knownhosts v1.2.1 // indirect
github.com/spf13/cobra v1.7.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/texttheater/golang-levenshtein v1.0.1 // indirect
github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/zclconf/go-cty v1.13.2 // indirect
go.uber.org/atomic v1.9.0 // indirect
golang.org/x/crypto v0.17.0 // indirect
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
golang.org/x/mod v0.14.0 // indirect
golang.org/x/net v0.19.0 // indirect
golang.org/x/sync v0.5.0 // indirect
golang.org/x/sys v0.15.0 // indirect
golang.org/x/term v0.15.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.15.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230706204954-ccb25ca9f130 // indirect
google.golang.org/grpc v1.57.1 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/frand v1.4.2 // indirect
)
If you include this go.mod
Run go tidy
, and this will fetch all the packages for you
Footnotes
1I fought against permissions for 5 days. The serviceAccountAdmin
predefined GCP role brought in the additional permissions needed.
Top comments (0)