This started less an article or blog, more a journal of my many failed attempts at containerising and deploying a small app. I was keen to practice with the technologies I use every day, but without the guardrails of a fantastic platform team with all the tools and support they provide.
The following describes the (eventual!) successful steps to...
Package a
Kotlin
web application with environment secrets in aDocker
imageUpload the image to a container registry, and run both manually and as part of a
CI build pipeline
Create a
Kubernetes
(k8s) cluster and deploy the app, both manually and as part of the CI build pipelineExpose the application to the internet
The code for the Tempo-Valence application used can be seen here
1. Package app in a docker image
Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format.
-
Using
gcr.io
container registry (after failing with other registry providers - see out takes below!)- Enable
Cloud Registry API
in Google Cloud Platform (GCP) console - Set up Service Account in GCP
Service Accounts - IAM & Admin
so that this can be used to access the Google container registry (gcr). This service account will need to have aStorage Admin
role - Download
key
from created GCP service account and add it to a gitlab-ci variable - calledGOOGLE_CREDENTIALS
in my example, which is accessed in the gitlab-ci.yml file below - (The key can also be saved to a local, non-committed keyfile somewhere if you wish to run the jib docker image locally to check)
- Update build step of
gitlab-ci.yml
to...
- Enable
build:
<<: *gradle-image
stage: build
variables:
CLIENT_KEY: $CLIENT_KEY
GOOGLE_CREDENTIALS: $GOOGLE_CREDENTIALS
before_script:
- echo "$GOOGLE_CREDENTIALS" > keyfile.json
script:
- export CLIENT_KEY="$CLIENT_KEY"
- gradle jib -Djib.console=plain
- gradle clean build
only:
- master
- Add
jib
config inbuild.gradle.kts
of the project as follows...
jib {
container {
mainClass = application.mainClassName
environment = mapOf(Pair("CLIENT_KEY", System.getenv("CLIENT_KEY")))
ports = listOf("8000")
}
from {
image = "openjdk:11"
}
to {
image = "gcr.io/<gcp-project-name>/tempo-valence:latest"
auth {
username = "_json_key"
password = file("keyfile.json").readText()
}
}
}
(The CLIENT_KEY
environment variable above refers to a secret env var required to access an external api from the application. The value of this env var is stored as a Variable in Gitlab CI so the pipeline can access to run the app.)
2. Upload docker image to a container registry
See previous step
To run locally...
docker run --rm -p 9000:9000 gcr.io/<gcp-project-name>/tempo-valence:latest
# in new terminal tab
curl http://localhost:9090
# Welcome to TempoValence!
3. Create a k8s container cluster
Following manual commands create cluster, deploy application and expose on an external IP
gcloud auth list
gcloud config set account m.........@gmail.com
gcloud config set project <gcp-project-name>
gcloud config set compute/zone europe-west2-a
gcloud container clusters create tempo-valence-cluster
gcloud compute instances list
kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest
kubectl get deployment
kubectl get pods
kubectl scale deployment tempo-valence --replicas=2
kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5
kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
# the ports here matter - think it has something to do with how we've set up the application in the image
kubectl get service
# and use the EXTERNAL_IP of tempo-valence-service to access, e.g...
http://35.234.150.122:9000/
4. Deploy
See step 3 for manual deployment
In order to deploy from the gitlab-ci pipeline, the
gitlab-ci.yml
looked like this for the deploy stage...
deploy:
stage: deploy
variables:
CI_SERVICE_ACCOUNT: $CI_SERVICE_ACCOUNT
image:
name: kiwigrid/gcloud-kubectl-helm
before_script:
- echo "$CI_SERVICE_ACCOUNT" > key.json
- gcloud auth activate-service-account --key-file=key.json
- gcloud config set project <gcp-project-name>
- gcloud config set container/cluster tempo-valence-cluster
- gcloud config set compute/zone europe-west2-a
- gcloud container clusters get-credentials tempo-valence-cluster --zone europe-west2-a
script:
- kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest
- kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
# Wait for the deployment to be applied
- kubectl rollout status deployment tempo-valence --watch
Few things to note here!
-
Set up another service account in GCP, to be used by gitlab ci to authenticate against both the container registry and k8s cluster (may be possible just to amend the roles of previous service account, but I added a new one to keep it separate)
- Give the service account 2 project roles ->
Kubernetes Engine Developer
andStorage Object Admin
- Download the key associated to the account and add in to a new CI variable in gitlab
- Give the service account 2 project roles ->
The
image
is the image used to run the deployment - bit of trial and error here, as this image needed to have bothkubectl
andgcloud
cli tools loaded (and I couldn't be bothered to create one and store it somewhere that didn't need authentication!) The image included above is open and does the job.For this to work, the
tempo-valence-cluster
of course needs to be up and running
Steps to run
As this will not be continually running here are the steps needed before this will be available on the internet
To setup
gcloud container clusters create tempo-valence-cluster
# Run ci pipeline, or...
kubectl create deployment tempo-valence --image=gcr.io/<gcp-project-name>/tempo-valence:latest
kubectl scale deployment tempo-valence --replicas=2
kubectl autoscale deployment tempo-valence --cpu-percent=80 --min=1 --max=5
kubectl expose deployment tempo-valence --name=tempo-valence-service --type=LoadBalancer --port 9000 --target-port 9000
To teardown
kubectl delete service tempo-valence-service
gcloud container clusters delete tempo-valence-cluster
Links and article credits
A lot of ideas from this Google k8s tutorial
And this GCP tutorial
One of our own from Matt Dowds on Kubernetes deployment form Gitlab CI
Outtakes - my many failed attempts at step 1!
Package app in a docker image
- Google Kubernetes Engine (GKE) accepts Docker images as the application deployment format. To build a Docker image, you need to have an application and a Dockerfile (in that application)
- Add
Dockerfile
to root of project. 1st attempt looked like this...
# use base image
FROM openjdk:11
# set working directory
WORKDIR /
# copy the runlocal.sh file (contains env vars and gradle run command)
COPY runlocal.sh .
# Inform Docker that the container is listening on the specified port at runtime
EXPOSE 8080
#Run the specified command within the container
RUN ./runlocal.sh
# Copy the rest of your app source code from your host to your image filesystem
COPY . .
- Run the following command to build and test docker image locally...
docker image build -t tempo-valence:1.0 .
Sadly this exited in error...
Step 5/6 : RUN ./runlocal.sh
---> Running in 7d1ddc474a6f
./runlocal.sh: line 5: ./gradlew: No such file or directory
The command '/bin/sh -c ./runlocal.sh' returned a non-zero code: 127
- So next I tried working with a gradle base image...
FROM gradle:5.4.1-jdk11 AS build
Gradle started, but sadly a config error was returned...
FAILURE: Build failed with an exception.
* What went wrong:
A problem occurred configuring root project ''.
- Then looked in to
--env-file
and using other ways of getting the relevant client key into the image without exposing it. Finally added the env_var to a.env
file and accessed inDockerfile
as follows...
FROM openjdk:11
WORKDIR /
EXPOSE 8080
ENV CLIENT_KEY .env
COPY . .
This successfully builds and tags the image, but my concern is now that this .env
file is only available locally, as it is .gitignore'd
- Running the image as a container using
docker container run --publish 8000:8080 --detach --name tv tempo-valence:1.0
# --publish 8000:8080 asks Docker to forward traffic incoming on the host’s port 8000, to the container’s port 8080. Containers have their own private set of ports, so if you want to reach one from the network, you have to forward traffic to it in this way. Otherwise, firewall rules will prevent all network traffic from reaching your container, as default
# --detach asks Docker to run this container in the background
# --name specifies a name with which you can refer to your container in subsequent commands, in this case tv
didn't run the app, as we have no current run command in the Dockerfile
So, added a couple of bits back in the Dockerfile
...
FROM gradle:5.4.1-jdk11
WORKDIR /
EXPOSE 8080
ENV CLIENT_KEY $CLIENT_KEY
CMD [ "gradle", "run" ]
COPY . .
This successfully runs the container, but I cannot access the app via the browser on http://localhost:8000
as expected
Some links being used so far...
- https://docs.docker.com/get-started/part2/
- https://www.reddit.com/r/docker/comments/7dhszj/dockercompose_how_do_i_expose_a_port_to_my/
Some useful commands...
-
docker exec tv env
(lists env vars set in the container)
New attempts... (13.03.20)
Tried using
jib
to containerise the appAdded the following into
build.gradle
file...
plugins {
id 'com.google.cloud.tools.jib' version '2.1.0'
}
jib {
container {
mainClass = application.mainClassName
environment = [CLIENT_KEY:System.getenv("CLIENT_KEY")]
ports = ['8000']
}
from {
image = "openjdk:11"
}
to {
image = "matttea/matttea-images/tempo-valence:latest"
}
}
- Run the following commands to build...
export CLIENT_KEY=<client_key_value>
gradle jib
- Returned error related to authentication...
Containerizing application to matttea/matttea-images/tempo-valence...
Base image 'openjdk:11' does not use a specific image digest - build may not be reproducible
The credential helper (docker-credential-desktop) has nothing for server URL: registry-1.docker.io
Got output:
credentials not found in native keychain
...
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build image failed, perhaps you should make sure your credentials for 'registry-1.docker.io/matttea/matttea-images/tempo-valence' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help
Using this link
New attempts... (25.05.2020)
tried changing job image to detail to
image = "https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest"
based ondocker logout
command url responsedidn't work...
> Task :jib FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':jib'.
> Invalid image reference https://index.docker.io/v1//matttea/matttea-images/tempo-valence:latest, perhaps you should check that the reference is formatted correctly according to https://docs.docker.com/engine/reference/commandline/tag/#extended-description
For example, slash-separated name components cannot have uppercase letters
(Originally posted on https://logical-progression.matttea.com)
Top comments (0)