Continuous integration (CI) is one important part of the software development process where a shared code repository is continuously changing due to new work of a team member being integrated into it.
To ensure the high quality of the code and reduce potential errors, each integration is usually verified by an automated build and test process.
In this article, we will learn how to setup that process using Github Action to automatically build and run unit tests for our simple bank project, which is written in Golang and uses PostgreSQL as its main database.
Github Action is a service offered by Github that has similar functionality as other CI tools like Jenkins, Travis, or CircleCI.
In order to use Github Actions, we must define a workflow. Workflow is basically an automated procedure that’s made up of one or more jobs. It can be triggered by 3 different ways:
- By an event that happens on the Github repository
- By setting a repetitive schedule
- Or manually clicking on the run workflow button on the repository UI.
To create a workflow, we just need to add a
.yml file to the
.github/workflows folder in our repository. For example, this is a simple workflow file
name: build-and-test on: push: branches: [ master ] schedule: - cron: '*/15 * * * *' jobs: build: runs-on: ubuntu-latest
The name of this workflow is
build-and-test. We can define how it will be triggered using the
In this flow, there's an event that will trigger the workflow whenever a change is pushed to the master branch, and another scheduled trigger that will run the workflow every 15 minute.
Then we define the list of jobs to run in the
jobs section of the workflow yaml file.
In order to run the jobs, we must specify a runner for each of them. A runner is simply a server that listens for available jobs, and it will run only 1 job at a time.
We can use Github hosted runner directly, or specify our own self-hosted runner.
The runners will run the jobs, then report the their progress, logs, and results back to Github, so we can easily check it on the UI of the repository.
We use the
run-on keyword to specify the runner we want to use.
jobs: build: runs-on: ubuntu-latest
In this example workflow, we’re using Github’s hosted runner for Ubuntu’s latest version.
Now let’s talk about Job. A job is a set of steps that will be executed on the same runner.
Normally all jobs in the workflow run in parallel, except when you have some jobs that depend on each other, then they will be run serially.
The jobs are listed inside the workflow under the
jobs: build: runs-on: ubuntu-latest steps: - name: Check out code uses: actions/checkout@v2 - name: Build server run: ./build_server.sh test: needs: build runs-on: ubuntu-latest steps: - run: ./test_server.sh
In this example, we have 2 jobs:
- The first one is
build, which has 2 steps: check out code, and build server.
- The second job is
test, which will run the tests of the application.
Here we use the
needs keyword to say that the
test job depends on the
build job, so that it can only be run after our application is successfully built.
test job only has 1 step that runs the
Steps are individual tasks that run serially, one after another within a job. A step can contain 1 or multiple actions.
Action is basically a standalone command like the one that run the
test_server.sh script that we’ve seen before. If a step contains multiple actions, they will be run serially.
An interesting thing about action is that it can be reused. So if someone has already written a github action that we need, we can actually use it in our workflow.
Let’s take a look at this example.
jobs: build: runs-on: ubuntu-latest steps: - name: Check out code uses: actions/checkout@v2 - name: Build server run: ./build_server.sh
Here we use the
steps keyword to list out all steps we want to run in our job.
The first step is to check out the code from Github to our runner machine. To do that, we just use the Github actions
checkout@v2, which has already been written by the Github action team.
The second step is to build our application server. In this case, we provide our own action, which is simply running the
build_server.sh script that we’ve created in the repository.
And that’s it!
Before jumping in to coding, let’s do a quick summary:
- We can trigger a workflow by 3 ways: event, scheduled, or manually.
- A workflow consists of one or multiple jobs.
- A job is composed of multiple steps.
- Each step can have 1 or more actions.
- All jobs inside a workflow normally run in parallel, unless they depend on each other, then in that case, they run serially.
- Each job will be run separately by a specific runner.
- The runners will report progress, logs, and results of the jobs back to github. And we can check them directly on Github repository’s UI.
Alright, now let’s learn how to setup a real workflow for our Golang application so that it can connect to Postgres, and run all the unit tests that we’ve written in previous lectures whenever new changes are pushed to Github.
In our simple bank repository on Github, let’s select the Actions tab.
Github knows that our project is written mainly in
Go, so it suggests us to setup the workflow for
Go. Let’s click this setup button.
As you can see, a new file
go.yml is being created under the folder
.github/workflows of our repository with this template:
name: Go on: push: branches: [ master ] pull_request: branches: [ master ] jobs: build: name: Build runs-on: ubuntu-latest steps: - name: Set up Go 1.x uses: actions/setup-go@v2 with: go-version: ^1.13 id: go - name: Check out code into the Go module directory uses: actions/checkout@v2 - name: Get dependencies run: | go get -v -t -d ./... if [ -f Gopkg.toml ]; then curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh dep ensure fi - name: Build run: go build -v . - name: Test run: go test -v .
We can edit this file directly here using this Github editor. However, I prefer to add the file to our local repository first, then edit it locally with visual studio code before pushing to Github.
So let’s open our simple bank project folder in the terminal. I’m gonna create a new folder
❯ cd ~/Projects/techschool/simplebank ❯ mkdir -p .github/workflows
Then create a new Yaml file for our workflow inside this folder. You can name it whatever you want, just make sure it has
yml extension. For me, I’m just gonna use
ci.yml to be simple.
❯ touch .github/workflows/ci.yml
Now let’s open this project in visual studio code.
Here we can see the
ci.yml file under
.github/workflows folder. Let’s go back to Github and copy the
go.yml file content, then paste it to our
First we need to set a name for this workflow, for example:
ci-test. This name will be displayed in our Github repository’s Actions page.
Then we define the events that can trigger this workflow. Normally we would want to run tests whenever there’s a change being pushed to the master branch, or when there’s a pull request to merge into the master branch.
on: push: branches: [ master ] pull_request: branches: [ master ]
There are many other events that you can use. Please refer to the Github Actions documentation to know more about them.
Next we’re gonna setup the jobs. In this template that Github provides us, we have only 1 job.
Its name is
build, and it runs on a Ubuntu runner. I think we should rename this job to
test because that’s the main purpose of it.
jobs: test: name: Test runs-on: ubuntu-latest
There are several steps in this job.
Step 1: Install Go
The first step is to setup or install Go into the runner. In this step, we just need to use the existing Github action called
steps: - name: Set up Go 1.x uses: actions/setup-go@v2 with: go-version: ^1.15 id: go
We use the
withkeyword to provide input parameters to this action. In this case, we can ask it to use a specific version of Go, such as version
idfield is just a unique identifier of this step. We might need it if we want to refer to this step in other context.
Step 2: Checkout code
The second step is to check out the code of this repository into the runner. To do that, we also reuse an existing action:
- name: Check out code into the Go module directory uses: actions/checkout@v2
Step 3: Get dependencies
The next step is to get all the dependencies, or external packages that our project is using.
- name: Get dependencies run: | go get -v -t -d ./... if [ -f Gopkg.toml ]; then curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh dep ensure fi
In fact, we don’t need this step because go mod will automatically download missing libraries when we build the application or run the tests. So let's remove it!
The build step is also not necessary because the application will be built automatically when we run go test.
- name: Build run: go build -v .
Step 4: Run the tests
So the last remaining step is to run our unit tests. We already have a
make testcommand defined in the Makefile for this purpose. Therefore, all we have to do in this step is to call it:
- name: Test run: make test
So we’re done with the first basic version of our CI workflow:
name: ci-test on: push: branches: [ master ] pull_request: branches: [ master ] jobs: test: name: Test runs-on: ubuntu-latest steps: - name: Set up Go 1.x uses: actions/setup-go@v2 with: go-version: ^1.15 id: go - name: Check out code into the Go module directory uses: actions/checkout@v2 - name: Test run: make test
It might not work yet because we haven’t setup the Postgres database. But let’s just push it to Github to see how it run:
❯ git status ❯ git add . ❯ git commit -m "init CI workflow"
Here we first run
git status to check the status of our local repository. After that, we run
git add . to add all new changes to the list of our commit.
git commit with a message saying
init CI workflow to commit it to our local repository. And finally, we run
git push origin master to push this change to our remote repository on Github.
Now let's go back to our Github repository page and select Actions tab.
Now we can see our
ci-test workflow here, and there’s a new run of it for our commit. When we open this run, we can see 1 job in progress:
All steps are listed on the right. The
Setup Go, and
Checkout code steps are finished successfully because there’s a green tick in front of them. The
Test step is still running because there’s a yellow circle before it.
Test step has finished, but it failed. We know that because of the red
x icon next to it.
This is expected, because as we’re seeing in the logs, the code cannot connect to port
5432 of Postgres, since we haven’t set it up in our workflow yet. So let’s do that now!
Let’s search for
github action postgres, and open this official Github Action documentation page about creating Postgres service containers.
Here in this section, we can see that Postgres is declared as an external service of this job. Let’s copy this block of code and paste it to our workflow file.
So we use the
services keyword to specify a list of external services that we want to run together with our job. In this case, we only need 1 service, which is Postgres.
And since we’re using Postgres version 12 in our project, let’s set this docker image name to
postgres:12. You can check out available versions and tags of this Postgres image on Docker Hub.
services: postgres: image: postgres:12 env: POSTGRES_USER: root POSTGRES_PASSWORD: secret POSTGRES_DB: simple_bank options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
Next we need to set some environment variables for the credentials to access the database.
If you still remember, we’re using
user = "root",
password = "secret", and
database = "simple_bank" in our local Postgres container. So let’s set the same value here for our CI workflow.
The health check option is very important because it is used by the runner to check if Postgres has started successfully or not, so that it can know when to run the next steps in the workflow.
That’s great because we only want our tests to be run after Postgres is started. Otherwise, the tests will still fail because it cannot connect to the database, right?
OK, now the Postgres service is defined, but in order for our tests to run successfully, we also need to run db migrations to create the correct database schema for our application.
So let’s define a new step here, after the check out code step. Its name will be
Run migrations. And the only action it needs to do is to run
- name: Run migrations run: make migrateup
Alright, now let’s try to push this new workflow changes to Github to see what will happen.
OK, now in our repository’s Actions page, we can see a new run for our new commit. Here the Test job is still running. The job is set up successfully, and now it’s initializing the containers.
From the logs, we know that it’s still waiting for Postgres service to be ready. As soon as Postgres is up, all following steps are run immediately.
Here we can see a log saying
Postgres service is healthy. The Setup Go step is also successful. Then it checkout new the code.
Now the migrations step is failing because migrate is not found. We forgot to install the
golang-migrate CLI tool to run the migrations.
So let’s search for
golang migrate, and open this Github page documentation.
There are several options depending on the OS that you use. We’re using Ubuntu for our runner, so I’m gonna copy this curl command to download a pre-built binary of the migrate CLI.
❯ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz
Now in the workflow, let’s add a new Step to
Install golang migrate. Then in the run action, let’s paste in the curl command.
We have to set the correct URL for the version of migrate CLI and the platform that we want to use. So let’s click on this Release downloads link.
The latest release is version
4.12.2. And since our ubuntu runner is a linux platform, let’s copy the
migrate.linux-amd64.tar.gz link address, then paste it to our curl command.
- name: Install golang-migrate run: curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
It will download the zip file, and unzip it to give us the migrate binary named
migrate.linux-amd64. Now in order for the migrate command to work, we have to move that binary to the
So this step will include more than just 1 curl command. We use this vertical pipe
| character here to specify a multi-line commands. Let's add this move command to the step:
- name: Install golang-migrate run: | curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz sudo mv migrate.linux-amd64 /usr/bin/ which migrate
Note that only a superuser can change the content of the
/usr/bin folder, so we have to run this command with
We also add 1 more command:
which migrate, just to check if the migrate CLI binary is successfully installed and ready to be used in the runner or not.
Now let's commit the new change of our workflow and push it to Github. Then check our repository’s Actions page.
Now the job is still failing, but this time it fails at the
Install golang-migrate step.
From the logs, we can say that the binary file was successfully downloaded. So it might fail because of the move command, or the which migrate command.
OK I know why! That’s because we’re just moving the file
/usr/bin, but we don’t rename it to
migrate. So when we run
which migrate, it cannot find any binary with that name.
All we have to do now is to add
migrate to the end of the move command, so that the binary file is moved to
/usr/bin with a new name:
- name: Install golang-migrate run: | curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz sudo mv migrate.linux-amd64 /usr/bin/migrate which migrate
This will ensure that when the
make migrateup command is run, the correct
migrate CLI binary will be used.
Alright, let’s add this new change, commit it, and push it to Github. Then go back to our Repository’s Action page to check the job’s status.
It's still failing. However, this time, the
Install golang-migrate step is successful. The step that fails is
And the reason is: it still cannot connect to port
5432 of our Postgres container. Why? We’ve already added Postgres to the services list right?
Well, yes! But we haven’t exposed its local port to the external host yet. That’s why our code still cannot connect to the port.
We can use the
ports keyword to specify the ports that we want to expose to the external host, just like what we normally do in our docker compose file. Let’s add it to our CI workflow.
services: postgres: image: postgres:12 env: POSTGRES_USER: root POSTGRES_PASSWORD: secret POSTGRES_DB: simple_bank ports: - 5432:5432 options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
The default port
5432 of Postgres is now available to our job runner to access.
Let’s add this new change, commit it, and push the change to Github. Hopefully this time it will work.
It’s done! All green ticks. So finally our CI-test workflow runs successfully.
After all steps in our workflow are completed, Github do some clean up steps and stop the containers.
And that’s it! We have learned about continuous integration by writing our first Github Action workflow to run Golang unit tests that need to connect to an external Postgres service.
Here's the complete workflow file
name: ci-test on: push: branches: [ master ] pull_request: branches: [ master ] jobs: test: name: Test runs-on: ubuntu-latest services: postgres: image: postgres:12 env: POSTGRES_USER: root POSTGRES_PASSWORD: secret POSTGRES_DB: simple_bank ports: - 5432:5432 options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 steps: - name: Set up Go 1.x uses: actions/setup-go@v2 with: go-version: ^1.15 id: go - name: Check out code into the Go module directory uses: actions/checkout@v2 - name: Install golang-migrate run: | curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz sudo mv migrate.linux-amd64 /usr/bin/migrate which migrate - name: Run migrations run: make migrateup - name: Test run: make test
There are a lot of more things that Github Action can do. I encourage you to check out its official documentation to learn more about them.
And that brings us to the end of this article. Thanks a lot for reading, and I will see you guys in the next lecture!