In this tutorial we will build a ci pipeline using concourse to build and push a image to dockerhub automatically, whenever a new git commit is made to the master branch.
Our Project Setup
Our Directory Tree:
$ find .
./Dockerfile
./ci
./ci/pipeline.yml
./README.md
./docker-tunnel
The project used in this example is not important, but you can check it out at https://github.com/ruanbekker/docker-remote-tunnel
Our Pipeline
A visual to see how the pipeline will look like in concourse:
Our pipeline definition will consist of 3 resources, github repo, dockerhub image and a slack resource to inform use whether a build has completed.
Then we are specifying that the job should be triggered on a git commit for the master branch, build and push to our dockerhub repo.
Our pipeline definition ci/pipeline.yml:
resources:
- name: git-repo
type: git
source:
uri: git@github.com:ruanbekker/docker-remote-tunnel.git
branch: master
private_key: ((github_private_key))
- name: docker-remote-tunnel-image
type: docker-image
source:
repository: ruanbekker/docker-remote-tunnel
tag: test
username: ((dockerhub_user))
password: ((dockerhub_password))
- name: slack-alert
type: slack-notification
source:
url: ((slack_notification_url))
resource_types:
- name: slack-notification
type: docker-image
source:
repository: cfcommunity/slack-notification-resource
tag: v1.3.0
jobs:
- name: build-cached-image
plan:
- get: git-repo
trigger: true
- task: build-cached-image-workspace
config:
platform: linux
image_resource:
type: docker-image
source:
repository: rbekker87/build-tools
outputs:
- name: workspace
inputs:
- name: git-repo
run:
path: /bin/sh
args:
- -c
- |
output_dir=workspace
cat << EOF > "${output_dir}/Dockerfile"
FROM alpine
ADD git-repo /tmp/git-repo
RUN mv /tmp/git-repo/docker-tunnel /usr/bin/docker-tunnel
RUN apk --no-cache add screen docker openssl openssh-client apache2-utils
RUN /usr/bin/docker-tunnel -h
RUN rm -rf /tmp/git-repo
EOF
cp -R ./git-repo "${output_dir}/git-repo"
- put: docker-remote-tunnel-image
params:
build: workspace
on_failure:
put: slack-alert
params:
channel: '#system_events'
username: 'concourse'
icon_emoji: ':concourse:'
silent: true
text: |
*$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) FAILED to build image
https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
on_success:
put: slack-alert
params:
channel: '#system_events'
username: 'concourse'
icon_emoji: ':concourse:'
silent: true
text: |
*$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) SUCCESS - Image has been published
https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
- name: test
plan:
- get: docker-remote-tunnel-image
passed: [build-cached-image]
trigger: true
- get: git-repo
passed: [build-cached-image]
- task: run-tests
image: docker-remote-tunnel-image
config:
platform: linux
inputs:
- name: git-repo
run:
dir: git-repo
path: sh
args:
- /usr/bin/docker-tunnel
- --help
on_failure:
put: slack-alert
params:
channel: '#system_events'
username: 'concourse'
icon_emoji: ':concourse:'
silent: true
text: |
*$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) FAILED - Testing image failure
https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
on_success:
put: slack-alert
params:
channel: '#system_events'
username: 'concourse'
icon_emoji: ':concourse:'
silent: true
text: |
*$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) SUCCESS - Testing image Succeeded
https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
Note that our secret information is templatized and saved in our local credentials.yml which should never be stored in version control:
slack_notification_url: https://api.slack.com/aaa/bbb/ccc
dockerhub_user: myuser
dockerhub_password: mypasswd
github_private_key: |-
-----BEGIN RSA PRIVATE KEY-----
some-secret-data
-----END RSA PRIVATE KEY------
Set the Pipeline:
Now that we have our pipeline definition, credentials and application code (stored in version control), go ahead and set the pipeline, which will save the pipeline configuration in concourse:
# pipeline name: my-docker-app-pipeline
$ fly -t scw sp -n main -c pipeline.yml -p my-docker-app-pipeline -l credentials.yml
Now the pipeline is saved on concourse but in a paused state, go ahead and unpause the pipeline:
$ fly -t scw up -p my-docker-app-pipeline
Test your Pipeline
Make a commit to master and head over to concourse and look at it go:
Thanks for reading, make sure to check out my other posts on #concourse
Top comments (2)
Nice post! I recently had to do this and having more resources like this available would have been helpful. 🙌
Have you played around with this new OCI registry image resource type/build task at all?
I haven't, but I think the goal of eliminating the Docker daemon (and eventually privileged containers as well) requirement is a noble one.
Running Docker within Concourse's Garden containers can be rough sometimes. 😔
Hey Tim,
I have not played around with them, but thanks for mentioning them, those look really useful. Think I will start to use them right away.
Totally agree about moving away from the docker daemon is a great one!
Thanks so much for the comment.