Hello! I haven't posted to Dev.to in a while. I have been learning more about Devops, Devsecops, and how I want to navigate my cybersecurity career.
I did some hands-on CI/CD by setting up my own pipeline in my local hosted Virtual Machines. This is sort of my documentation of what I did, a tutorial for others, but more importantly to guide my future self should I want to build something like this again.
To preface, I am a cybersecurity student from Singapore. Apart from school projects, internships, and a couple of side hustles, I do not have true web development/ system adminstration experience in a real-world, large scale production environment. My information may not be 100% accurate; I may be deviating from best practices. Nevertheless, I know that it works. I couldn't find information on this specific permutation of Gitlab and Docker so I hope my sharing would be useful for others who want to build something similar. Let me know if there's anything I could have done better!
The infrastructure is actually very simple. It's just 2 docker containers running inside an ubuntu VM. You could even run the 2 docker containers inside Windows. Just need to configure slightly differently. The containers are even pulled from Dockerhub. What made this difficult for me was the configuration files. There's small details that I had to dig from several sources to get my pipeline working.
If running in a VM, you might want to allocate more resources. My Gitlab was quite laggy. I suspect its hardware ... If your Gitlab starts becoming unresponsive, just restart the machine. Avoid pausing the VM too.
Apart from that, you just need Docker.
It's very simple with apt on Ubuntu. Steps here
You could also use Docker Desktop for a GUI process however most guides and documentation online give steps through CLI so I used CLI too.
After installing Docker, you might realise that most Docker commands require sudo. If you do not want to keep typing in sudo:
sudo usermod -aG docker $USER
HOWEVER this opens up a privilege escalation security vulneraibility. This entire write up is for home testing and learning. This is NOT for production.
That's all! On to the actual installation of Gitlab!
Gitlab has a CE and EE version (community and enterprise). Both are free to download. I used the EE version. Download the Gitlab EE image from Dockerhub
docker pull gitlab/gitlab-ee
In a Gitlab CI/CD pipeline, there are jobs. Different machines can run the jobs in whatever permutation needed for load balancing and scaling. I'm not very clear on this feature of Gitlab CI/CD but I know that to run a job you need a runner. As quoted in Runner documentation,
For security and performance reasons, you should install GitLab Runner on a machine that is separate to the machine that hosts your GitLab instance
I'm not too sure about the specifics, but I just wanted a single runner to run a simple pipeline. Based on this thread, it seems okay for my use case to run both the Gitlab container and the Runner container on the same machine.
So that's what I did.
docker pull gitlab/gitlab-runner
Now this is the part where I couldn't find a single source of answers.
From Gitlab documentation, first create a directory for the Gitlab files:
Then create a file named
docker-compose.yml (actually you can name it anything you want but this step makes some processes more convenient). Docker compose files allow you to write whatever configuration you want before running the container. Multiple containers can be run with one single compose file too. But I'm not too sure about the best practices of this. I believe IAC tools such as Ansible can be used too but for my testing I just used one compose file for one container.
I don't remember exactly what each config is for but I can roughly explain a few key ones:
volumes this is to make storage persistent
restart I think this is to run the container on boot
ports apart from the common ports, I believe port 8093 is for session_server.
shm_size the Gitlab app is very resource heavy so apparently this increases RAM allocation... I think
Next, in a separate directory, create another
Then in the Gitlab container directory and the Runner directory:
docker compose up -d
Gitlab takes some time to start. To see the status of the containers:
Next, you need to access the Gitlab app's config files. You could try open an editor within the container or do what I did lol. Install the docker extention in VS code.
etc/gitlab/gitlb.rb and look for the
external_url. For home testing/learning you can use your own IP address.
hostname -I. Read here
If everything goes well, open your web browser and enter your IP address. You should see the Gitlab login. The default username should be
root while the password can be found in
etc/gitlab/initial_root_password. You can change the password through the web GUI afterwards. Once logged in, click click click ~ create your project.
The Gitlab app guides you on the runner registration. Just click new runner registration and follow the steps. Allow the runner to run untagged jobs. This is just for convenience. Next screen shows you the steps to continue.
To run commands in the Runner app:
docker exec -it gitlab-runner bash
With that, you can start testing your CI/CD pipeline script! Perhaps try some devsecops tools. Snyk? Synopsys?
Thank you for reading my post :) I am still learning so please let me know if I can do this better! I might just be spreading misinformation lol