DEV Community

Cover image for Setting up Continuous Integration
Amnish Singh Arora
Amnish Singh Arora

Posted on • Edited on

Setting up Continuous Integration

Continuous Integration is a philosophy that emphasizes frequent integration of code changes into a shared repository.

Imagine you're a group of people working on a project at the same time. You're all working at a separate issue and you keep pushing changes to the repository. At the end of the week, everyone meets to build and test the application only to find everything is broken and now you have to spend hours and hours on figuring out where thing went wrong. This is exactly why we need to setup CI in our project.

Continuous Integration

It basically means we need to build and test our code everytime a new change is pushed to the shared branch. By doing this, we make sure that the project is always functioning as expected and regression is prevented.
The key to CI is that instead of relying on humans to run the tests manually before pushing, we delegate this responsibily to machines that are automated to run this workflow, on every push or a pull request to master (or whatever the branch name in your case).
There are many cloud services that solve this purpose for us.

🌟 In this post, I'll be talking about how I setup a CI workflow for my python project til-the-builder using Github Actions.

Table of Contents

 1. Github Actions 🎬
       1.1. Setting up CI ✔️
       1.2. Breakdown of the workflow 🧐
 2. Adding a new test to check CI
 3. Adding tests to other project
       3.3. Setting up ⚙️
       3.4. Adding Unit Tests
 4. Conclusion 🎊

Github Actions 🎬

Github provides us certain events that run configured workflows everytime they are triggered. It could be a commit pushed or a pull request made to a certain branch, or even a tag pushed to the repository.

From the official docs,
Github Actions

Setting up CI ✔️

1. The first step is navigating to the Actions tab on Github.

Actions Tab

2. If you do not have any workflows already configured, you'll see a page with recommended workflow templates for your project.

In my case, I get these since it is a Python project.
Workflow templates

3. I went with the last one called Python Application, and this is what I got as template.

# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

name: Python application

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

permissions:
  contents: read

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python 3.10
      uses: actions/setup-python@v3
      with:
        python-version: "3.10"
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install flake8 pytest
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
    - name: Lint with flake8
      run: |
        # stop the build if there are Python syntax errors or undefined names
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
    - name: Test with pytest
      run: |
        pytest
Enter fullscreen mode Exit fullscreen mode

4. Since I was using pylint as my linter, I made some changes to the lint step and the dependency installation step and ended up with the following yaml configuration.

# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

name: Python application

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

permissions:
  contents: read

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Set up Python 3.10
      uses: actions/setup-python@v3
      with:
        python-version: "3.10.11"

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt

    - name: Lint with pylint
      run: |
        pylint src/

    - name: Test with pytest
      run: |
        pytest

    - name: Format code
      run: |
        black .
Enter fullscreen mode Exit fullscreen mode

Breakdown of the workflow 🧐

Although the workflow is self expanatory, let's take a look at each aspect.

1. The first thing we do is name our workflow, that is showed on in Github actions tab.

name: Python application
Enter fullscreen mode Exit fullscreen mode

2. Next, we define the triggers that fire this workflow.

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]
Enter fullscreen mode Exit fullscreen mode

3. After that, we need to define the jobs that need to be run. We only have a build job at this point, and it runs on ubuntu-latest operating system.

jobs:
  build:
    runs-on: ubuntu-latest
Enter fullscreen mode Exit fullscreen mode

4. Jobs run in parallel and have a certain number of steps that all need to pass in order for the job to pass. The following configuration defined the steps for our build job.

The first step is to fetch the code from our repository to the file system of CI machine.

steps:
    - uses: actions/checkout@v3
Enter fullscreen mode Exit fullscreen mode

Next, we setup the runtime environment out application needs, which is python-3.10.1 in this case.

- name: Set up Python 3.10
      uses: actions/setup-python@v3
      with:
        python-version: "3.10.11"
Enter fullscreen mode Exit fullscreen mode

Once the runtime is configured, we need to install all the packages/dependencies required by our application. For Python, they are typically defined in a requirements.txt file.

- name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Now that the application we have the runtime and dependecies in place, we need to run our linter and tests.

- name: Lint with pylint
      run: |
        pylint src/

- name: Test with pytest
      run: |
        pytest
Enter fullscreen mode Exit fullscreen mode

And that is it for our workflow configuration! The file is added to the .github/workflows directory and we can add as much workflows as we wish to it.

Adding a new test to check CI

Once I was confident that my CI workflow was in place, I added unit tests for another method of HeadingItem class in my project.

test_heading_item.py

from builder.toc_generator.heading_item import HeadingItem


class TestHeadingItem:
    """
    Test suite for 'HeadingItem' class
    """

    class TestHeadingItemConstructor:
        """
        Tests for 'HeadingItem' class __init__ method
        """
        ...
        ...

    class TestHeadingItemGetHtml:
        """
        Tests for 'HeadingItem' class get_html method
        """

        def test_return_type(self):
            """
            get_html returns a string
            """
            sample_heading_value = "This is a sample heading"
            heading = HeadingItem(sample_heading_value)

            assert isinstance(heading.get_html(), str)

        def test_return_value(self):
            """
            get_html returns an html 'li' element, and a nested anchor with expected properties
            """
            sample_heading_value = "This is a sample heading"
            heading = HeadingItem(sample_heading_value)

            assert (
                heading.get_html()
                == f"""
            <li>
                <a href='#{heading.id}'>{heading.value}</a>
            </li>
            """
            )
Enter fullscreen mode Exit fullscreen mode

And I got a successful CI run ✔️

SuccessfulCIonMaster

SuccessfulRun

Adding tests to other project

I also added some tests to one of my friend's project who is using Java instead of Python. I don't have a lot of experience in Java and always wanted to write a test in it. This seemed like a perfect oppurtunity.

This is her repository, feel free to check out.
https://github.com/WangGithub0/ConvertTxtToHtml

Setting up ⚙️

Unlike I expected, setting up the project with Junit proved to be really time-consuming for me.

One, I don't understand Java environments really well. All I have done so far is create some GUI applications using JavaFX. Wish I could share my code, but unfortunately, its part of my assignments and can't be open-sourced.
Second, the instructions I found in the Contributing docs were bare-minimum, and kinda hard to follow for a beginner. An experienced Java developer would get them really quickly, no doubt about that.

So I spent about 5 hours broken in sessions to figure out where I was going wrong and how to include required JAR files to my classpath.

Here's a glimpse of some chat with my best friend, the one and only...

GPTChatChat

And finally I was able to figure out how to compile and run test files.

After navigating into src folder,

cd src/
Enter fullscreen mode Exit fullscreen mode

and downloading the jars in libs folder

Jars

I ran the following command to compile the test file,

javac -cp ".;../libs/junit-4.10.jar;../libs/hamcrest-core-1.3.jar" .\application\ConvertTxtMdToHtml.java
Enter fullscreen mode Exit fullscreen mode

and to run the generated class file

java -cp ".;../libs/junit-4.10.jar;../libs/hamcrest-core-1.3.jar" org.junit.runner.JUnitCore junit.ConvertTxtMdToHtmlT
est
Enter fullscreen mode Exit fullscreen mode

And I got a successfull run.

Successful Run

Later, I also found out how to import the existing sources as an Intellij Idea project, and run the files using the GUI tools.

Adding Unit Tests

Now that I could run the existing tests, it was time to add mine. I noticed that there was a test for the -v option.

@Test
  public void testVersionOption() {
    final ByteArrayOutputStream outContent = new ByteArrayOutputStream();
    System.setOut(new PrintStream(outContent));

    ConvertTxtMdToHtml.main(new String[] {"-v"});

    assertEquals("Version command is triggered", "convertTxtToHtml version 0.1",
        outContent.toString().trim());
  }
Enter fullscreen mode Exit fullscreen mode

Taking inspiration from that, I decided to add tests for -h or --help option as that would be the easiest to start with.

Hence, I added a couple of tests - one for the short form -h and one for the longer version --help.

src/junit/ConvertTxtMdToHtmlTest.java

@Test
  public void testHelpOptionShort() {
    final ByteArrayOutputStream outContent = new ByteArrayOutputStream();
    final String expectedMessage = """
            Usage: convertTxtToHtml [options] <input>
            Options:
              --help, -h           Print this help message
              --version, -v        Print version information
              --output <dir>, -o   Specify the output directory (default: convertTxtToHtml)
              --lang, -l           Specify the language (default: en-CA)""";

    System.setOut(new PrintStream(outContent));

    ConvertTxtMdToHtml.main(new String[] {"-h"});

    assertEquals(expectedMessage,
            outContent.toString().trim().replaceAll("\r\n", "\n"));
  }
Enter fullscreen mode Exit fullscreen mode

src/junit/ConvertTxtMdToHtmlTest.java

@Test
  public void testHelpOptionVerbose() {
    final ByteArrayOutputStream outContent = new ByteArrayOutputStream();
    final String expectedMessage = """
            Usage: convertTxtToHtml [options] <input>
            Options:
              --help, -h           Print this help message
              --version, -v        Print version information
              --output <dir>, -o   Specify the output directory (default: convertTxtToHtml)
              --lang, -l           Specify the language (default: en-CA)""";

    System.setOut(new PrintStream(outContent));

    ConvertTxtMdToHtml.main(new String[] {"--help"});

    assertEquals(expectedMessage,
            outContent.toString().trim().replaceAll("\r\n", "\n"));
  }
Enter fullscreen mode Exit fullscreen mode

Let's try to run them now.

Tests Pass

And with that, I was able to add my very first unit tests in a Java project.

Here's the pull request!

Pull Request

And the successful CI workflow run.

Sucessful CI run

Conclusion 🎊

In this post, we discussed about what is Continuous Integration, why it is absolutely necessary in projects with multiple contributors to be successsful.
We also discussed about how to setup a CI workflow for a Python project, and in the end, I shared my experince about adding unit tests to another project that had recently been setup with a CI workflow (build and test).

Hope it was a fun read for you.
Any feedback is welcome in comments!

Attributions

Image by Freepik
Image by upklyak on Freepik

Top comments (2)

Collapse
 
michaeltharrington profile image
Michael Tharrington

Great post, Amnish! 🙌

Collapse
 
amnish04 profile image
Amnish Singh Arora

Thanks Michael!