Recently I noticed that Mathworks has released a set of GitHub actions for running short Matlab scripts and unit tests. Last year I considered adding CI support for one of my Matlab project but trying to work out how to setup the Matlab license proved too difficult for me at the time. Now with the new actions, I thought: Why not try setting up CI again?

This post describes the steps I took to learn the basics of GitHub Actions and add CI to one of my Matlab projects. I've never used GitHub Actions before, so what better time than during the Actions hackathon! The project I choose to add Actions to is a branch of the Optical Tweezers Toolbox.

Part 1: Adding the action to the project

This step was relatively simple: I stared by going to the Matlab tests example and copying the GitHub hosted tests example into a new action on my repository. In my development branch I created a file .github/workflows/runtests.yml:

name: Unit Tests

# Controls when the workflow will run
# Add [skip ci] to the commit to skip running this job
on:
  push:
    # My development branch was called draft162
    branches: [ draft162 ]

jobs:
  test:
    name: Run MATLAB Tests and Generate Artifacts
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v2
      - name: Set up MATLAB
        uses: matlab-actions/setup-matlab@v1

      - name: Run tests and generate artifacts
        uses: matlab-actions/run-tests@v1
        with:
          test-results-junit: test-results/results.xml
          code-coverage-cobertura: code-coverage/coverage.xml

          # I already had a folder called `tests` containing my
          # existing Matlab unit tests that I would run locally.
          # Each test file matches the format: test*.m
          select-by-folder: tests

As soon as I pushed the commit the action started running! No additional steps to setup Matlab! No additional third party tools required!

The Action seems to use a Sponsored Matlab license, although I have no idea what this means. The result was that I didn't need to point Matlab at my universities Matlab license, which made things much easier to setup/run.

After about 4 minutes my tests had run and I could view the logs to find out which tests passed and which tests failed. Overall the process was much easier than I expected. However, to view the logs, we need to be able to download the coverage report or display it after the Action runs; and browsing through the action logs isn't very convenient for finding out if the tests were successful. It would be nice if there was some graphical indication of the testing/coverage status (such as a badge or a success/failure count).

Part 2: Adding a badge

I'd seen testing badges displayed on other repositories that used GitHub Actions. After a quick google I found instructions for how to generate the appropriate URL. Now I can easily display the status of the action from within the project readme or elsewhere in the documentation.

Later I discovered that I could also have clicked the "Create Status Badge" button within the test results page to automatically format this URL:

Part 3: Saving coverage and test reports

The action included two directives to specify output files for the test-results and coverage-report

          test-results-junit: test-results/results.xml
          code-coverage-cobertura: code-coverage/coverage.xml

In order to view the contents of these files I needed to create an artifact by adding the upload-artifact action immediately following my testing action, for example:

 - name: Upload results artifact
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: test-restuls
          path: test-results/results.xml

In this action I needed to specify the if: always() line to force the action to always be run, even if some of the unit tests failed.

I also tried an alternative approach involving adding the continue-on-error: true statement to the testing action and adding an additional non-zero return status after all the other actions (see this SO for further details). This approach also worked, but it resulted in a slightly more verbose yaml file.

The result of adding the upload-artifact action was two archives that I could now download from the Action summary page:

I was then able to view these XML files in my favorite viewer.

Part 4: In browser display of testing reports

Downloading the XML files is not the most convenient workflow. It would be useful to have at least a summary in the browser with the current number of passing/failing test. Taking a quick look at the GitHub marketplace, I found a action that displays exactly this (and more)

- name: Publish Test Results
  uses: EnricoMi/publish-unit-test-result-action@v2
  if: always()
  with:
    files: |
      test-results/**/*.xml
      test-results/**/*.trx
      test-results/**/*.json

- name: Publish Test Results
  uses: EnricoMi/publish-unit-test-result-action/macos@v2
  if: always()
  with:
    files: |
      test-results/**/*.xml
      test-results/**/*.trx
      test-results/**/*.json

- name: Publish Test Results
  uses: EnricoMi/publish-unit-test-result-action/windows@v2
  if: always()
  with
    files: |

I added this action to my yaml file, just as I did for the upload action previously:

- name: Publish Unit Test Results
        uses: EnricoMi/publish-unit-test-result-action/composite@v1
        if: always()
        with:
          files: test-results/results.xml
          report_individual_runs: "true"

Running this action adds an additional tab to the action summary which has annotations for each failed test.

The report_individual_runs option seems to be needed to get GitHub to display an annotation for each test (as opposed to just one annotation for the whole file).

Summary

So far I've been able to setup an action to test most of the code, there are still many things that could be improved including:

• Work out how to display coverage reports. I tried using Code Coverage Summary but couldn't work out why it was not running.
• Look into speeding up CI time by running tests only for affected directory
• CI for graphical user interfaces (not yet supported by Activity)

My Workflow

Actions used:

• EnricoMi/publish-unit-test-result-action/composite@v1
• matlab-actions/run-tests@v1
• actions/upload-artifact@v2

Submission Category:

Wacky Wildcards or Maintainer Must-Haves, not sure which fit is better.

Yaml File or Link to Code

Full yaml file (and a couple of extras):

# Action to runs tests on repository

name: Unit Tests

# Controls when the workflow will run
# Add [skip ci] to the commit to skip running
on:
  push:
    branches: [ draft162 ]

jobs:
  test:
    name: Run MATLAB Tests and Generate Artifacts
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v2
      - name: Set up MATLAB
        uses: matlab-actions/setup-matlab@v1

      - name: Run tests and generate artifacts
        uses: matlab-actions/run-tests@v1
        with:
          test-results-junit: test-results/results.xml
          code-coverage-cobertura: code-coverage/coverage.xml

          # Exclude GUI, not yet supported by test runner
          select-by-folder: >
              tests/beam; tests/bsc; tests/drag; tests/dynamics;
              tests/examples; tests/particle; tests/shape; tests/tmatrix;
              tests/tools; tests/utils
          # Folders we should run coverage reports on
          source-folder: +ott; examples

      - name: Publish Unit Test Results
        uses: EnricoMi/publish-unit-test-result-action/composite@v1
        if: always()
        with:
          files: test-results/results.xml
          report_individual_runs: "true"

      # Not sure why, but this doesn't seem to work :(
      - name: Code Coverage Summary Report
        uses: irongut/CodeCoverageSummary@v1.2.0
        if: always()
        with:
          filename: code-coverage/coverage.xml
          format: markdown
          badge: true

      - name: Upload results artifact
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: test-restuls
          path: test-results/results.xml

      - name: Upload coverage artifact
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: code-coverage
          path: code-coverage/coverage.xml

Additional Resources / Info

This page describes the Actions added to a branch of the Optical Tweezers Toolbox, a toolbox for modelling particles trapped in optical fields. The Actions should hopefully be merged into the master branch sometime later this year.