DEV Community

Vincent A. Cicirello
Vincent A. Cicirello

Posted on

Bonus Tip: How to Use GitHub Actions to Test a GitHub Action Whose Output Must be Visually Inspected

Last week, I posted a tutorial on How to Test a GitHub Action with GitHub Actions. The audience for that post, as well as this one, are maintainers of GitHub Actions. In that post, I explained my automated approach to testing GitHub Actions, including unit testing and integration testing, all within GitHub Actions itself. I assumed in that post that the Action you are testing is implemented in Python, although most of the post is applicable more generally, or can easily be adapted to other languages.

This week's post is a bonus tip related to testing a GitHub Action within GitHub Actions. In last week's post, I assumed that it was possible to fully automate the validation of the results of the Action we are testing. In particular, I had a step that used Python's unittest module to run a set of tests that verified correctness of any files produced or modified by the action. But what if the action produces something that cannot be verified automatically? For example, what if it produces an image or something else that must be visually inspected for correctness?

Table of Contents: The rest of this post is organized as follows:

Workflow Artifacts to the Rescue

I'm assuming that you have unit tests and already have of a workflow that runs your unit tests, such as something like my example from last week. Now let's consider a case where my previous approach to integration testing isn't feasible, such as if the output of the action itself is an image or something else that must be visually inspected. We can replace the integration testing steps of last week's workflow with the following step (or add this as an additional step if you can partially automate validating your integration test results). In this example step, I'm assuming your action produces an svg file that you want to inspect, but this can work with anything.

    - name: Upload files produced by action as a workflow artifact for inspection if necessary
      uses: actions/upload-artifact@v3
      with:
        name: name-for-the-artifact
        path: path/to/file/filename.svg
Enter fullscreen mode Exit fullscreen mode

Or perhaps your action produces an entire directory of files, then you can use something like:

    - name: Upload files produced by action as a workflow artifact for inspection if necessary
      uses: actions/upload-artifact@v3
      with:
        name: name-for-the-artifact
        path: path/to/directory
Enter fullscreen mode Exit fullscreen mode

In both of the above cases, I've used the actions/upload-artifact action to upload either a file or an entire directory of files as a workflow artifact. In both cases, that action will zip up what the path input points to into a zip file with name based on the name input, such as name-for-the-artifact.zip in this example. It is then attached to the run of the workflow that produced it.

In this way, you can download from the Actions tab of your repository, and inspect the results. Just go to the Actions tab, and find the relevant workflow run, such as from a PR or push.

Real Example

I use this approach in the user-statistician, which generates an SVG with a detailed summary of your GitHub activity, such as contributions, language statistics, etc. I have unit tests implemented in Python with the unittest module. But, the SVG as a whole can only really be validated visually, such as to inspect that content fits appropriately, image dimensions are correct based on content, etc.

The last step of my workflow that runs the tests for that project is the following:

    - name: Upload generated SVG as a workflow artifact for inspection if necessary
      uses: actions/upload-artifact@v3
      with:
        name: generated-image
        path: images/userstats.svg
Enter fullscreen mode Exit fullscreen mode

After the workflow runs, such as while reviewing a PR, I can navigate to the Actions tab of the repository, find the workflow run, and I'll see the following (screenshot of an actual run).

Example Workflow Artifacts

Be aware that if you are using this approach in a private repository that workflow artifacts count against your storage limit. In public repositories, they do not count against your quota, but they are removed after some time period. The default is 90 days.

The complete workflow for this project is found at: build.yml. The repository itself is:

GitHub logo cicirello / user-statistician

Generate a GitHub stats SVG for your GitHub Profile README in GitHub Actions

user-statistician

user-statistician

Check out all of our GitHub Actions: https://actions.cicirello.org/

About user-statistician Mentioned in Awesome README

GitHub Actions GitHub release (latest by date) Count of Action Users
Build Status build samples CodeQL
Source Info License GitHub top language
Contributors GitHub contributors good first issue
Support GitHub Sponsors Liberapay Ko-Fi

The cicirello/user-statistician GitHub Action generates a detailed visual summary of your activity on GitHub in the form of an SVG suitable to display on your GitHub Profile README Although the intended use-case is to generate an SVG image for your GitHub Profile README you can also potentially link to the image from a personal website, or from anywhere else where you'd like to share a summary of your activity on GitHub. The SVG that the action generates includes statistics for the repositories that you own, your contribution statistics (e.g., commits, issues, PRs, etc), as well as the distribution of languages within public repositories that you own The user stats image can be customized, including the colors such as with one of the built-in themes or your own set of custom…

Complete Example Workflow

Here's last week's full example workflow for automated testing, but with the example artifact upload step added at the end.

name: build

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:

  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.10'

    - name: Run Python unit tests
      run: python3 -u -m unittest tests/tests.py

    - name: Verify that the Docker image for the action builds
      run: docker build . --file Dockerfile

    - name: Integration test 1
      uses: ./
      with:
        input-one: something
        input-two: true

    - name: Integration test 2
      uses: ./
      with:
        input-one: something else
        input-two: false

    - name: Verify integration test results
      run: python3 -u -m unittest tests/integration.py

    - name: Upload files produced by action as a workflow artifact for inspection if necessary
      uses: actions/upload-artifact@v3
      with:
        name: name-for-the-artifact
        path: path/to/file/filename.svg
Enter fullscreen mode Exit fullscreen mode

Where You Can Find Me

Follow me here on DEV:

Follow me on GitHub:

GitHub logo cicirello / cicirello

My GitHub Profile

Vincent A Cicirello

Vincent A. Cicirello

Sites where you can find me or my work
Web and social media Personal Website LinkedIn DEV Profile
Software development Github Maven Central PyPI Docker Hub
Publications Google Scholar ORCID DBLP ACM Digital Library IEEE Xplore ResearchGate arXiv

My bibliometrics

My GitHub Activity

If you want to generate the equivalent to the above for your own GitHub profile, check out the cicirello/user-statistician GitHub Action.




Or visit my website:

Vincent A. Cicirello - Professor of Computer Science

Vincent A. Cicirello - Professor of Computer Science at Stockton University - is a researcher in artificial intelligence, evolutionary computation, swarm intelligence, and computational intelligence, with a Ph.D. in Robotics from Carnegie Mellon University. He is an ACM Senior Member, IEEE Senior Member, AAAI Life Member, EAI Distinguished Member, and SIAM Member.

favicon cicirello.org

Top comments (0)