Months ago, a collaborator asked for a way to keep track of the docstring coverage of our Python package. We wanted to measure the current coverage of the docstrings in our code and prevent pull requests from decreasing that value.
To solve this problem, a workflow needed to fulfill the following requirements:
- The workflow should fail if the coverage is lower than the score of the base branch (on
pull requests) or the previous commit (onpush), and files that made the job fail should be blamed. - Pushing to main branch should update the current coverage of the project and display the result on a nice badge.
Then, I found the excellent docstr-coverage package that suited most of our needs, and in combination with GitHub Actions, shields.io and jsonbin.org I came up with a solution.
jsonbin
jsonbin.org is a personal key/value JSON store as a service.
In this case, we use this service to create dynamic badges with the shields.io endpoint. A small JSON dictionary is passed to the endpoint and the badge is created on-the-go. Thus, we avoid to push a new badge to the repository each time the coverage changes.
{
"schemaVersion": 1,
"label": "docstr-cov",
"message": "75%",
"color": "green"
}
You will need to sign in to jsonbin with your GitHub account and store the token as a repository secret named JSONBIN_APIKEY.
Configuration file
Place a file named .docstr.yml at the root of your repository.
paths:
- your_package
verbose: 2 # int (0-3)
skip_magic: True
skip_file_doc: True
skip_init: True
skip_class_def: False
skip_private: True
follow_links: True
percentage_only: False
For more information, see the package documentation.
The workflow
This workflow should work out the box. It's not necessary to tweak any environment variable, but you can choose different values for RANGE.
For example, a RANGE of 50..75 means that coverage below 50 will display a red badge, and above 75 a green one. It works exactly like Codecov!
name: docstr-cov
on:
push:
branches:
- main
pull_request:
branches:
- main
env:
RANGE: 50..75
ENDPOINT: https://jsonbin.org/${{ github.repository_owner }}/${{ github.event.repository.name }}
TOKEN: ${{ secrets.JSONBIN_APIKEY }}
At the job level, the steps are easy to follow:
- Commits to compare (
HEADandBASE) are chosen depending on the event trigger:HEADof the branch vs. previous commit onpushevents, orHEADof the PR vs. base of the branch onpull_request. - Get coverage score on
BASEandHEAD. Fails if HEAD score is lower. - Blame files (if failed).
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: 3.x
- name: Install docstr-coverage
run: pip install docstr-coverage
- name: Get SHAs
run: |
if [[ ${{ github.event_name }} == 'push' ]]; then
echo "BASE=$(git rev-parse HEAD^)" >> $GITHUB_ENV
echo "HEAD=$(git rev-parse HEAD)" >> $GITHUB_ENV
elif [[ ${{ github.event_name }} == 'pull_request' ]]; then
echo "BASE=${{ github.event.pull_request.base.sha }}" >> $GITHUB_ENV
echo "HEAD=${{ github.event.pull_request.head.sha }}" >> $GITHUB_ENV
else
echo "Unexpected event trigger"
exit 1
fi
- name: Get base coverage
run: |
git checkout $BASE
echo "BASE_COV=$(docstr-coverage -p)" >> $GITHUB_ENV
- name: Test head coverage
run: |
git checkout $HEAD
docstr-coverage --fail-under=$BASE_COV
- name: Blame
run: |
git diff --name-only $(git merge-base $BASE $HEAD) | \
xargs docstr-coverage --accept-empty
if: failure()
Only on push events:
- Get coverage (rounded)
- Set the color label according to
RANGE. - Post results to jsonbin.
- Make the endpoint public.
- Print the badge URL.
- name: Get new coverage
run: echo "NEW_COV=$(printf "%.f" $(docstr-coverage -p))" >> $GITHUB_ENV
if: always() && github.event_name == 'push'
- name: Set label color
run: |
if [[ $NEW_COV -ge $(echo {${{ env.RANGE }}} | awk '{print $NF;}') ]]; then
echo "COLOR=green" >> $GITHUB_ENV
elif [[ $NEW_COV -lt $(echo {${{ env.RANGE }}} | awk '{print $1;}') ]]; then
echo "COLOR=red" >> $GITHUB_ENV
else
echo "COLOR=orange" >> $GITHUB_ENV
fi
if: always() && github.event_name == 'push'
- name: Post results
run: |
curl -X POST $ENDPOINT/badges/docstr-cov \
-H "authorization: token $TOKEN" \
-d "{ \"schemaVersion\": 1, \"label\": \"docstr-cov\", \
\"message\": \"$NEW_COV%\", \"color\": \"$COLOR\" }"
if: always() && github.event_name == 'push'
- name: Set public endpoint
run: |
curl -X PUT $ENDPOINT/_perms -H "authorization: token $TOKEN"
if: always() && github.event_name == 'push'
- name: Show badge URL
run: echo "https://img.shields.io/endpoint?url=$ENDPOINT/badges/docstr-cov"
if: always() && github.event_name == 'push'
The badge should be updated dynamically on every push to main. Remember to add it to your README.md.
Get the code
epassaro
/
docstr-cov-workflow
Measure docstring coverage of Python packages with GitHub Actions
docstr-cov-workflow
Measure docstring coverage of Python packages with GitHub Actions
Usage
- Copy
.github/workflows/docstr-cov.ymland.docstr.yamlto your repository. - Tweak the configuration file following the package documentation.
- Login to https://jsonbin.org and store the API key as a repository secret named
JSONBIN_APIKEY. - The workflow will fail if the coverage is lower than the score of the base branch (on
pull_request) or the previous commit (onpush), and files that made the job fail will be blamed. - Pushing to
mainbranch updates the current coverage of the project by updating a nice badge. - You can change the color range of the badge by tweaking the
RANGEvariable at the top of the workflow. For example,50..75means that coverage below50will display a red badge, and above75a green one. - Remember to add the badge to your README.md
Example
Make changes to example/base.py and see the workflow…
Top comments (0)