In the previous article we started working with the CircleCI platform, by setting up testing infrastructure for our code.
In this article, we are going to set up code coverage for the project.
The Plan
- Instruct test runner (
jest
) to generate and save code coverage files for our tests. - Upload those reports to external service for aggregation and display. For that, we are going to use codecov.io for code coverage.
First, you will need to go to codecov.io to create an account and chose the repository you want to show code coverage for. Then you need to install codecov.io application from the github marketplace.
As a reminder, this is the CircleCI configuration file from the previous article.
version: 2.1
jobs:
node-v10:
docker:
- image: circleci/node:10
steps:
- test
node-v12:
docker:
- image: circleci/node:12
steps:
- test
node-v13:
docker:
- image: circleci/node:13
steps:
- test
commands:
test:
steps:
- checkout
- restore_cache:
key: dependency-cache-{{ checksum "package-lock.json" }}
- run:
name: install-dependancies
command: npm ci
- save_cache:
key: dependency-cache-{{ checksum "package-lock.json" }}
paths:
- $HOME/.npm
- run:
name: unit test
command: |
npm run ci:test
workflows:
version: 2
build_and_test:
jobs:
- node-v10
- node-v12
- node-v13
We are going to continue to build upon this file by adding code coverage functionality.
How is Code Coverage Generated
Since we are using Jest for testing the code, there is not much that we need to do to set up code coverage, because jest comes with code coverage out of the box.
Next, we need to modify ci:test
npm script inside the package.json
Our previous npm
script for running jest
tests was:
"ci:test": "jest --runInBand --ci"
We are going to change the script slightly so we can instruct jest to also generate code coverage files and save them in the coverage
directory in the root of the project.
"ci:test": "jest --runInBand --ci --reporters=default --coverage --coverageDirectory=coverage"
Setting up Code Coverage on the CircleCI platform
Setting up code coverage is fairly easy. After jest
tests the code and generates code coverage files we need to save them somewhere and then upload them to the codecov.io
In the previous article we talked about how code is running inside Linux containers and how they are ephemeral and are discarded as soon as the jobs
that are running inside them are done, so we are going to need to take the files that are generated by jest and save them somewhere outside of the containers before the containers are destroyed.
Then, after we save the files somewhere outside the containers, the only step left is to upload those files to the codecov.io service.
To use the codecov.io service via the circleCI platform, we are going to use the official codecov.io "orb".
CircleCI Orbs Concept Explained
CircleCI platform has a concept of orbs
,
from the docs:
Orbs are reusable packages of CircleCI configuration that you may share across projects, enabling you to create encapsulated, parameterized commands, jobs, and executors that can be used across multiple projects.
You can consider them as classes in object oriented programming. A blueprint for doing certain tasks. Orbs simplify tasks by hiding all necessary steps that need to be done and only exposing a few methods to get the tasks going.
You can take a look at the actual code for the codecov orb
to see how much typing using the orbs
saves us. Also when using officially supported orbs you can be sure that your CircleCI jobs will work as expected when integrating with the external services. Many online services that integrate with CircleCI have their own orbs
that make it easy to integrate with the continuous integration process.
To use any orb we need to declare it under the orbs
key, so let's add the orbs
key to the config.yml
file.
version: 2.1
orbs:
codecov: codecov/codecov@1.0.5
jobs:
Next, we are going to add additional steps to be executed after the code is successfully tested.
jobs:
# ... node-v10 job
node-v12:
docker:
- image: circleci/node:12
steps:
- test # test the code
- store-coverage-data # store coverage files outside of the container command
- upload-coverage # upload files to the codecov.io command
# ... node-v13 job
You might notice that in the previous example I have omitted the code for the node-v10
and node-v13
jobs. The reason for this is that I have decided to only use code coverage from one job/container (node-v12
). Coverage data from the node-v10
and node-v13
jobs will be ignored, Jest will create code coverage but it will not be used by the codecov.io orb (we could further optimize those jobs to not generate code coverage at all but that is beyond the scope of this article).
Code Coverage Commands
Now let's take a look at the commands that are used in the steps above.
commands:
store-coverage-data:
steps:
- store_artifacts:
path: coverage
upload-coverage:
steps:
- codecov/upload:
file: coverage/coverage-final.json
-
step:
store-coverage-data
-
store_artifacts
- This is a special built-in command in the CircleCI platform that can save anything inside the container to the CircleCI permanent storage (move it outside of the container). We are using it to save our coverage files. From the docs:
Artifacts persist data after a job is completed and may be used for longer-term storage of the outputs of your build process. For example, when a Java build/test process finishes, the output of the process is saved as a .jar file. CircleCI can store this file as an artifact, keeping it available long after the process has finished.
-
path
This is the path inside the container that we want to be saved externally. Please note that we are using a relative path to thecoverage
directory inside our working directory (root directory of our checked out code)
-
-
step:
upload-coverage
-
codecov/upload
: This is a custom command (upload
) defined viacodecov
orb. As you can see you can treat it as a class instance methodcodecov->upload
This command uploads required files for the code coverage service. -
file
path to the file we want to upload.
-
Using Jest-Junit for Test Results
There is one additional step that we can add when setting up code coverage. We can store test performance results for later review.
Inside the circleCI UI, there can be an additional tab that can show you which tests are failing and which tests took a long time to run.
To enable this functionality, we need to install additional node module jest-junit and configure it via package.json
"jest-junit": {
"outputDirectory": "./reports/junit",
"outputName": "test-results.xml"
}
once again, we need to modify ci:test
npm script inside the package.json
"ci:test": "jest --runInBand --ci --reporters=default --reporters=jest-junit --coverage --coverageDirectory=coverage"
Next we need to modify circleCI configuration file and add another step
to the test
command: store_test_results
From the docs
store_test_results
- Special step used to upload and store test results for a build. Test results are visible on the CircleCI web application, under each build’s “Test Summary” section. Storing test results is useful for timing analysis of your test suites.
modified configuration file:
commands:
test:
steps:
- checkout
- restore_cache:
key: dependency-cache-{{ checksum "package-lock.json" }}
- run:
name: install-dependancies
command: npm ci
- save_cache:
key: dependency-cache-{{ checksum "package-lock.json" }}
paths:
- $HOME/.npm
- run:
name: unit test
command: |
mkdir -p ./reports/junit
npm run ci:test
- store_test_results: #<--- store test results for CircleCI
path: ./reports/junit/
And that's it, that is all it takes to set up code coverage via the CircleCI platform.
Complete config file:
version: 2.1
orbs:
codecov: codecov/codecov@1.0.5
jobs:
node-v10:
docker:
- image: circleci/node:10
steps:
- test
node-v12:
docker:
- image: circleci/node:12
steps:
- test
- store-coverage-data
- upload-coverage
node-v13:
docker:
- image: circleci/node:13
steps:
- test
commands:
store-coverage-data:
steps:
- store_artifacts:
path: coverage
test:
steps:
- checkout
- restore_cache:
key: dependency-cache-{{ checksum "package-lock.json" }}
- run:
name: install-dependancies
command: npm ci
- save_cache:
key: dependency-cache-{{ checksum "package-lock.json" }}
paths:
- $HOME/.npm
- run:
name: unit test
command: |
npm run ci:test
- store_test_results:
path: ./reports/junit/
upload-coverage:
steps:
- codecov/upload:
file: coverage/coverage-final.json
workflows:
version: 2
build_and_test:
jobs:
- node-v10
- node-v12
- node-v13
Bonus: Configuring Jest to Fail
You can also set up Jest to fail the test if code coverage stats are below a certain threshold. You can do it on a global level or per directory/file.
This goes into your package.json
file.
"jest":{
"coverageThreshold": {
"global": {
"branches": 50,
"functions": 50,
"lines": 50,
"statements": 50
},
"./src/components/": {
"branches": 40,
"statements": 40
},
}
}
You can read more about jest.io coverage threshold here
To use code coverage with private repositories, you will need to set up a $CODECOV_TOKEN
environment token via CircleCI dashboard
You can obtain the token when you connect your repository with codecov.io via their website.
Top comments (1)
Please take a look at LCOV.SH full BASH implementation of coverage… no need for additional interpreters like RUBY or BINARY executable in your machine. Check coverage of BASH just with BASH github.com/javanile/lcov.sh