DEV Community


Posted on • Updated on

Integrate tests with Buildkite Test Analytics

Buildkite introduced a new feature called Test Analytics ๐Ÿ“ˆ this month and in my attempt to rank high on SEO on this subject, you're in the right article on how I did it (considering you already have access to Buildkite).

Just a bit of context (which no one cares), I just joined a scale-up where we started tracking stuff and understandably, it includes the quality of our delivery lifecycle (or else, what's the point of my existence ๐Ÿ˜‚). One of the metrics I personally wanted to check is how predictable and trust-worthy our tests are. The tendency is to ignore tests (especially on the Slack alerts where it becomes normal) when there is pressure to deliver and the amount of failing tests exceed the succeeding ones. That is just unacceptable for me as a Quality engineer and something that should be closely monitored.

"You can't improve what you don't measure." โ€” Peter Drucker

For our jest frontned unit tests, there is a readily-available integrator (which makes life so easy!) where you just simply need to install (and obviously, include in your source code repo) the npm package called buildkite-test-collector and include the reporter in your config file (e.g. jest.config.js):

reporters: [
testLocationInResults: true
Enter fullscreen mode Exit fullscreen mode

jest setup page

For our cypress end-to-end tests, it is a bit trickier as it involves a couple more steps. You need to use the default junit reporter and include the reporter in your config file (e.g. cypress.json or cypress.config.ts for Cypress 10 and above):

  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "results/junit-[hash].xml",
    "toConsole": true
Enter fullscreen mode Exit fullscreen mode

Once done, you need to go to your test or build pipeline (or wherever you run your tests) > Build steps (or pipeline.yml), and import the JUnit XML file. As lazy as I am (some call it smart ๐Ÿ˜Ž), I used the Test collector plugin in this instance.

  - label: '๐Ÿงช Run cypress tests'
    commands: 'npx cypress run --record --parallel --ci-build-id $BUILDKITE_BUILD_NUMBER'
    plugins: # pass details to Test Analytics
      - test-collector#v1.0.0:
          files: 'results/junit-*.xml'
          format: 'junit'
      - 'results/**/*'
Enter fullscreen mode Exit fullscreen mode

Once the configuration is done, include the BUILDKITE_ANALYTICS_TOKEN value on your pipeline's Build Steps or pipeline.yml
Build step page

Once the tokens are set, run your pipelines and then the magic happens! ๐Ÿช„
Test analytics page

Note: Unsure why the test summary is not included in the main test suite page for the unit tests (maybe the Buildkite Gods can help us and I will update this write-up).

Click a specific test suite and the "devil (if heaps of failing tests) is in the details" ๐Ÿ‘ฟ
Test suite details page

This is good alternative to Cypress dashboards too but free of charge if you're already subscribed to Buildkite.

Top comments (0)