CircleCI Workflows to rerun flaky RSpec examples

Here is the overview. The point is separating jobs to run all test and run failed test. That allows you to rerun the failed test only. It could save the time because You don't need to rerun whole test suite, You can rerun just failed ones.

# spec/spec_helper.rb
RSpec.configure do |config|
  config.example_status_persistence_file_path = "spec/examples.txt"
end

# .circleci/config.yml
version: 2.1

executors:
  ruby:
    docker:
      - image: circleci/ruby

jobs:
  test:
    executor: ruby
    steps:
      - checkout
      - run: gem install rspec
      - run: rspec --failure-exit-code=0
      - persist_to_workspace:
          root: .
          paths:
            - '*'

  test_again:
    executor: ruby
    steps:
      - attach_workspace:
          at: ~/project
      - run: gem install rspec
      - run: rspec --only-failures

workflows:
  test:
    jobs:
      - test
      - test_again:
          requires: [test]

The test job run whole test suites, that fail sometimes because of flaky test. But it's ok. I'll rerun the failed tests in the test_again job.

RSpec has example_status_persistence_file_path configuration to store the test result.
In this .circleci/config.yml, I use persist_to_workspace to save the test result of the test job and pass them to the test_again job.

test_again job requires the test job, which means rspec exit successfully whether test passed or not to run the test_again job.

A list of jobs that must succeed for the job to start
https://circleci.com/docs/2.0/configuration-reference/#requires

To do this, pass --failure-exit-code=0 option to rspec command.

c.f. \-\-failure\-exit\-code\ option (exit status) - Command line - RSpec Core - RSpec - Relish

Then the test_again job receives the test result of the test job by attach_workspace and run rspec with --only-failures option to run failed test only..

c.f. Only Failures - Command line - RSpec Core - RSpec - Relish

Sample project

I create an example repository

https://github.com/hanachin/rspec-soumen

It has randomly failing test.

using Module.new {
  refine(Integer) do
    def 冷やしそうめんがうまい?
      self >= 25
    end
  end
}

RSpec.describe '冷やしそうめん' do
  it { is_expected.to be }

  context '気候変動' do
    let(:temperature) { rand(10..40) }

    subject { temperature.冷やしそうめんがうまい? }

    it { is_expected.to be }
  end
end

Then the test failed in test job and test_again job.

https://circleci.com/workflow-run/2cced3b3-59c4-4e3e-9f53-33172388b74b

You can see the test job runs whole test suite, and it failed but marked as success.

https://circleci.com/gh/hanachin/rspec-soumen/6

And the test_again job runs failed test only.

https://circleci.com/gh/hanachin/rspec-soumen/7

You can rerun workflows from failed job.
That time the test_again job passed.

https://circleci.com/workflow-run/09fb370d-bbdd-4648-acba-0bebe5a7bddf

Conclusion

I separate the test jobs to run all test and run failed test.

Result:

• If tests failed again in test_again, I don't need to rerun whole test suite, I can rerun just failed ones.

• I can rerun failed tests without any changes to the test code like rspec-retry or extending Capybara.default_wait_time or something like that.

• I don't need to investigate flaky test, just rerun the failed tests.
  I know fixing the flaky test is good but it may take tons of times,
  I avoid it.

• My failed tests rerun automatically in the test_again job once.
  I can chain the test_again_again, test_again_again_again jobs as well 😘

• I can take statistics of flaky tests in test_again job, maybe.

Cheers🍻