Here is the overview. The point is separating jobs to run all test and run failed test. That allows you to rerun the failed test only. It could save the time because You don't need to rerun whole test suite, You can rerun just failed ones.
# spec/spec_helper.rb
RSpec.configure do |config|
config.example_status_persistence_file_path = "spec/examples.txt"
end
# .circleci/config.yml
version: 2.1
executors:
ruby:
docker:
- image: circleci/ruby
jobs:
test:
executor: ruby
steps:
- checkout
- run: gem install rspec
- run: rspec --failure-exit-code=0
- persist_to_workspace:
root: .
paths:
- '*'
test_again:
executor: ruby
steps:
- attach_workspace:
at: ~/project
- run: gem install rspec
- run: rspec --only-failures
workflows:
test:
jobs:
- test
- test_again:
requires: [test]
The test
job run whole test suites, that fail sometimes because of flaky test. But it's ok. I'll rerun the failed tests in the test_again
job.
RSpec has example_status_persistence_file_path
configuration to store the test result.
In this .circleci/config.yml
, I use persist_to_workspace
to save the test result of the test
job and pass them to the test_again
job.
test_again
job requires
the test
job, which means rspec
exit successfully whether test passed or not to run the test_again
job.
A list of jobs that must succeed for the job to start
https://circleci.com/docs/2.0/configuration-reference/#requires
To do this, pass --failure-exit-code=0
option to rspec
command.
c.f. \-\-failure\-exit\-code\
option (exit status) - Command line - RSpec Core - RSpec - Relish
Then the test_again
job receives the test result of the test
job by attach_workspace
and run rspec
with --only-failures
option to run failed test only..
c.f. Only Failures - Command line - RSpec Core - RSpec - Relish
Sample project
I create an example repository
https://github.com/hanachin/rspec-soumen
It has randomly failing test.
using Module.new {
refine(Integer) do
def 冷やしそうめんがうまい?
self >= 25
end
end
}
RSpec.describe '冷やしそうめん' do
it { is_expected.to be }
context '気候変動' do
let(:temperature) { rand(10..40) }
subject { temperature.冷やしそうめんがうまい? }
it { is_expected.to be }
end
end
Then the test failed in test
job and test_again
job.
https://circleci.com/workflow-run/2cced3b3-59c4-4e3e-9f53-33172388b74b
You can see the test
job runs whole test suite, and it failed but marked as success.
https://circleci.com/gh/hanachin/rspec-soumen/6
And the test_again
job runs failed test only.
https://circleci.com/gh/hanachin/rspec-soumen/7
You can rerun workflows from failed job.
That time the test_again
job passed.
https://circleci.com/workflow-run/09fb370d-bbdd-4648-acba-0bebe5a7bddf
Conclusion
I separate the test jobs to run all test and run failed test.
Result:
If tests failed again in
test_again
, I don't need to rerun whole test suite, I can rerun just failed ones.I can rerun failed tests without any changes to the test code like
rspec-retry
or extendingCapybara.default_wait_time
or something like that.I don't need to investigate flaky test, just rerun the failed tests.
I know fixing the flaky test is good but it may take tons of times,
I avoid it.My failed tests rerun automatically in the
test_again
job once.
I can chain thetest_again_again
,test_again_again_again
jobs as well 😘I can take statistics of flaky tests in
test_again
job, maybe.
Cheers🍻
Top comments (2)
It's helpful, many thanks!
Great stuff! I just wish I can do this using Python. So I can rerun the specific scenarios that fails. I will appreciate any pointers please