Flaky Tests meaning:
A flaky test is an unstable test which sometimes passes and sometimes fails without a touchable reason. Flaky tests cause time-wasting and delay till discovering the real reason for the failure. So, it slows down team productivity.
There are common reasons for flaky tests:
You should have a solid structure for your test framework. Using a repeated code can cause flaky tests. So, try to use a clean code and DRY principles for example page object design pattern or factory design pattern may eliminate flakiness.
Using hardcoded data may cause, as if you want to run your tests on different environments, you will have to change the hardcoded data like file paths or IP addresses.
Caching a previous test state and starting a new test with cached data may cause flakiness, so it is preferred to clean the cache for every test run. Setup the required data before each test and clean up the environment state after each test.
The flakiness may be occurred by an external reason which is not related to the test itself. Network connection, database connection, browser version and memory leaks are candidate reasons for failure.
Losing a response from a 3rd party may lead to failure so that stubbing the 3rd party response can prevent these types of failures.
Using ineffective element selectors like XPATH or element coordinates can lead to test failure. Those selectors can be easily changed while updating the page design. It is recommended to use more solid selectors like id or name.
Using many of sleep command stops the test till this period ends. Replacing the sleep command by an implicit wait till the target element appears saves the test time and reduces flakiness.
Time may be a cause. Is your test failing only during runs around midnight, at the beginning of the year, at the beginning of the month, or during the full moon? If you find a pattern of failures like this, it might be due to time or date issues.
Environment may be a cause so, you should also try to understand if it happens in a particular environment, i.e. does it only fail on staging, on a specific OS, etc... anything you can find in common about the failures may be useful in understanding if this is specific to where you run your tests.
The test that dependents on an AJAX request the page triggers, and the test runs before the request is loaded. This cause is very common and may be very difficult to identify. However, because of its frequency, testing tools generally have ways to deal with it.
Dealing with flaky tests:
Document the flaky tests and Set flaky tests in a quarantined area. After running tests, check the recorded data. Check logs, memory dumps, the system's current state, or even screenshots in UI tests to investigate what caused the failure.
Create tickets to tasks for these tests and pick them up one by one from a quarantined area. And start investigating into the test by checking the previous reasons for failure. Exclude the quarantined area tests from a deployment pipeline as we can not depend on them till the investigation is done and tests become stable.
When Google faced flaky tests, they analyzed the data from flaky test executions,seeing promising correlations with features that should enable identifying a flaky result accurately without re-running the test.
Conclusion:
In this article, we discussed flaky tests and how to mitigate flaky tests. Eliminating flakiness from your tests is not impossible, but it needs effort and time to investigate and mitigate it.
Top comments (0)