I have a major pet peeve with any test strategy that includes something called “regression tests”.
I recently saw someone ask, “Do we need to run regression tests on a project where the CI/CD pipeline runs all tests on any git commit?”
On a different project, someone asked, “Now that we run this new test suite on every release candidate, can we stop running the regression tests?”
It made me wonder: What do these people think “regression tests” are, if they’re not the tests that run on every commit or every release?
If I can get pedantic for a moment, it’s worth looking at what “regression” actually means and why we have a test for it. Put simply, a regression is a return to some previous or lesser state. In the context of testing, this usually means a test to ensure that a bug doesn’t recur or a previously-working feature doesn’t break. Technically, you can think of any test run after it passed once as a regression test. It worked once, and by running the tests again we’re testing whether it has regressed.
If a CI pipeline is running tests on every commit, presumably you’re doing that to ensure the code remains in a working state, so those are regression tests by definition. New tests added in one commit become regression tests as soon as you run them on the 2nd commit. If you run a particular suite of tests on every release candidate, those too are regression tests. In both cases, though, the phrase “regression tests” was actually being used as shorthand for a particular subset of tests that was different from the tests in two use cases. As for which subset, we have no idea. At least not without a lot more contextual knowledge about how the project is structured.
I have the same problem with “functional tests”. The term is traditionally used to differentiate tests from so-called “non-functional” tests. (I.e., the tests focused around performance, security, and other “-ilities” that aren’t strictly describing the business logic or individual user interactions.) By that measure, though, unit tests are also “functional” — especially if they are testing a unit of business logic — even though that’s not what most people mean when they say it. I expect the only reason unit tests aren’t typically included in the “functional test” umbrella is because of the historic division between testers and developers. When someone does use the term, I typically guess that they’re referring to front-end or user-facing functionality, but it’s really anybody’s guess without more information.
Two things brought this to a head for me recently. First, when working on a test strategy template that used three categories for tests: unit, functional, and regression. None of those are mutually exclusive with each other! Second, one team I work with had something they kept calling “the regression suite” even though they didn’t use it to test their releases for regressions any more. “Regression” and “functional” are both so nebulously defined and inconsistently applied that they have become meaningless across the industry.
So what should we do instead? If you really do mean regression tests in general, by all mean say regression tests. But consider that there are many types of regression tests that might run at different times. It gets wordy, but I try to be as specific as possible: “post-deployment API regression tests”, “end-to-end user journey regression tests”, “release candidate regression tests”, “accessibility regression tests”, and so on. In all of those cases, the word “regression” doesn’t add anything other than imply that you run them regularly. The differentiators I find most useful are usually based on when the tests run — commit-time, post-deployment, etc — or what is being tested. That team that kept referring to one suite of tests as “the regression suite” now jokes that they can no longer speak of it around me; they can only talk about the “landing page test suite” or the “release candidate test suite” (which were once the same thing but now no longer are). But your mileage will vary.
If we use “regression suite” to only refer to a subset of the tests that run between a commit and deploying to prod, we’re missing the bigger picture of what that process actually requires. Your actual regression suite likely has many more components than you realize.
And now I’ll get off my soapbox…
Top comments (0)