My team is using testify as the testing framework for Go. We noticed that many methods provided by testify/suite can be easily achieved by the built-in testing package. For example, using Cleanup can do the same of AfterTest provided by testify/suite
. It makes me wondering if we can just use Cleanup
to replace as much methods from third-party packages as possible.
TL;DR
Cleanup
can replace methods that are not relevant to subtests, which are AfterTest
, TearDownAllSuite
and TearDownTestSuite
.
Cleanup
can’t replace TearDownSubTest
because there is a risk that a subtest pollute its parent test.
The experiment
I use a global variable externalResource
as the external resource. It’s value is 1
.
When preparing a subtest at SetupSubTest
, externalResource
is changed to 2
. At the same time, a cleanup function is registered to restore externalResource
to 1
.
I expect that externalResource
is set to 2
before the subtest and restored to 1
once the subtest is finished. So, from the point of parent test, externalResource
is always 1
. From the point of subtest, externalResource
is always 2
.
The code is
var externalResource = 1
type TestUnexpectedCleanupOrder struct {
suite.Suite
}
func (s *TestUnexpectedCleanupOrder) TearDownTest() {
fmt.Println("TearDownTest")
}
func (s *TestUnexpectedCleanupOrder) SetupSubTest() {
externalResource = 2
fmt.Printf("SetupSubTest. The externalResoure is: %d\n", externalResource)
s.T().Cleanup(func() {
externalResource = 1
fmt.Printf("SetupSubTest. The externalResoure is: %d\n", externalResource)
})
}
// Test case
func (s *TestUnexpectedCleanupOrder) TestCaseA() {
fmt.Printf("Start TestCase A. The externalResoure is: %d\n", externalResource)
s.Run("Sub-TestCase A", func() {
fmt.Printf("Finish Sub-TestCase A. The externalResoure is: %d\n", externalResource)
})
fmt.Printf("Finish TestCase A. The externalResource is: %d\n", externalResource)
}
The result
The output of the above code is (with line number)
1 Start TestCase A. The externalResoure is: 1
2 SetupSubTest. The externalResoure is: 2
3 Finish Sub-TestCase A. The externalResoure is: 2
4 Back to TestCase A. The externalResource is: 2
5 TearDownTest
6 Cleanup subTest. The externalResoure is: 1
Notice line 4 and line 6. The externalResource
is still 2
when the subtest is finished and the parent test is resuming! externalResource
is not cleaned until the parent test is finished.
According to the documentation of Cleanup
:
Cleanup registers a function to be called when the test (or subtest) and all its subtests complete. Cleanup functions will be called in last added, first called order.
It only guarantee that a registered function is called after something.
It does NOT guarantee when a registered function is called before something.
The solution
If we add a TearDownSubTest
to reset externalResource
, then its value is cleaned as we need
func (s *TestUnexpectedCleanupOrder) TearDownSubTest() {
externalResource = 1
fmt.Printf("\tTeardown subTest. The externalResoure is: %d\n", externalResource)
}
Start TestCase A. The externalResoure is: 1
SetupSubTest. The externalResoure is: 2
Finish Sub-TestCase A. The externalResoure is: 2
Teardown subTest. The externalResoure is: 1
Back to TestCase A. The externalResource is: 1
TearDownTest
Cleanup subTest. The externalResoure is: 1
Summary
We can use Cleanup
to clean up resources for normal tests. However, we need to use TearDownSubTest
to clean the resource held by subtests.
The source code
https://github.com/xuanyuwang/testify-examples/tree/main/unexpected-cleanup-order
Top comments (0)