DEV Community

Waldemar Quevedo
Waldemar Quevedo

Posted on

Introducing go_test.mod

#go

How to avoid meta test dependencies across Go modules

Since the Go v1.14 release, the go command now includes a little known flag called -modfile that can be used to manage multiple set of dependencies within the same repository.

The -modfile flag can also be really helpful to also manage better what are the dependencies that importers of your package end up bringing into their project, specially when dealing with test dependencies.

The problem

For example, let's say that we have the following couple of repos that do not have any dependencies:

# github.com/wallyqs/go-mod-a
package hello

func Hello() string {
    return "Hello"
}
Enter fullscreen mode Exit fullscreen mode 
# github.com/wallyqs/go-mod-b
package world

func World() string {
    return "World"
}
Enter fullscreen mode Exit fullscreen mode

Though for testing purposes, they do depend on each other. We add hello_test.go first in this case in the go-mod-a repo:

package hello

import (
    "testing"
    "github.com/wallyqs/go-mod-b"
)

func TestHello(t *testing.T) {
    got := Hello()
    expected := "Hello"
    if got != expected {
        t.Fatalf("Expected %v, got: %v", expected, got)
    }
}

func TestHelloWorld(t *testing.T) {
    got := fmt.Sprintf("%s %s", Hello(), world.World())
    expected := "Hello World"
    if got != expected {
        t.Fatalf("Expected %v, got: %v", expected, got)
    }
}
Enter fullscreen mode Exit fullscreen mode

The result of this will be something like the following in the
go.mod and go.sum in the go-mod-a repo:

$ cat go.mod 
module github.com/wallyqs/go-mod-a

go 1.17

require github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245 // indirect

$ cat go.sum 
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245 h1:vD21YUG9esVWngkuysL7tmR4l9EdwvaghEfdGNcBDVw=
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245/go.mod h1:IMoYMTpn8BaoMzQP+9DeRp7bHQv/n+jAhQfguxXc9oM=
Enter fullscreen mode Exit fullscreen mode

At this point, things look like this in the go-mod-a repo:

tree .
.
├── LICENSE
├── README.md
├── go.mod
├── go.sum
├── hello.go
└── hello_test.go
Enter fullscreen mode Exit fullscreen mode

Now, if we add the same test to the go-mod-b, we will have a circular testing dependency which Go modules is actually fine with:

package world

import (
    "fmt"
    "testing"
    "github.com/wallyqs/go-mod-a"
)

func TestWorld(t *testing.T) {
    got := World()
    expected := "World"
    if got != expected {
        t.Fatalf("Expected %v, got: %v", expected, got)
    }
}

func TestHelloWorld(t *testing.T) {
    got := fmt.Sprintf("%s %s", hello.Hello(), World())
    expected := "Hello World"
    if got != expected {
        t.Fatalf("Expected %v, got: %v", expected, got)
    }
}
Enter fullscreen mode Exit fullscreen mode 
$ ~/go/src/github.com/wallyqs/go-mod-b (main) $ go get github.com/wallyqs/go-mod-a
go: downloading github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460
go get: added github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460

$ ~/go/src/github.com/wallyqs/go-mod-b (main) $ go test ./... -v
=== RUN   TestWorld
--- PASS: TestWorld (0.00s)
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
ok      github.com/wallyqs/go-mod-b 0.014s
Enter fullscreen mode Exit fullscreen mode

In the go-mod-b repo, now we will have something like this though:

$ tree .
.
├── LICENSE
├── README.md
├── go.mod
├── go.sum
├── world.go
└── world_test.go

$ cat go.sum 
github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460 h1:alim5qw73L9EfbCEF7TW3KDlbfWoMs580PtS4+1fF38=
github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460/go.mod h1:+86UH/9vUOVQadGIBaupYvn1FMXQIzXpXqmGtq28JPk=
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245/go.mod h1:IMoYMTpn8BaoMzQP+9DeRp7bHQv/n+jAhQfguxXc9oM=
Enter fullscreen mode Exit fullscreen mode

Notice how go-mod-b includes a meta test dependency of itself, if left as is now all importers of go-mod-b will be actually importing two different dependencies of the same module which is not really necessary.

Let's consider that this is now v0.1.0 version of both packages, so we then update our original test dependencies to be a tagged version:

$ go get github.com/wallyqs/go-mod-b@v0.1.0
go: downloading github.com/wallyqs/go-mod-b v0.1.0
go get: upgraded github.com/wallyqs/go-mod-b v0.0.0-20210924202312-2deb2c1c1be4 => v0.1.0

$ ~/go/src/github.com/wallyqs/go-mod-a (main) $ cat go.mod
module github.com/wallyqs/go-mod-a

go 1.17

require github.com/wallyqs/go-mod-b v0.1.0
Enter fullscreen mode Exit fullscreen mode

Now after go mod tidy on the repo, the library will still be
bringing the last untagged version of itself as well:

$ cd ~/go/src/githubsrc/github.com/wallyqs/go-mod-a
$ cat go.sum
github.com/wallyqs/go-mod-a v0.0.0-20210924201556-e67af2677460/go.mod h1:+86UH/9vUOVQadGIBaupYvn1FMXQIzXpXqmGtq28JPk=
github.com/wallyqs/go-mod-b v0.0.0-20210924195232-cc93d43aa245/go.mod h1:IMoYMTpn8BaoMzQP+9DeRp7bHQv/n+jAhQfguxXc9oM=
github.com/wallyqs/go-mod-b v0.1.0 h1:3Qz5nQ0B98wlrwiXv7dEW2PhPfRqPB+dTmHPI08Dr/8=
github.com/wallyqs/go-mod-b v0.1.0/go.mod h1:BtD/yaPmz5Vf2OMu9+VcCNvR9I60VpeZTKvBQsUD5wg=
Enter fullscreen mode Exit fullscreen mode

How to solve this?

With the -modfile flag, we can better separate what dependencies are being imported if we change a bit how we use the testing dependencies. We will follow these steps then:

  • Make a test folder

  • Include the tests with test dependencies in the test folder. This means that the HelloWorld test is now moved to its own test file like:

// test/hello_test.go
package hello

import (
    "fmt"
    "testing"
    "github.com/wallyqs/go-mod-a"
    "github.com/wallyqs/go-mod-b"
)

func TestHelloWorld(t *testing.T) {
    got := fmt.Sprintf("%s %s", hello.Hello(), world.World())
    expected := "Hello World"
    if got != expected {
        t.Fatalf("Expected %v, got: %v", expected, got)
    }
}
Enter fullscreen mode Exit fullscreen mode 
$ tree ~/go/src/github.com/wallyqs/go-mod-a
├── LICENSE
├── README.md
├── go.mod
├── go_test.mod
├── go_test.sum
├── hello.go
├── hello_test.go
└── test
    └── hello_test.go
Enter fullscreen mode Exit fullscreen mode
  • Create a go_test.mod file: 
cp go.mod go_test.mod && go mod tidy -modfile go_test.mod
Enter fullscreen mode Exit fullscreen mode
  • Then use go test -modfile go_test.mod to run the tests with the extra deps: 
$ mkdir test
$ cp go.mod go_test.mod && go mod tidy -modfile go_test.mod

$ go test -modfile=go_test.mod ./... -v
=== RUN   TestHello
--- PASS: TestHello (0.00s)
PASS
ok      github.com/wallyqs/go-mod-a (cached)
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
ok      github.com/wallyqs/go-mod-a/test    (cached)
Enter fullscreen mode Exit fullscreen mode

Next, to generate a minimal go.mod for your library without the testing dependencies, just test the package:

$ cd ~/go/src/github.com/wallyqs/go-mod-a
$ rm go.mod
$ go mod init
$ go test github.com/wallyqs/go-mod-a
Enter fullscreen mode Exit fullscreen mode

To run the tests with the tests with extra dependencies you will need to use go_test.mod instead:

$ go test -modfile=go_test.mod -v github.com/wallyqs/go-mod-a/test
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
Enter fullscreen mode Exit fullscreen mode

And if you want to test all you can do:

$ go test ./... -modfile=go_test.mod -v
=== RUN   TestHello
--- PASS: TestHello (0.00s)
PASS
ok      github.com/wallyqs/go-mod-a (cached)
=== RUN   TestHelloWorld
--- PASS: TestHelloWorld (0.00s)
PASS
ok      github.com/wallyqs/go-mod-a/test    (cached)
Enter fullscreen mode Exit fullscreen mode

Look, no dependencies!

Since both packages didn't actually depend on each other other than for the tests this will mean that the go.mod is just empty:

cd ~/go/src/github.com/wallyqs/go-mod-a (main) $ cat go.mod
module github.com/wallyqs/go-mod-a

go 1.17
Enter fullscreen mode Exit fullscreen mode

Bonus: This will also mean that it does not depend on previous untagged versions of itself for testing!

cd ~/go/src/github.com/wallyqs/go-mod-a (main) $ cat go_test.mod 
module github.com/wallyqs/go-mod-a

go 1.17

require github.com/wallyqs/go-mod-b v0.1.0
Enter fullscreen mode Exit fullscreen mode

Upgrading a test dependency

Let's say now that we tag the next version of wallyqs/go-mod-a, we can then upgrade our test dependency in the wallyqs/go-mod-b as follows:

$ cd ~/go/src/github.com/wallyqs/go-mod-b (main) $ 
go get -modfile=go_test.mod github.com/wallyqs/go-mod-a@v0.2.0
Enter fullscreen mode Exit fullscreen mode

The go_test.mod as a result is just the latest tagged version:

head go*
==> go.mod <==
module github.com/wallyqs/go-mod-b

go 1.17

==> go_test.mod <==
module github.com/wallyqs/go-mod-b

go 1.17

require github.com/wallyqs/go-mod-a v0.2.0

==> go_test.sum <==
github.com/wallyqs/go-mod-a v0.2.0 h1:6o2qaAI3ssJhSx89cHvvY449+V3h/1Z09Z1ySoUPNwI=
github.com/wallyqs/go-mod-a v0.2.0/go.mod h1:N2dbiJxS0vlXnYKNv9J5JE6ULgOhcHA81/VkivvCr08=
Enter fullscreen mode Exit fullscreen mode

Summary

If possible, avoid including test dependencies to your library and instead move them to a subfolder of our package. Being able to do this is good for the ecosystem around your package since will mean less interactions with goproxy and simplified dependencies overall.

The example repos in this post can be found at:

We have been using this approach in the nats.go client for some time already where both the server and the client depend on each other for some of the tests. As a result, the go.mod of the client is very small and users do not require to download dependencies of the server:

https://github.com/nats-io/nats.go/blob/main/go.mod

module github.com/nats-io/nats.go

go 1.16

require (
    github.com/nats-io/nkeys v0.3.0
    github.com/nats-io/nuid v1.0.1
)
Enter fullscreen mode Exit fullscreen mode

Hope this helps!

  • Wally

Top comments (0)

Read next

ayushniroula profile image

10,000 Times Cleaner Code Just By Using This Tool.

Ayush Niroula -

ayushniroula profile image

Eradicate (era): A Powerful Open-Source Tool for Code Cleanup

Ayush Niroula -

mbv-labs profile image

Building a Saas in five days

MBV -

johnscode profile image

Sending IoT Device Data via MQTT broker.

J Fowler -