You might have noticed that GO111MODULE=on
is flourishing everywhere. Many readmes have that:
GO111MODULE=on go get golang.org/x/tools/gopls@latest
In this short post, I will explain why GO111MODULE
exists, its caveats and interesting bits that you need to know when dealing with Go Modules.
Table of content:
From GOPATH
to GO111MODULE
First off, let's talk about GOPATH. When Go was first introduced in 2009, it was not shipped with a package manager. Instead, go get
would fetch all the sources by using their import paths and store them in $GOPATH/src
. There was no versioning and the 'master' branch would represent a stable version of the package.
Go Modules (previously called vgo -- versioned Go) were introduced with Go 1.11. Instead of using the GOPATH for storing a single git checkout of every package, Go Modules stores tagged versions with go.mod
keeping track of each package's version.
Since then, the interaction between the 'GOPATH behavior' and the 'Go Modules behavior' has become one of the biggest gotchas of Go. One environment variable is responsible for 95% of this pain: GO111MODULE
.
The GO111MODULE
environment variable
GO111MODULE
is an environment variable that can be set when using go
for changing how Go imports packages. One of the first pain-points is that depending on the Go version, its semantics change.
GO111MODULE
with Go 1.11 and 1.12
-
GO111MODULE=on
will force using Go modules even if the project is in your GOPATH. Requiresgo.mod
to work. -
GO111MODULE=off
forces Go to behave the GOPATH way, even outside of GOPATH. -
GO111MODULE=auto
is the default mode. In this mode, Go will behave- similarly to
GO111MODULE=on
when you are outside ofGOPATH
, - similarly to
GO111MODULE=off
when you are inside theGOPATH
even if ago.mod
is present.
- similarly to
Whenever you are in your GOPATH and you want to do an operation that requires Go modules (e.g., go get
a specific version of a binary), you need to do:
GO111MODULE=on go get github.com/golang/mock/tree/master/mockgen@v1.3.1
GO111MODULE
with Go 1.13
Using Go 1.13, GO111MODULE
's default (auto
) changes:
- behaves like
GO111MODULE=on
anywhere there is ago.mod
OR anywhere outside the GOPATH even if there is nogo.mod
. So you can keep all your repositories in your GOPATH with Go 1.13. - behaves like
GO111MODULE=off
in the GOPATH with nogo.mod
.
GO111MODULE
with Go 1.14
The GO111MODULE
variable has the same behavior as with Go 1.13.
Note that some slight changes in behaviors unrelated to GO111MODULE
happened:
- The
vendor/
is picked up automatically. That has the tendency of breaking Gomock (issue) which were unknowingly not usingvendor/
before 1.14. - You still need to use
cd && GO111MODULE=on go get
when you don't want to mess up your current project’sgo.mod
(that's so annoying).
So, why is GO111MODULE
everywhere?!
Now that we know that GO111MODULE
can be very useful for enabling the Go Modules behavior, here is the answer: that's because GO111MODULE=on
allows you to select a version. Without Go Modules, go get
fetches the latest commit from master. With Go Modules, you can select a specific version based on git tags.
I use GO111MODULE=on
very often when I want to switch between the latest version and the HEAD version of gopls
(the Go Language Server):
GO111MODULE=on go get golang.org/x/tools/gopls@latest
GO111MODULE=on go get golang.org/x/tools/gopls@master
GO111MODULE=on go get golang.org/x/tools/gopls@v0.1
GO111MODULE=on go get golang.org/x/tools/gopls@v0.1.8
GO111MODULE="on" go get sigs.k8s.io/kind@v0.7.0
The pitfall of go.mod
being silently updated
And to make things even worse, some projects have an even more complicated one-liners:
(cd && GO111MODULE=on go get golang.org/x/tools/gopls@latest)
Note: the
@latest
suffix will use the latest git tag of gopls. Note that-u
(which means 'update') is not needed for@v0.1.8
since this is a 'fixed' version, and updating a fixed version does not really make sense. It is also interesting to note that with@v0.1
,go get
will fetch the latest patch version for that tag.
That's yet another Go ideocracy: by default (and you can't turn that off), if you are in a folder that has a go.mod
, go get
will update that go.mod
with what you just installed. And in the case of development binaries like gopls or kind, you definitely don't want to have these appearing in the go.mod
file!
So the workaround is to give a one-liner that makes sure that you won't be in a go.mod
-enabled folder: (cd && go get)
does exactly that.
I hope that (sooner or later) we will have a clear separation of concerns between go get
that is adding a dependency to your go.mod
(like npm install) and go install
that is meant to install a binary without messing up your go.mod
.
- First caveat: we all use
go get
to install dev dependencies, so moving togo install
would kind of not work (habits...) - Second caveat:
go install
doesn’t allow you to give a version (e.g.,@latest
or@v1.4.5
), andgo run
either by the way. Sogo install
isn't that useful after all... 😞
$ export GO111MODULE=on
$ go get golang.org/x/tools/gopls@v0.1.8 # ✅
$ go install golang.org/x/tools/gopls@v0.1.8 # ❌
can t load package: package golang.org/x/tools/gopls@v0.1.8: cannot use path@version syntax in GOPATH mode
$ go run golang.org/x/tools/gopls@v0.1.8 # ❌
package golang.org/x/tools/gopls@v0.1.8: can only use path@version syntax with 'go get'
The -u
and @version
pitfall
I have been bitten multiple times by this: when using go get @latest
(for a binary, at least), you should avoid using -u
so that it uses the dependencies as defined in go.sum
. Otherwise, it will update all the dependencies to their latest minor revision... And since a ton of projects choose to have breaking changes between minor versions (e.g. v0.2.0 to v0.3.0), using -u
has a large chance of breaking things.
So if you see this:
# Both -u and @latest!
GO111MODULE=on go get -u golang.org/x/tools/gopls@latest
then you will immediately realize that it is wrong: you want to be using the recorded versions given in go.sum
when go-getting a binary!
Rebecca Stambler reminds us that we should not use -u
in conjunction with a version:
-u
should not be used in conjunction with the@latest
tag, as it will give you incorrect versions of the dependencies.
But it's kind of hidden in this issue... I guess it is written somewhere in the Go help (btw, what a hideous help compared to git help
) but that kind of caveat should be more visible: maybe print a warning when installing a binary with both @version
and -u
?
Caveats when using Go Modules
Now, let's go through some caveats I encountered when working with Go Modules.
Remember that go get
also updates your go.mod
That’s one of the weird things with go get
: sometimes, it serves the purpose of installing binaries or downloading packages. But with Go modules, if you are in a repo with a go.mod
, it will silently add the package you go get to your go.mod.
That’s one of the catches of Go modules! 😁
Where are the sources of the dependencies with Go Modules
When using Go Modules, the packages that are used during go build
are stored in $GOPATH/pkg/mod
. When trying to inspect an 'import' in vim or VSCode, you might end up in the GOPATH version of the package instead of the pkg/mod one used during compilation.
A second issue that arises is when you want to hack one of your dependencies, for example for testing purposes.
Solution 1: use go mod vendor
+ go build -mod=vendor
. That will force go
to use the vendor/ files instead of using the $GOPATH/pkg/mod
ones. This option also solves the problem of vim and VSCode not opening the right version of a package’s file.
Solution 2: add a 'replace' line at the end of your go.mod
:
replace github.com/maelvls/beers => ../beers
where ../beers
is a local copy I made of the dependency I want to inspect and hack.
Set GO111MODULE
on a per-folder basis with direnv
During the migration from GOPATH-based projects (mainly using Dep) to Go Modules, I found myself struggling with two different places: inside and outside GOPATH. All Go Modules had to be kept outside of GOPATH, which meant my projects were in different folders.
To remediate that, I used GO111MODULE
extensively. I would keep all my projects in the GOPATH, and for the Go Modules-enabled projects, I would set export GO111MODULE=on
.
This is where direnv
comes in handy. Direnv is a lightweight command written in Go that will load a file, .envrc
, whenever you enter a directory and .envrc
is present. For every Go Module-enabled project, I would have this .envrc
:
# .envrc
export GO111MODULE=on
export GOPRIVATE=github.com/mycompany/\*
export GOFLAGS=-mod=vendor
The GOPRIVATE disables the Go Proxy (Go 1.13) for certain import paths. I also found useful to set -mod=vendor
so that every command uses the vendor
folder (go mod vendor
).
Private Go Modules and Dockerfile
At my company, we use a lot of private repositories. As explained above, we can use GOPRIVATE
in order to tell Go 1.13 to skip the package proxy and fetch our private packages directly from Github.
But what about building Docker images? How can go get
fetch our private repositories from a docker build?
Solution 1: vendoring
With go mod vendor
, no need to pass Github credentials to the docker build context. We can just put everything in vendor/
and the problem is solved. In the Dockerfile, -mod=vendor
will be required, but developers don't even have to bother with -mod=vendor
since they have access to the private Github repositories anyway using their local Git config
- Pros: faster build on CI (~10 to 30 seconds less)
- Cons: PRs are bloated with
vendor/
changes and the repo's size might be big
Solution 2: no vendoring
If vendor/
is just too big (e.g., for Kubernetes controllers, vendor/
is about 30MB), we can very well do it without vendoring. That would require to pass some form of GITHUB_TOKEN as argument of docker build
, and in the Dockerfile, set something like:
git config --global url."https://foo:${GITHUB_TOKEN}@github.com/company".insteadOf "https://github.com/company"
export GOPRIVATE=github.com/company/\*
Illustration by Bailey Beougher, from The Illustrated Children's Guide to Kubernetes.
Update 22 June 2020: it said use replace
instead of just replace
Discussion (23)
Great post, thanks for sharing. One question. We use
GO111MODULE=off
to install tools likegoimport
and not to pollute ourgo.mod
file:How is it worse than:
?
Thank you for this article Maël!
You might find this as a breath of fresh air, regarding "Remember that go get also updates your go.mod"
: groups.google.com/g/golang-tools/c...
go build, go test, and others - stop updating go.mod by default
Seems like there is a flag that certain Go tools have, the "-mod=readonly" flag. Might be worth adding to your article.
That's such a helpful post, so many of my questions got answered here. One more:
I haven't fully got this working yet. Could you elaborate more on this a bit more please?
go mod vendor
is actually doing, how would it be different without doing it? Would-mod=vendor
work without such step? and,-mod=vendor
is forgo build
right?vendor/
? Can you give a detailed example please -- i.e., Should I put mine in theGOPATH
way, or thego mod
way (with twisted name and added versions), Can I not put everything, but only a selectful of them thatgo get
is failing to get?Thanks
Hi! Thanks for the kind words!!! 🙂
go mod vendor
will vendor all the .go files that are imported by your project (it is smart and only vendors the files actually used, not the whole repo for each import path). It keeps track of what is being vendored invendor/modules.txt
(not sure but I think this module list is only "for the humans" so that we can see in a diff what import paths have been added tovendor/
)vendor/
, you first need to have modules turned on (go.mod
at the root of your project) and then you can rungo mod vendor
once and here you go. Whenever you add import paths in your project, you also want to re-rungo mod vendor
to add/delete any missing/unused vendored files.go mod vendor
you can fall back to using the good oldGOPATH
way (i.e.GO111MODULE=off
) and not care about modules anymore.go build
would by default use the modules in$GOPATH/pkg/mod
instead ofvendor/
and you had to rungo build -mod=vendor
vendor/
folder is used by default and if you want to force using the$GOPATH/pkg/mod
you can usego build -mod=mod
.I think
go mod vendor
puts things in the GOPATH way (except it only puts the files that are "actually used"). Not sure I answered the question properly though 😞Ah, right, I see the use case: you want to be able to manually add anything that
go get
doesn't know how to download (e.g. private networks, corporate firewalls).I guess you can definitely do that:
go mod vendor
for anything that works and manually copy-paste the import paths that are failing. I guess you would have to updatevendor/modules.txt
too but not sure. The important part is that the hash of the files you manually add must match the hashes in thego.sum
(I guess). 👍Thanks a lot for such detailed explanation.
Thanks Maël, you single-handedly saved me about 8 hours of RTFM. Now I can get started on a project that enjoys sane, 21st-century style version control of its dependencies. And thanks for the
direnv
link, which appears to work with Emacs (although I can't vouch for that just yet)Would you share something about govendor(github.com/kardianos/govendor) vs Go111module=on . I've a doubt about Go111module=on which saves one version of packages only in GOPATH but Govendor works as individual project's versions which exactly as node_modules. Any views appreciated.
Hi sameen,
Unfortunately, I do not know about
govendor
. But from what I understand, when building a binary withGO111MODULE=on
, a single version of each dependency will be used for the whole build. Russ Cox calls it minimal version selection (see his blog post on Dec 3rd, 2019).What do you mean?
GO111MODULE=on
will remember (in go.sum) the version of dependencies and save each dependency copy inGOPATH/pkg/mod
(i.e. one git clone at a fixed version if you will).Note that
GO111MODULE=on
works very differently fromnode_modules
since a single version is used per package, even when the package is used in multiple dependencies across the dependency tree.Thanks Maël
Maybe you can answer the following, related, question:
If I'm using Go 1.14, with go.mod, no vendor folder, should I have a GOPATH environment variable? Does this affect GO111MODULE in any way?
Thanks
/Martin
What a software language. A simple start require great effort and time. It must be great to become GO developer ( ͡° ͜ʖ ͡°).
Wanderful article to sooth my pain while I was trying to figure out the GOPATH and its minions. :D
I thank and salute your sir ...
Hi, your example able says
use replace github.com/maelvls/beers => ../beers
... but I think the worduse
is not supposed to be there. Thanks for the post, btw.Good point, thank you!!
I fixed that in the article 👍
It could be good to add note on behaviour in Go 1.14. Does it differ from one in Go 1.13?
Definitely, I’ll try to find time to update to 1.14 🙂
Awesome!
BTW, this is a go to post about understanding GO111MODULES. IMO should be linked / part of official docs.
Done! I updated with 1.14 and two "new" pitfalls 🙂
Reading the discussion I'm glad I'm not a lone who find Go things counter intuitive.
This article has helped me to solve the problem when
go get
did not create binary and exited with success.Great article thanks for sharing.
Great Article !
Great Article. Please also make a tutorial about Go modules
This was really useful. Thanks!
Very helpful! That spares me lot of pain.
Helps me a lot, thank you!