DEV Community

Cover image for Load Private Module in Golang Project

Posted on • Updated on

Load Private Module in Golang Project

Load Private Module in Golang Project

Table of Contents

I. How Does go get Work?

Before we delve into the process of obtaining private Go modules, let's first take a look at how the go get command functions within the Go ecosystem. To illustrate this, we will perform a simple test:

1. Create Your Own Private Module:
Assume you have a private module hosted on a version control system (VCS) repository, such as This private module contains code that you want to use in your Go project.

2. Trying to Get This Module on Your Machine:
In a typical scenario, you might expect to be able to use the go get command to fetch the private module since your machine has the necessary permissions. However, you might encounter an unexpected challenge.

Image description

Image description

When you run go get {module}, the Go tooling, by default, attempts to fetch the module from a public Go proxy. If the module is not found there, it will then fall back to attempting to pull from the public repository associated with the module. If the module is still not found or you do not have the necessary permissions to access the private module, you will encounter an error.

II. How to Load Private Modules

To load a private module from a specific environment, you can follow the steps below:

1. Specifying Private Modules

Let Go Know Which Modules Are Private to Load Directly.

To address this issue, Go provides the GOPRIVATE environment variable. This variable allows you to specify a pattern for module paths that should be considered private. When a module path matches the patterns specified in GOPRIVATE, the Go tooling will attempt to fetch the module directly from the source repository rather than relying solely on public proxies.

For example, to mark the domain as private, you can set the GOPRIVATE variable like so:

go env -w GOPRIVATE='*'
Enter fullscreen mode Exit fullscreen mode

To verify the result, you can run:

go env | grep GOPRIVATE
Enter fullscreen mode Exit fullscreen mode

2. Configuring Credentials

To access private modules, you may need to configure credentials for authentication. There are several ways to achieve this:

Use SSH:
You can configure Git to use SSH for authentication when fetching private modules. To do this, run the following command:

git config --global url."ssh://".insteadOf ""
Enter fullscreen mode Exit fullscreen mode

Configure Basic Authentication for HTTPS:

  • .netrc File in Linux: The .netrc file is used to store credentials for various applications. To configure basic authentication for accessing private modules, create or update the .netrc file:
echo "machine login $GITLAB_USERNAME password $GITLAB_TOKEN" > ~/.netrc
Enter fullscreen mode Exit fullscreen mode

Note: You should create an Access token for it instead of using your password

III. Build Docker Image

If you need to build a Docker image for your project that imports private modules, let's see my Dockerfile example:

# Stage 1: Building the Application
FROM golang:1.21 as builder

# Pass build-time arguments for Git credentials

# Set GOPRIVATE to specify private module pattern
RUN go env -w GOPRIVATE='*'

# Set up authentication credentials using .netrc
RUN echo "machine login $GIT_USER password $GIT_TOKEN" > ~/.netrc

# Set the working directory

# Copy go.mod and go.sum to enable efficient dependency fetching
COPY go.mod .
COPY go.sum .

# Download project dependencies
RUN go mod download

# Copy the rest of the application code
COPY . .

# Build the application binary
RUN go build -o app

# Stage 2: Creating the Final Minimal Image
FROM ubuntu:20.04

# Set the working directory

# Copy the binary from the builder stage
COPY --from=builder /app/app .

# Define the command to run the application
CMD ["./app"]
Enter fullscreen mode Exit fullscreen mode

In this Dockerfile example, we've divided the process into two stages for clarity and optimization:

1. Building the Application (builder stage):

This stage is responsible for fetching dependencies, compiling the application, and preparing it for execution. It utilizes build-time arguments (GIT_USER and GIT_TOKEN) to configure authentication credentials for accessing private modules. The GOPRIVATE environment variable is set to ensure that private modules are fetched correctly. The ~/.netrc file is used to store authentication credentials securely.

2. Creating the Final Minimal Image:

In this stage, we copy the compiled binary (app) from the builder stage into this final image. This image does not hold credentials.

Building the Docker Image:

To build the Docker image using the provided Dockerfile, you can use the following command:

docker build \
--build-arg GIT_USER=jack \
--build-arg GIT_TOKEN='{YOUR_TOKEN}' \
-t my-golang-app .
Enter fullscreen mode Exit fullscreen mode

Replace {YOUR_TOKEN} with your actual Git token. This command sets the build-time arguments GIT_USER and GIT_TOKEN to provide the required authentication credentials during the image build process.

IV. Run it on gitlab-ci

So how to pass credentials into build time when using gitlab-ci?

1. Configure your credentials in Repo/Group Variables:

You can configure GIT_USER and GIT_TOKEN in CI/CD Variables:

Your repo OR Group > Settings > CI/CD > Variables > Add Variable

Image description

2. Write gitlab-ci.yml:

Now, you can access these variables in gitlab-ci:

  - build

  stage: build
    - docker build --build-arg GIT_USER=$GIT_USER --build-arg GIT_TOKEN=$GIT_TOKEN -t .
    - docker push
Enter fullscreen mode Exit fullscreen mode

V. Additional Resources

Depending on your objectives and the scale of your company, it's prudent to consider suitable models. Explore the array of options described in this article: Link

VI. References

Top comments (0)