Load Private Module in Golang Project

Load Private Module in Golang Project

Table of Contents

• I. How Does go get Work?
• II. How to Load Private Modules
• III. Build Docker Image
• IV. Run it on gitlab-ci
• V. Additional Resources
• VI. References

I. How Does go get Work?

Before we delve into the process of obtaining private Go modules, let's first take a look at how the go get command functions within the Go ecosystem. To illustrate this, we will perform a simple test:

1. Create Your Own Private Module:
Assume you have a private module hosted on a version control system (VCS) repository, such as gitlab.com. This private module contains code that you want to use in your Go project.

2. Trying to Get This Module on Your Machine:
In a typical scenario, you might expect to be able to use the go get command to fetch the private module since your machine has the necessary permissions. However, you might encounter an unexpected challenge.

When you run go get {module}, the Go tooling, by default, attempts to fetch the module from a public Go proxy. If the module is not found there, it will then fall back to attempting to pull from the public repository associated with the module. If the module is still not found or you do not have the necessary permissions to access the private module, you will encounter an error.

II. How to Load Private Modules

To load a private module from a specific environment, you can follow the steps below:

1. Specifying Private Modules

Let Go Know Which Modules Are Private to Load Directly.

To address this issue, Go provides the GOPRIVATE environment variable. This variable allows you to specify a pattern for module paths that should be considered private. When a module path matches the patterns specified in GOPRIVATE, the Go tooling will attempt to fetch the module directly from the source repository rather than relying solely on public proxies.

For example, to mark the gitlab.com domain as private, you can set the GOPRIVATE variable like so:

go env -w GOPRIVATE='gitlab.com/*'

To verify the result, you can run:

go env | grep GOPRIVATE

2. Configuring Credentials

To access private modules, you may need to configure credentials for authentication. There are several ways to achieve this:

Use SSH:
You can configure Git to use SSH for authentication when fetching private modules. To do this, run the following command:

git config --global url."ssh://git@gitlab.com".insteadOf "https://gitlab.com"

Configure Basic Authentication for HTTPS:

• .netrc File in Linux: The .netrc file is used to store credentials for various applications. To configure basic authentication for accessing private modules, create or update the .netrc file:

echo "machine gitlab.com login $GITLAB_USERNAME password $GITLAB_TOKEN" > ~/.netrc

Note: You should create an Access token for it instead of using your password

III. Build Docker Image

If you need to build a Docker image for your project that imports private modules, let's see my Dockerfile example:

# Stage 1: Building the Application
FROM golang:1.21 as builder

# Pass build-time arguments for Git credentials
ARG GIT_USER
ARG GIT_TOKEN

# Set GOPRIVATE to specify private module pattern
RUN go env -w GOPRIVATE='gitlab.com/*'

# Set up authentication credentials using .netrc
RUN echo "machine gitlab.com login $GIT_USER password $GIT_TOKEN" > ~/.netrc

# Set the working directory
WORKDIR /app

# Copy go.mod and go.sum to enable efficient dependency fetching
COPY go.mod .
COPY go.sum .

# Download project dependencies
RUN go mod download

# Copy the rest of the application code
COPY . .

# Build the application binary
RUN go build -o app

# Stage 2: Creating the Final Minimal Image
FROM ubuntu:20.04

# Set the working directory
WORKDIR /app

# Copy the binary from the builder stage
COPY --from=builder /app/app .

# Define the command to run the application
CMD ["./app"]

In this Dockerfile example, we've divided the process into two stages for clarity and optimization:

1. Building the Application (builder stage):

This stage is responsible for fetching dependencies, compiling the application, and preparing it for execution. It utilizes build-time arguments (GIT_USER and GIT_TOKEN) to configure authentication credentials for accessing private modules. The GOPRIVATE environment variable is set to ensure that private modules are fetched correctly. The ~/.netrc file is used to store authentication credentials securely.

2. Creating the Final Minimal Image:

In this stage, we copy the compiled binary (app) from the builder stage into this final image. This image does not hold credentials.

Building the Docker Image:

To build the Docker image using the provided Dockerfile, you can use the following command:

docker build \
--build-arg GIT_USER=jack \
--build-arg GIT_TOKEN='{YOUR_TOKEN}' \
-t my-golang-app .

Replace {YOUR_TOKEN} with your actual Git token. This command sets the build-time arguments GIT_USER and GIT_TOKEN to provide the required authentication credentials during the image build process.

IV. Run it on gitlab-ci

So how to pass credentials into build time when using gitlab-ci?

1. Configure your credentials in Repo/Group Variables:

You can configure GIT_USER and GIT_TOKEN in CI/CD Variables:

Your repo OR Group > Settings > CI/CD > Variables > Add Variable

2. Write gitlab-ci.yml:

Now, you can access these variables in gitlab-ci:

stages:

  - build

build-docker:

  stage: build

  script:

    - docker build --build-arg GIT_USER=$GIT_USER --build-arg GIT_TOKEN=$GIT_TOKEN -t gitlab.com/jack/my-app .

    - docker push gitlab.com/jack/my-app

V. Additional Resources

Depending on your objectives and the scale of your company, it's prudent to consider suitable models. Explore the array of options described in this article: Link

VI. References

• Go Module Proxies
• Private Go Modules Article