Using go modules from private repositories in Azure DevOps Pipelines

This post will explain how to use go modules that you keep in private repositories in GitHub. Sometimes you have internal modules that you do not really want to expose to the open source community. There might be several reasons for it, for example you are still working on the solution and it is not ready to see public, or it is maybe a protected intellectual property. You keep your code in a private repo then and locally go get uses your ssh keys to access the repo and download the package to the go mod cache.

However, your CI/CD tooling like Azure DevOps (ADO) do not have access to the these private repositories immediately. For that it needs to be equipped with an SSH key that it then can use to access github. This post shows how you can configure it end to end.

In TL;DC what we will do is:

• generate an ssh key pair
• add a public key to the github repo
• upload private key to the Azure DevOps secure files
• configure the Azure DevOps pipeline via YAML
• have fun doing it

Generate private key pair

This is a simple step, let's generate a key pair to be used to authenticate ADO to GitHub.

ssh-keygen -t rsa -b 4096 -C "your@email.com"

When asked, save the key pair under mykey name. You will have mykey which stores the private key and mykey.pub with a public key.

Add public key to the github repo

Assuming your repo is called my-go-module, navigate to
https://github.com/{your_org}/my-go-module/settings/keysand upload the contents of mykey.pub to the deploy keys.

Upload private key to the Azure DevOps secure files

Now you need to upload the private key contents to the ADO secure files. You can find them in the Pipelines -> Library -> Secure files. Upload the mykey private file there and call it myPrivateKey.

Configure the Azure DevOps pipeline via YAML

Now we have all the things in place. Last thing to do is install the SSH key in the pipeline so that go get can use it to access github. For that we will use the InstallSSHKey task.

      - task: InstallSSHKey@0
        inputs:
          knownHostsEntry: <here you put github known host entry>
          sshPublicKey: <here you put your public key content> 
          sshKeySecureFile: myPrivateKey

First run this command and copy the line not starting with #. Paste it into the knownHostsEntry parameter. This will make sure git will not ask for adding github into the known_hosts file but it would be already there.

➜ ssh-keyscan  github.com
# github.com:22 SSH-2.0-babeld-95694f5e
# github.com:22 SSH-2.0-babeld-95694f5e
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
# github.com:22 SSH-2.0-babeld-95694f5e

Now copy contents of the file mykey.pub to sshPublicKey parameter and finally set sshKeySecureFile to the secure file name you have chosen (like myPrivateKey).

Now this task will configure access to the private repository. You need still do to one thing before you can download the module.

      - script: |
          git config --global url."git@github.com:{yourorg}/my-go-module".insteadOf "https://github.com/{yourorg}/my-go-module"
          go build
        displayName: 'Build the binaries'

This step is needed so that go get tries to access the module with ssh not with https by default. Substitute {yourorg} with your organization name or your nickname so that it matches the URI of your module.