Gitlab's Deploy Keys and Deploy Tokens for CI/CD

UPDATE: Poetry no longer supports deploy tokens on the package URLs due to a change on how they parse git urls, go read this other post for a solution

Scenario: Need to install a requirement that is only available on a private repository, from my CI pipeline.

There are two ways (I found) to do this, and the one you use depends on what you need to do:

1. I have a simple library with no dependencies or with only public dependencies
2. I have a dependency that depends on other private repositories

For context and illustration I used gitlab, python and poetry.

Simple Dependencies: Use a Deploy Token

A deploy token allows you to access a repository without your user name and password or ssh keys, and it is setup per repository or group.

How to:

1. Add the dependency on your requirements file as you would normally
2. Setup the deploy token on the private repository you want to access
3. Store the user and token on CI Variables on the repository you will access the private repository from
4. On the CI Script, fill in the deploy token using the CI variables.

Example:

Dependency on pyproject.toml

[tool.poetry.dependencies]
python = "^3.9"
private-project = {git = "ssh://git@gitlab.com/mundo03/sample-private-repo.git", rev = "main"}

Note the SSH URL is kept so you don;t have to use a Personal Access Token every time you install your dependencies locally.

gitlab-ci.yml

image: python:3.9-buster

poetry_token:
  script:
    # Install Poetry
    - pip install poetry
    # setup string replacement
    - export SSH_URL=ssh:\\/\\/git@gitlab.com\\/mundo03
    - export TOKEN_URL=https:\\/\\/$deploy_user:$deploy_token@gitlab.com\\/mundo03
    - sed -i "s/$SSH_URL/$TOKEN_URL/g" pyproject.toml
    # Install repo
    - poetry install
    # Use Clonned Repo
    - poetry run print_something "LOOK AT ME!!"

Notice sed is being used to replace the SSH URL with an HTTP URL that has the Deploy Token in it.

Complex Dependencies: Use a deploy key

Deploy keys also give you access to a repository with out your user name and password or SSH key, it is an SSH key you generate and can be used on multiple repositories, on multiple groups if necessary.

There are also Public deploy keys setup at Gitlab Instance level and can be given access to any of the projects in that instance by an admin.

How to:

1. Add the dependency on your requirements file as you would normally
2. Generate an ssh key
3. Deploy the generated public key on any repo you need
4. Set up ssh-agent on the CI script

Example

Dependency on pyproject.toml

[tool.poetry.dependencies]
python = "^3.9"
private-project = {git = "ssh://git@gitlab.com/mundo03/sample-private-repo.git", rev = "main"}

gitlab-ci.yml

image: python:3.9-buster

poetry_key:
  script:
      # Install OS dependencies
    - apt-get update -y -qq && apt-get install -y -qq make openssh-client python3-pip
    # Start ssh-agent
    - eval $(ssh-agent -s)
    # Add Private key to SSH Agent
    - ssh-add <(echo "$SSH_PK")
    # Add gitlab as known host
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
    # Install Poetry
    - pip install poetry
    # Install repo
    - poetry install
    # Use Clonned Repo
    - poetry run print_something "LOOK AT ME!!"

Enter the Vault

CI Variables are not the best way available to store a secret, for that gitlab has Secrets, which is meant to store things like SSH keys, Passwords Etc.

Read up: Secrets
Spoilers: You need a Hashicorp's Vault account

TL;DR

I have a dependency in my project that is hosted on a private repository, I need to install it during my CI/CD, I have two options:

1. Deploy Token: Are setup by project or group, I can add the token info in a CI variable and manipulate my requirements file to insert the token.
2. Deploy Keys: These are SSH keys that can be added to any project I need or in any group. I need to setup the SSH-Agent on the CI/CD script, the Public Key goes in the repository settings, the Private key goes in a CI variable or Vault, and use the ssh URL when requiring the dependency.

Have a look at the Complete Sample here:
Use a Deploy Token
Use a Deploy Key