Pulling Git Submodules on Gitlab CI/CD Pipeline From a Different Repo

This is for solving the problem of building a repo with a different ssh key.

In this case, we had some legacy code on a Gerrit server, and we wanted to pull this code so we could build it on our Gitlab server.

In this example, our service account for Gerrit will be gitlab. The name shows that the Gerrit account is only for Gitlab to have access, and it is not tied to a real human's account. The linux user account on our workstation that we use for this example is testuser.

The 1st step is to create a new ssh key solely for this purpose.

I used ssh-keygen:

ssh-keygen -t rsa

When it askes for the name of the key, I use gerrit.pem.

Enter file in which to save the key (/home/testuser/.ssh/id_rsa): gerrit.pem

You should have 2 new files:

1. gerrit.pem - private key

2. gerrit.pem.pub - public key

Put the private key in your Gitlab variables and call it GERRIT_PEM:

https://docs.gitlab.com/ee/ci/variables/#define-a-cicd-variable-in-the-ui

Put the public key into Gerrit. See Register SSH public key.

https://haiyanghaiyang.gitbooks.io/gerrit-user-manual/content/sheng-cheng-yu-zhu-ce-ssh-key.html

Now you should test the ssh key on an linux account did not previously have access to the Gerrit repo. In the case, we'll call the user testuser.

Copy the gerrit.pem file to the testuser home folder and give ownership of it to testuser.

sudo cp gerrit.pem /home/testuser
sudo chown testuser /home/testuser/gerrit.pem

For this test, we will use ssh-agent.

ssh-agent is a way to allow us to put ssh keys into RAM so we can use keys which are not part of our ~/.ssh folder.

In this example, I pull one of the git submodules that I wanted to pull to make the test simple.

sudo su testuser # become testuser
eval "$(ssh-agent -s)" # start ssh-agent
ssh-add ~/gerrit.pub # add ssh key to the agent
git clone ssh://gitlab@gerrit.testexample.com/source_code # note this will change depending upon what your actual server, username, and source code is called.

This should pull the source code.

If this works, we are almost home free.

We need to add new Gitlab variables. In my case it was 2 new ones:
GERRIT_KNOWN_HOSTS1 and GERRIT_KNOWN_HOSTS2.

These are gotten from known hosts. Still as test user:

cat /home/testuser/.ssh/known_hosts

There should be 2 new lines here. Add them to Gitlab variables as the variable names above. We have to split each line into a separate variables because of the way that line breaks get broken if we don't, and the known_hosts becomes invalid due to a missing line break.

Next create a .gitlab-ci.yml file. We'll use the standard header:

build-code:
  stage: build
  image: centos:7

Next create the .ssh folder and known_hosts and change permissions. Ssh is super picky about these things:

    - mkdir ~/.ssh
    - touch ~/.ssh/known_hosts
    - chmod 600 ~/.ssh
    - chmod 644 ~/.ssh/known_hosts

Now start the ssh-agent. Note it has to be installed in the container:

    - eval "$(ssh-agent -s)"

Now add the key to ssh-agent. But this time, we don't have it on our disk, we need to pull from Gitlab variables that we set above:

    - ssh-add - <<< "${GERRIT_PEM}"

This is the magic. We need to populate known_hosts with what worked with testuser.

    - echo ${GERRIT_KNOWN_HOSTS1} >> ~/.ssh/known_hosts
    - echo ${GERRIT_KNOWN_HOSTS2} >> ~/.ssh/known_hosts

Next we need to tell ssh that we are using user gitlab instead of our container user. This will allow git submodules to work with a different user:

    - echo "Host gerrit.testexample.com" > ~/.ssh/config
    - echo "User gitlab" > ~/.ssh/config

Then we need to actually pull the submodules:

    - git submodule update --init --recursive

Finally, we need to build our code. Your build system may differ:

    - make

All in 1 script:

build-code:
  stage: build
  image: centos:7
  script:
    - mkdir ~/.ssh
    - touch ~/.ssh/known_hosts
    - chmod 600 ~/.ssh
    - chmod 644 ~/.ssh/known_hosts
    - eval "$(ssh-agent -s)"
    - ssh-add - <<< "${GERRIT_PEM}"
    - echo ${GERRIT_KNOWN_HOSTS1} >> ~/.ssh/known_hosts
    - echo ${GERRIT_KNOWN_HOSTS2} >> ~/.ssh/known_hosts
    - echo "Host gerrit.testexample.com" > ~/.ssh/config
    - echo "User gitlab" > ~/.ssh/config
    - git submodule update --init --recursive
    - make

Check this .gitlab-ci.yaml into your Gitlab repo and test it out.