EC2 instance as self-hosted GItHub runners

By default, GitHub offers hosted runners to run our workloads inside. For their Linux runners, GitHub offers runners with specs: 2-core CPU, 7 GB of RAM, 14 GB of SSD disk space.

Some of your CI/CD workloads may require more powerful hardware than GitHub-hosted runners could provide. In such case, you can configure an EC2 instance to act as your GitHub runner instead.

For example, you may run a c5.4xlarge instance type for some of your workloads, or even a r5.xlarge for workloads that process large data sets in-memory.

Let us begin:

#1. Create a GitHub personal access token

• Go to your GitHub Settings
  

• Click the Developer settings

• Click Personal access token to create one

• Create a new GitHub personal access token with the repo scope in the name of GH_PERSONAL_ACCESS_TOKEN. The action will use the token for self-hosted runners management in the GitHub account on the repository level.

#2. Create an AMI image

• Create a new EC2 instance based on a Linux distribution of your choice
• Connect to the instance

   ssh -i <path_to_pemkey> user_name@IP_ADDRESS

• Install packages and configure the instance according to your workflow requirements
• After that, land on the EC2 console, select your instance, right-click on it to find Image and templates , hover to see Create image, click it

• Enter the Image name, Image description, and check No reboot, and at last , hit Create
• The AMI should show up in the AMI console
• You can safely terminate the instance as it's of no use any more, of course, you could just keep it running if you plan on making any changes to it, however

#3. Create VPC with subnet and security group

• Create a new VPC and Subnet or use an existing
• Create a new Security group for the runners in the VPC and only allow outbound traffic to port 443, for pulling jobs from GitHub. No port opening for inbound traffic is necessay.

#4. Configure the environment secrets

• The following environment variables must be set for this action

• Go to your project repo settings to add the above Secrets

• Expand the Secrets drop-down and click Actions

• In the Actions secrets tab click New repository secret

• Name your secrets and add their values to be consumed by the action

• A complete listing of the required secrets should look something like

#5. Configure the GitHub workflow

• Create a new GitHub Action with the below example workflow
• Please don't forget to set up a job for removing the EC2 instance at the end of the workflow execution. Otherwise, the EC2 instance won't be removed and continue to run even after the workflow execution is finished.

Now you're ready to go!

Inputs

Outputs

Example workflow

ci.yml

name: ci
on: pull_request
jobs:
  start-runner:
    name: Start self-hosted EC2 runner
    runs-on: ubuntu-latest
    outputs:
      label: ${{ steps.start-ec2-runner.outputs.label }}
      ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      - name: Start EC2 runner
        id: start-ec2-runner
        uses: machulav/ec2-github-runner@v2
        with:
          mode: start
          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
          ec2-image-id: ${{ secrets.AMI }}
          ec2-instance-type: ${{ secrets.INSTANCE_TYPE }}
          subnet-id: ${{ secrets.SUBNET }}
          security-group-id: ${{ secrets.SECURITY_GROUP }}
          iam-role-name: my-role-name # optional, requires additional permissions
          aws-resource-tags: > # optional, requires additional permissions
            [
              {"Key": "Name", "Value": "ec2-github-runner"},
              {"Key": "GitHubRepository", "Value": "${{ github.repository }}"}
            ]
  do-the-job:
    name: Do the job on the runner
    needs: start-runner # required to start the main job when the runner is ready
    runs-on: ${{ needs.start-runner.outputs.label }} # run the job on the newly created runner
    steps:
      - name: Hello World
        run: echo 'Hello World!'
  stop-runner:
    name: Stop self-hosted EC2 runner
    needs:
      - start-runner # required to get output from the start-runner job
      - do-the-job # required to wait until the main job is done
    runs-on: ubuntu-latest
    if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      - name: Stop EC2 runner
        uses: machulav/ec2-github-runner@v2
        with:
          mode: stop
          github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
          label: ${{ needs.start-runner.outputs.label }}
          ec2-instance-id: ${{ needs.start-runner.outputs.ec2-instance-id }}

After a successful first workflow execution, you should see this