DEV Community

Andrew
Andrew

Posted on

How FirstPort manage GitHub, using code stored in GitHub

#github #terraform #devops #tutorial

FirstPort GitHub

Introduction

As many of you will already know, I recently started an exciting new role as Head of Technology at FirstPort, having previously been at a great consultancy called DevOpsGroup.

A key part of my role is delivering FirstPort’s vision of ‘People First’ technology. To do this, it is imperative that I select the right technology to underpin the delivery of services that help make customers’ lives easier.

Today I want to talk about my selection of GitHub Enterprise

GitHub is a best-in-breed, secure, collaborative, code hosting platform with built-in CI/CD capabilities with over 56 million users.

Many organisations use it, and all of them have to manage users, repositories, branch rules, and much more. Standardising all of this is quite challenging and involves creating things like CLI tools, scripts, and other ways to try to automate how people deal with GitHub.

Here at FirstPort, I decided to use Terraform to manage and standardise all of the repository and user management, and today I want to share with you how we are doing this!

Team and user management

If you are using GitHub enterprise, you will probably use Single Sign-on (SSO). This does the job but isn’t great, especially when having to manage collaborators outside of your Identity Provider (However, you can use Terraform to manage sync groups) - This still leaves you to manage repositories, labels, pull requests all manually. In our case, I decided to use Terraform's GitHub provider to automate most of this work.

Firstly, I created the teams:

# Creates a parent team
resource "github_team" "engineering" {
  name        = "engineering"
  description = "This is a parent team"
  privacy     = "closed"
}

# Creates a sub team of front-end developers
resource "github_team" "frontend" {
  name           = "frontend"
  description    = "This is a sub team "
  parent_team_id = github_team.engineering.id
  privacy        = "closed"
}
Enter fullscreen mode Exit fullscreen mode

Every time we have someone new join, anyone at FirstPort can open a PR and invite them to the team.

# Invites a user to the organization
resource "github_membership" "anewstarter" {
  username = "firstportuser"
  role     = "member"
}

# Adds the user to a team
resource "github_team_membership" "frontend-firstportuser1" {
  username = github_membership.firstportuser.username
  team_id  = github_team.frontend.id
  role     = "maintainer"
}
Enter fullscreen mode Exit fullscreen mode

This will run in a GitHub Actions workflow, and the user will receive an invite to join our organisation.

Repository creation

This is probably the main reason why I decided to use Terraform for managing GitHub. If repositories aren't standardised this can create a chaotic situation if the way engineers open issues, raise PRs etc is different in every repository. To avoid this I decided to create most of our repositories in the same way.

A few things were very important to us that every repository had to have. Let's go through the mains ones:

Release notes

I wanted to provide a way to be able to easily create release notes, to enhance transparency on what we are delivering. I chose to use a tool called Release Drafter in our workflow. I wanted to configure how labels would be used in every repository, and for that, we standardised all labels across all repositories. I use: docs, dependencies, bug, feature, and maintenance.

Every time someone opens a PR, they need to select at least one of these labels to define how this PR is categorised (This is enforced using the Enforce Label Action). When a PR is merged, I use the GitHub Action release drafter, which creates a draft release with the commits from the PR into one of those categories. This helps us to create release notes for all of our services!

This is how the release notes would look like:

Release Drafter

Every time someone creates a new repo, the release drafter comes pre-configured and you can start enjoying good release notes without any effort.

Branch protection

Setting up rules for when you can merge a branch is very important for code quality, and in our case for legal reasons (at FirstPort, we deal with personal user data, so all changes must go through strict code quality checks). GitHub gives the possibility for you to configure branch protection rules to solve that.

I wanted to define the same set of basic branch protection for all repositories. This was easily solved with our Terraform module.

Terraform module

To achieve all of this, I created a Terraform module that manages users, teams, and repositories. Here is an example on how we use the module:

module "my-firstport-repo" {
  source = "./modules/repo"
  version = "~> 1.0"


  name                   = "my-firstport-repo"
  description            = "A firstport repository description"
  visibility             = "private"
  delete_branch_on_merge = true
  vulnerability_alerts   = true
  teams                  = { (data.github_team.engineering.id) = "push" }
  collaborators          = { (data.github_user.external.username) = "push" }

  labels = {
    "type: bug"          = "d73a4a"
    "type: docs"         = "0f727f"
    "type: feature"      = "a2eeef"
    "type: maintenance"  = "a5f7da"
    "type: dependencies" = "0366d6"
  }

  branch_protection = {
    master = {
      enforce_admins    = true
      push_restrictions = []
      required_status_checks = {
        strict   = true
        contexts = []
      }
      required_pull_request_reviews = {
        dismiss_stale_reviews      = true
        require_code_owner_reviews = true
        dismissal_restrictions     = []
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

GitHub Actions

GitHub Actions is our CI tool of choice. GitHub Actions is a hosted runner service provided by GitHub. Any user can write individual tasks, called actions, and put them together into a workflow. These workflows can trigger off numerous events, such as pull requests, comments, labels, releases, and so forth. I think of it as having a box of LEGO bricks that can be put together as needed; I can build a space ship or a pirate ship as my heart desires.

Users are free to write their own actions or consume them from the GitHub Marketplace. For example, the action that performs code checkout is written by GitHub and is on the Marketplace. For a more in-depth introduction to GitHub Actions, I suggest reading the Getting started with GitHub Actions documentation.

For the purpose of this article, I am using GitHub Actions to construct a workflow to provide CI functionality.

There is one workflow that runs on every PR:

name: Terraform Plan
on:
  - pull_request

jobs:
  terraform:
    name: Terraform
    runs-on: ubuntu-latest

    env:
      ARM_CLIENT_ID: ${{ secrets.CLIENT_ID }}
      ARM_CLIENT_SECRET: ${{ secrets.GH_TERRAFORM_PRD }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.SUB_ID }}
      ARM_TENANT_ID: ${{ secrets.TENANT_ID }}
      ARM_ACCESS_KEY: ${{secrets.GH_TERRAFORM_PRD_STATE_BLOB_KEY}}

    steps:
      - name: Checkout
        uses: actions/checkout@master

      - name: "Security Scan"
        uses: triat/terraform-security-scan@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Lint Code Base
        uses: github/super-linter@master
        env:
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
          VALIDATE_ALL_CODEBASE: true
          VALIDATE_MD: true
          VALIDATE_TERRAFORM: true

      - uses: hashicorp/setup-terraform@master
        with:
          terraform_version: latest

      - name: Terraform Format
        id: fmt
        run: terraform fmt -check

      - name: Terraform Init
        id: init
        run: terraform init

      - name: Terraform Validate
        id: validate
        run: terraform validate -no-color
        env:
          TF_VAR_github_token: ${{ secrets.GH_TOKEN }}

      - name: Terraform Plan
        id: plan
        run: terraform plan -no-color
        continue-on-error: true
        env:
          TF_VAR_github_token: ${{ secrets.GH_TOKEN }}

      - uses: actions/github-script@master
        env:
          PLAN: "terraform\n${{ steps.plan.outputs.stdout }}"
        with:
          github-token: ${{ secrets.GH_TOKEN }}
          script: |
            const output = `#### Terraform Security Scan ☠ \`Success! The configuration is secure.\`
            #### Terraform Format and Style 🖌 \`${{ steps.fmt.outcome }}\`
            #### Terraform Initialization ⚙️ \`${{ steps.init.outcome }}\`
            #### Terraform Validation 🤖 ${{ steps.validate.outputs.stdout }}
            #### Terraform Plan 📖 \`${{ steps.plan.outcome }}\`

            <details><summary>Show Plan</summary>

            \`\`\`${process.env.PLAN}\`\`\`

            </details>

            *Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`, Working Directory: \`${{ env.tf_actions_working_dir }}\`, Workflow: \`${{ github.workflow }}\`*`;

            github.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            })

      - name: Terraform Plan Status
        if: steps.plan.outcome == 'failure'
        run: exit 1
Enter fullscreen mode Exit fullscreen mode

On a merge, we run the following workflow:

name: Terraform Apply
on:
  push:
    branches:
      - main

jobs:
  terraform:
    name: Apply
    runs-on: ubuntu-latest

    env:
      ARM_CLIENT_ID: ${{ secrets.CLIENT_ID }}
      ARM_CLIENT_SECRET: ${{ secrets.GH_TERRAFORM_PRD }}
      ARM_SUBSCRIPTION_ID: ${{ secrets.SUB_ID }}
      ARM_TENANT_ID: ${{ secrets.TENANT_ID }}
      ARM_ACCESS_KEY: ${{ secrets.GH_TERRAFORM_PRD_STATE_BLOB_KEY }}

    steps:
      - name: Checkout
        uses: actions/checkout@master

      - uses: hashicorp/setup-terraform@master
        with:
          terraform_version: 0.14.3 

      - name: Terraform Init
        id: init
        run: terraform init

      - name: Terraform Apply
        id: apply
        run: terraform apply -auto-approve -input=false
        env:
          TF_VAR_github_token: ${{ secrets.GH_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

Release Drafter is set up as follows:

name: Release Drafter

on:
  push:
    branches:
      - main

jobs:
  update_release_draft:
    runs-on: ubuntu-latest

    steps:
      - uses: release-drafter/release-drafter@master
        env:
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

with the following YAML for config:

categories:
  - title: '🚀 Features'
    labels:
      - 'feature'
  - title: '🐛 Bug Fixes'
    labels:
      - 'bug'
  - title: '🧰 Maintenance'
    label: 'maintenance'
  - title: '📖 Docs'
    label: 'docs'
  - title: '⚙️ Dependencies'
    label: 'dependencies'
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
change-title-escapes: '\<*_&' # You can add # and @ to disable mentions, and add ` to disable code blocks.
template: |
  ## Changes

  $CHANGES
Enter fullscreen mode Exit fullscreen mode

To ensure everything is captured in the release notes, we also use the Enforce Label Action:

name: Enforce PR labels

on:
  pull_request:
    types: [labeled, unlabeled, opened, edited, synchronize]
jobs:
  enforce-label:
    runs-on: ubuntu-latest

    steps:
    - uses: yogevbd/enforce-label-action@master
      with:
        REQUIRED_LABELS_ANY: "bug,docs,feature,maintenance,dependencies"
        REQUIRED_LABELS_ANY_DESCRIPTION: "Select at least one label ['bug','docs','feature','maintenance','dependencies']"
Enter fullscreen mode Exit fullscreen mode

This is what you see in the Pull Request Tab, each time the PR workflow runs. Allowing you to quickly see if a PR can be merged:

PR_Workflow

We also push this information into Microsoft Teams using the GitHub Integration App

GitHub_Teams

DevSecOps

An additional benefit of using a CI workflow is adding automated tests. In this scenario, I’ve added a step leveraging tfsec to scan for static code vulnerabilities. In the example below, tfsec warns against creating an Azure network security rule which is fully open. This will halt and fail the workflow unless I provide an ignore comment to accept the warning.

tfsec

Summary

In this post, I explored using GitHub Actions as a CI workflow that could build and maintain a GitHub organization including users, teams, permissions, security etc. I started by generating a new GitHub repository, then wrote the GitHub Workflow files, and finally started testing the CI workflow and introducing small, incremental changes.

Using Terraform to manage GitHub organisations can really help your developer teams to onboard quickly and securely.

I hope I could help you learn something new today, and share how we do things here at FirstPort.

Any questions, get in touch on Twitter

Top comments (0)

Read next

vmihailenco profile image

Uptrace v1.6 is available

Vladimir Mihailenco -

konmaz profile image

GUI in GitHub Codespaces

Konstantinos Mazgaltzidis -

evrone profile image

Developing a cloud backend for a streaming service

Evrone -

allanpablo profile image

Resilience and Failure Management in DevOps

Allan Pablo -