Enable Developers on SAP BTP with Terraform, GitHub Actions and Backstage

Motivation

One important topic in the software industry is the Developer Experience (DevEx). Citing the blog post published on GitHub Blogs DevEx can be defined as follows:

“DevEx refers to the systems, technology, process, and culture that influence the effectiveness of software development. It looks at all components of a developer’s ecosystem—from environment to workflows to tools—and asks how they are contributing to developer productivity, satisfaction, and operational impact.”
(source: https://github.blog/2023-06-08-developer-experience-what-is-it-and-why-should-you-care/ ; visited 18.03.2024).

The main goal of DevEx is to enable developers in their work. In the year 2024 of course, this can be connected to (generative) AI and tools like GitHub CoPilot improve DevEx a lot.

However, this blog post has no signs of AI in it. I want to focus on another aspect of DevEx, namely giving the developers the best possible starting point for their work which is building business applications. In addition, I will cover a special ecosystem namely development on the SAP Business Technology Platform (SAP BTP).

Starting Point

Before diving into the technical ingredients of such a setup, I want to elaborate a bit more about the starting point. Let us a assume that a development team wants to start a new development project. The requirements that need to be covered are clear and the technical work can start. What’s next? The worst thing that could happen now is that the team must do the technical project setup from scratch. This comprises:

• Setting up their development environment (tooling etc.)
• Investigating on the best practices, rules, and guidelines that they need to adhere in the company and that are continuously evolving.
• Setup the project structure from a coding perspective
• Setup their source code repository
• Setup CI/CD
• Provide the infrastructure/resources in the cloud.

While these tasks are important and necessary, they basically prevent the team from getting their job done. “Reinventing the wheel” with every development project also means that the basic technical setup will not be aligned and look different for every project.
This also causes issues on the long run when the projects need to be maintained and the teams working on the projects change as they need to be onboarded into a “new” project although technically the basic setup should be the same.

Wouldn’t it be great to start with some development templates that already represent a basic project outline that comprises:

• The structure of a source code repository according to the best practices
• Some (basic) source code as blueprint following the best practices of the company.
• The infrastructure definition via Infrastructure as Code
• The CI/CD setup following the best practices and helping with staying compliant.

I am a big fan of devcontainers, so for me a sample setup should also contain a devcontainer definition.

Wouldn’t it be greater to have the ability to search for such curated templates in a developer portal and then (now fasten your seatbelt) to kick off the creation of the setup for the development project as a self-service?

What if I told you that this is already possible today? Curious? Then let us look how we can achieve this and get things cooking.

Ingredients

What do we need as ingredients for this setup? Let us start with the developer portal. There are several options available. The cloud-native ecosystem is in favor of Backstage which is a CNCF project (https://backstage.io/), so let us use this as ingredient “developer portal” in our recipe.

We also need to define the Infrastructure as Code. As we are looking at SAP BTP, we will use the Terraform provider for SAP BTP (https://registry.terraform.io/providers/SAP/btp/latest) for this task.

To deal with repositories and CI/CD we leverage the GitHub ecosystem, namely organizations on GitHub, repositories as well as GitHub Actions.

Note: you can of course also use other tools for that like e.g., GitLab as Backstage provides many integration options.

That’s already it. Now let’s mix these things together and see what we will get.

Recipe

Let us reflect on what we want to achieve: we want to have one (or multiple templates) that a development team can chose from in our Backstage developer portal. When selecting such a template we want to query the user some basic information about the project setup and then:

• Trigger the creation of a new GitHub repository that uses a predefined layout and provides some basic content.
• Trigger the creation of needed resources on SAP BTP. The configuration should also be part of the source code repository to keep it self-contained.
• Register the newly created project in the developer portal with a link to the source code repository.

Let's do that!

Laying the foundation in GitHub

We use an organization on GitHub. Within this organization we create a repository that will serve as a blueprint for the project setup. This could look like this:

The repository provides a layout of a sample project, the Terraform configuration, GitHub Actions for CI/CD and governance (like a dependabot configuration) and the definition of a devcontainer.

The concrete layout and content are tailored based on the requirements of and the best practices established by the company.

The sample code, the used ecosystem like Node.js, the used libraries and frameworks like CAP and the structure and tools (e.g., for linting) are all company specific topics and there is no silver bullet how to best structure that.

Nevertheless, I am convinced that every company can standardize this to some extent. There is an initial effort to achieve this, but the result helps developers a lot. Especially when thinking about governance topics: why not deliver a compliant pipeline out of the box that ensures requirements around security etc.?

The Backstage Part

Note: This will not be a comprehensive guide on how to setup Backstage. We will focus on the setup and integration of the template.

Once we have the sample GitHub repository in place let us do the setup on Backstage.

First you need to setup Backstage per se. I did a local setup of Backstage. The installation is described in the official documentation. I also added a local PostgreSQL and connected it to Backstage as described in the documentation.

There are some pitfalls that I stumbled across that I want to share:

• When initially creating the Backstage app via yarn, make sure that the build environment is set up in accordance with the Backstage requirements see the release notes of Backstage on “Scaffolder build requirements” as well as the information in this GitHub repository of isolated-vm. Otherwise, the yarn installation and the build will fail.
• When you first start Backstage locally, and you are using Node.js 20 make sure that you have set the environment variable NODE_OPTIONS='--no-node-snapshot'. Otherwise, the template execution will fail later.

Once Backstage is up and running, we want to configure a software template. The template consists of several parts:

• The metadata of the template
• The parameters that we want from our user to be able to execute the template
• The scaffolding actions that are executed to instantiate the template

The configuration is done in YAML format. I created a dedicated folder named btp-sample-remote to store the configuration.

Let us look at the layout of the template. The first part is the metadata:

apiVersion: scaffolder.backstage.io/v1beta3
# https://backstage.io/docs/features/software-catalog/descriptor-format#kind-template
kind: Template
metadata:
  name: sample-btpsubaccount-remote-template
  title: Remote Template for SAP BTP Subaccount Setup
  description: A remote template that creates a basic SAP BTP Subaccount setup
  tags:
    - sap
    - btp
    - basic
    - javascript
spec:
  owner: user:guest
  type: service

This represents what you will see in the Backstage developer portal as part of the template tile.

The next part is the parameters section that describe what input a user must provide and how the input flow will be structured:

parameters:
    - title: Project Data
      required:
        - projectName
        - region
      properties:
        projectName:
          title: Project Name
          description: Unique name of the project
          type: string
          maxLength: 50
          pattern: '^[a-zA-Z0-9_\\-]{1,200}'
          ui:autofocus: true
        region:
          title: Region of BTP Subaccount
          type: array
          items:
            type: string
            enum: 
              - 'us10'
              - 'eu10'
              - 'ap10'
          uniqueItems: true
          ui:widget: checkboxes    
    - title: Organizational Data
      required:
        - costCenter
        - stage
        - orgUnit
      properties:
        costCenter:
          title: Cost Center
          description: Cost center for the project
          type: string
          pattern: '^[0-9]{10}'
        stage:
          title: Development stage
          description: Development stage of the project
          type: array
          items:
            type: string
            enum:
              - 'DEV'
              - 'TST'
              - 'SBX'
            enumNames:
              - Development
              - Testing
              - Sandbox  
          uniqueItems: true
          ui:widget: checkboxes  
        orgUnit:
          title: Organizational Unit
          description: Organizational unit for the project
          type: array
          items:
            type: string
            enum:
              - B2B
              - B2C
              - ECOMMERCE
          uniqueItems: true
          ui:widget: checkboxes 

As you can see the visual representation of the input can be well structured and basic layout option as well as validations can be configured.

The last part of the template is the scaffolding action that will be executed when the user finished the input:

steps:
    # This step creates a new repository in the org's GitHub account.
    - id: fetchBase
      name: Fetch Base
      action: fetch:template
      input:
        url: https://github.com/btp-automation-scenarios/backstage-base-btp-template
        copyWithoutTemplating:
          - .devcontainer/*.json
          - .devcontainer/withenvfile/*.json
          - .github/workflows/*.yml
          - .github/**/*.yml
          - infra/*.tf
          - src/*.*

        values:
          name: ${{ parameters.projectName }}

    # This step publishes the contents of the working directory to GitHub.
    - id: createRepo
      name: Publish
      action: publish:github
      input:
        repoUrl: 'github.com?repo=${{ parameters.projectName }}&owner=btp-automation-scenarios'
        description: 'Project Repository for ${{ parameters.projectName }} by Backstage.io'
        repoVisibility: 'public' 
        defaultBranch: 'main'

    # This step triggers the execution of the Terraform setup via GitHub Actions.
    - id: setupViaGHAction
      name: Execute Terraform Setup via GitHub Actions
      action: github:actions:dispatch
      input:
        repoUrl: 'github.com?repo=${{ parameters.projectName }}&owner=btp-automation-scenarios'
        workflowId: 'create_base_project.yml'
        branchOrTagName: 'main'
        workflowInputs: 
          PROJECT_NAME: '${{ parameters.projectName }}'
          REGION: '${{ parameters.region[0] }}'
          COST_CENTER: '${{ parameters.costCenter }}'
          STAGE: '${{ parameters.stage[0] }}'
          ORGANIZATION: '${{ parameters.orgUnit[0] }}'


    # The final step is to register our new component in the catalog.
    - id: register
      name: Register
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['createRepo'].output.repoContentsUrl }}
        catalogInfoPath: '/catalog-info.yaml'

  # Outputs are displayed to the user after a successful execution of the template.
  output:
    links:
      - title: Repository
        url: ${{ steps['publish'].output.remoteUrl }}
      - title: Open in catalog
        icon: catalog
        entityRef: ${{ steps['register'].output.entityRef }}

Without going into the details, you can see that we use several actions that are available out-of-the-box in Backstage:

• The first action fetches the template from the GitHub repository and copies the content to the working directory. In this step it also executes some templating i.e., replacing placeholders with input data. That is why you see the exclusion of several files.

• The second action publishes the content to a new repository on GitHub.

• The third action triggers the execution of a GitHub Action workflow that is part of the newly created repository.

• The fourth and last action registers the new component in the Backstage catalogue so that we disclose the project to the developers.

The output section provides links to the newly created repository and the registered component in the Backstage portal.

To make this template visible in the portal we add the template to the app-config.yaml file of our Backstage application. The file contains the overall Backstage configuration. The template is added to the catalog section:

catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  rules:
    - allow: [Component, System, API, Resource, Location]
  locations:
    # Example of a template for SAP BTP subaccount setup 
    - type: file
      target: ../../btp-sample-remote/sap-btp-subaccount-remote.yaml
      rules:
        - allow: [Template]       

That's it from a Backstage app perspective.

To make the template repository a "real" template we add a catalog-info.yaml file to our GitHub repository. It contains some metatdata and will be filled with the user input (where applicable) during the templating step in Backstage:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.name | dump }}
  region: ${{ values.region | dump }}
  stage: ${{ values.stage | dump }}
  orgUnit: ${{ values.orgUnit | dump }}
  costCenter: ${{ values.costCenter | dump }}
spec:
  type: service
  owner: user:guest
  lifecycle: experimental

Last thing to do is to finalize the setup of the GitHub Action workflow that is triggered by the scaffolding action in Backstage.

The GitHub Action/Terraform part

The workflow is part of the template repository and is responsible for the Terraform setup. The workflow is triggered by the dispatch event once the new repository is created. It is defined as follows:

name: Basic Subaccount via Terraform

on:
  workflow_dispatch:
    inputs:
      PROJECT_NAME:
        description: "Name of the project"
        required: true
      REGION:
        description: "Region for the sub account"
        required: true
      COST_CENTER:
        description: "Cost center for the project"
        required: true
      STAGE:
        description: "Stage for the project"
        required: true
      ORGANIZATION:
        description: "Organization for the project"
        required: true

env:
  PATH_TO_TFSCRIPT: 'infra'

jobs:
  execute_base_setuup:
    name: BTP Subaccount Setup
    runs-on: ubuntu-latest
    steps:
    - name: Check out Git repository
      id: checkout_repo
      uses: actions/checkout@v4

    - name: Setup Terraform
      id : setup_terraform
      uses: hashicorp/setup-terraform@v3
      with:
        terraform_wrapper: false
        terraform_version: latest

    - name: Terraform Init
      id: terraform_init
      shell: bash
      run: |
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} init -no-color

    - name: Terraform Apply 
      id: terraform_apply
      shell: bash
      # We do not store the state - in a real setup we would reference a remote backend to store the state
      run: |
        export BTP_USERNAME=${{ secrets.BTP_USERNAME }}
        export BTP_PASSWORD=${{ secrets.BTP_PASSWORD }}
        terraform -chdir=${{ env.PATH_TO_TFSCRIPT }} apply -var globalaccount=${{ secrets.GLOBALACCOUNT }} -var region=${{ github.event.inputs.REGION }} -var project_name=${{ github.event.inputs.PROJECT_NAME }} -var stage=${{ github.event.inputs.STAGE }} -var costcenter=${{ github.event.inputs.COST_CENTER }} -var org_name=${{ github.event.inputs.ORGANIZATION }} -auto-approve -no-color           

As you can see the workflow receives the information needed for the setup via input variables.

The Terraform configuration is stored in the infra folder of the repository. The Terraform configuration is not part of this blog post but you can find a sample in the GitHub repository.

I stored the secrets needed for the setup in the GitHub organization settings, to make them available for all repositories in the organization.

And there is nothing left to do than ... to try it out.

Let's start cooking

We start the Backstage app and navigate to the developer portal. Next, we create a new project by selecting the template:

As defined in our template we are asked for some input that we happily provide:

After that we can review our input and kick off the setup:

The actions triggered by the scaffolding action are executed:

After successful execution we can navigate to the new project in the developer portal:

From there we can jump to the newly created repository:

And finally see the GitHub Action running for the setup:

Once the GitHub action has finished, we can check the provided subaccount in the SAP BTP cockpit:

So, mission accomplished!

You want to see this in action? Here you go:

.

Conclusion

With very few ingredients, we were able to provide a self-service for the setup of a new development project on SAP BTP. The setup is standardized self-contained and can be executed with a few clicks and enables the development team to start their development work immediately.

Of course, the setup presented here has some rough edges and further refinement is needed, but you see what is already possible today in SAP BTP development projects when combining the right ingredients.

It is worth to mention that there is a learning curve when it comes to Backstage. I clearly see the power of Backstage and the uncountable options that come with Backstage. But this is accompanied by a certain degree of complexity. Unfortunately, I do not think that the documentation is leveling this out. Nevertheless, the power of Backstage is from my perspective worth the effort.

What do you think?

I am curious to hear your thoughts on this setup. Do you see potential in this? Do you already have experience with comparable setups especially when it comes to development on SAP BTP?

I am looking forward to your comments and feedback.