Corey Sutphin for ThinkNimble

Posted on Mar 30, 2020

How to automatically create and destroy Heroku Apps using Bitbucket Pipelines

#devops #bash #heroku #bitbucket

You may have heard of Heroku, a service that greatly simplifies the process of building, deploying, and hosting your web applications. If you've already used Heroku, then you know that deploying is as easy as running git push heroku master. In 2016 Heroku launched a feature called Review Apps, which are temporary, disposable apps that are created whenever a pull request is opened. They integrate directly with your Github repository, and are created when the pull request is opened and destroyed when it is closed. They provide you with a temporary and isolated environment to test your changes in, which becomes very useful in situations where multiple changes are being committed to a repository at once. If you are using Bitbucket as your version control platform then unfortunately Review Apps are not available to you, however with a little effort we can mimic their functionality. In this guide we will learn how to use the Heroku CLI along with Bitbucket Pipelines to listen for pull requests and programatically create and destroy copies of your application!

Note that for the rest of this guide when we say app, we are referring to a Heroku app.

Read This If...

You are an individual who wants the ability to host multiple versions of an application with different versions at one time.
You are a team working on multiple features at once, and you want to be able to test the different features in isolation of each other.
If you were looking to use Review Apps for your project, but are using Bitbucket!

Introduction

Over at ThinkNimble we needed a tool to dynamically spin up versions of our applications containing new features to test them in isolation. Because Review Apps are not integrated with Bitbucket, we had to get creative and chose to create a script that creates an app when a pull request is opened and destroys that app when the pull request is merged. We use Bitbucket Pipelines, a CI/CD platform that integrates directly with your repository, to hook into the pull-request lifecycle of Bitbucket. Pipelines allows you to run services such as automated testing or deployments whenever branches are updated or pull requests are created. These services are configured using a YAML file included at the root of your Bitbucket repository called bitbucket-pipelines.yml. We also utilize the Heroku CLI through a bash script to interact with the Heroku API for managing apps. At the end of the guide we share the full bitbucket-pipelines.yml configuration file and bash script necessary to get this automation configured.

Prerequisites

A Heroku API Key with the ability to create and delete apps. You can generate an API key for yourself on your Heroku account page.
A Bitbucket repository(repo) with Bitbucket Pipelines enabled.
A Bitbucket app password with the ability to read pull request and commit history for your repo. This can be generated by going to the Settings screen of your Bitbucket account. From there go to Access Management -> App Passwords and generate a new password with at least read permissions for repositories and pull-requests. Concatenate this with your username in the format BitbucketUsername:AppPassword and save it somewhere secure you can access later. This will be your authentication string to use the Bitbucket API.
A pre-configured Heroku app for your application.
Some previous experience with bash is recommended, but not required.

Setting up Heroku and Bitbucket

We need to allow Pipelines to access our Heroku API key, Heroku app name, and Bitbucket authentication string when running our various automation scripts. In the settings section of your repo navigate to Settings -> Pipelines -> Repository Variables. Add a HEROKU_API_KEY, HEROKU_APP_NAME, and BITBUCKET_AUTH_STRING repository variable, filling in the appropriate values. Make sure you check 'Secured' when adding sensitive variables so their values will be masked from any log outputs.

This will allow us to access these variables in our pipeline scripts as $HEROKU_API_KEY. Since these repository variables are set as environment variables, this HEROKU_API_KEY variable will also be used to authenticate our requests through the Heroku CLI.

Bitbucket Pipelines

First let's take a look at a bare-bones Pipelines configuration file that pushes code to an app whenever the master branch of our repository is updated.

# bitbucket-pipelines.yml

# Change this to your project's node version
image: node:12.0.0

clone:
  depth: full

# Deploy the master branch to our app.
# Pulls the Heroku API Key and the name of the app from our
# configured environment variables.
herokuAppDeployment: &herokuAppDeployment
  name: Deploy Heroku App
  script:
    - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_NAME.git master -f

pipelines:
  # Deploy the master branch to our app whenever it is updated
  branches:
    master:
      - step: *herokuAppDeployment

We are not going to be covering all of the set-up of a Pipelines configuration file, but at a high-level we are defining a pipeline for the master branch that will run our herokuAppDeployment script. This script will push the branch up to the git remote of our app. We now need to add a new pipeline to this file that will run on pull requests created/updated from a branch. This pipeline will create a new app for testing the changes added in the branch.

reviewAppDeployment: &reviewAppDeployment
  name: Deploy a Review App
  script:
    - echo "Deploy Review App"

pipelines:
  # Whenever a pull-request is opened or updated, create/update a Heroku app for the pull-request.
  pull-requests:
    "*":
      - step: *reviewAppDeployment

  # Deploy the master branch to our app whenever it is updated
  branches:
    master:
      - step: *herokuAppDeployment

Now that we have the structure set up to hook into opened and updated pull requests, let's start fleshing out the reviewAppDeployment script.

reviewAppDeployment: &reviewAppDeployment
  name: Deploy a Review App
  script:
    # Download the Heroku CLI
    - curl https://cli-assets.heroku.com/install.sh | sh
    # Format the name of the app that will be created.
    # EX: With a branch named 'chatbot-ui' and a repo called 'thinknimble', this will become 'thinknimble-chatbot-ui'.
    - NEW_APP_NAME=$BITBUCKET_REPO_SLUG-$BRANCH_NAME
    # Export the Heroku authentication token so it can be accessed in other scripts
    - export HEROKU_API_KEY
    # If an app exists for this branch, push the new code up.
    # Else, create a copy of the main app and push the new branch to it.
    - heroku apps:info -a $NEW_APP_NAME && git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$NEW_APP_NAME.git $BITBUCKET_BRANCH:master -f || /bin/bash copy-app.sh $HEROKU_APP_NAME $NEW_APP_NAME $BITBUCKET_BRANCH

In the first few steps of the function we download the Heroku CLI and construct the name of the app to be created from the name of the branch and the name of the repo. This name is important as it is later retrieved in order to find the app to delete when the pull request is merged. The final line of the function is where the magic happens. We first poll Heroku to see if the app exists. If it does, push the new code up to the app. If it doesn't, we need to create the new app by copying our main app. We have put the collection of commands required to create a copy of our main app in a bash script called copy-app.sh located at the root of our application. You ou could include all of the commands in the .yml file itself.

We now have a script that will run whenever a pull request is opened from a branch, and either update the app created from the branch or create a new app for the branch if it doesn't already exist .In the next section we'll cover the contents of this script and how it creates the new apps.

Creating Apps Using the Heroku CLI

When a pull request is opened for a new branch, we use Heroku CLI commands to create a new app, copy over the configuration and data from the main app, and deploy the source code of the branch. Some of the steps in this script may not be needed for your specific application, or you may need to add additional steps to configure other services.

# copy-app.sh

#!/bin/bash

#
# Script to copy an existing app and deploy a specific branch to it.
# 1. Creates the new app
# 2. Copies over all environment variables from existing app
# 3. Provisions a new Postgres database and copies the rows over from the existing app
# 4. Deploys the specified branch to the new app
#
# USAGE:
#   ./copy-app.sh parentApp newApp branchName
#

# Defines the parameters passed in to the script. Requires the name of the
# parent app, the name of the new app, and the name of the branch to push to the new app to build.
set -e
parentAppName=$1
newAppName=$2
branchName=$3

# Create the new app
heroku create "$newAppName"

# Add the standard buildpacks for your app. Here we add NodeJS first, then python.
heroku buildpacks:add heroku/nodejs -a "$newAppName"
heroku buildpacks:add heroku/python -a "$newAppName"

# Copy environemnt variables from parent app to new app
configFile="config-temp.txt"
heroku config -s -a "$parentAppName" > "$configFile"
cat "$configFile" | tr '\n' ' ' | xargs heroku config:set -a "$newAppName"
rm "./$configFile"

# Remove the DATABASE_URL environment variable as it will be set when we provision a new database
heroku config:unset DATABASE_URL -a "$newAppName"

# Update the ALLOWED_HOSTS and CURRENT_DOMAIN environment variables
heroku config:set ALLOWED_HOSTS="$newAppName.herokuapp.com" CURRENT_DOMAIN="$newAppName.herokuapp.com" -a "$newAppName"

# Provision new Postgres database
heroku addons:create heroku-postgresql:hobby-dev -a "$newAppName"

# Push the source code of the branch up to the new app to build
git push https://heroku:"$HEROKU_API_KEY"@git.heroku.com/"$newAppName".git $branchName:master -f

# Copy rows from parent app's database
heroku pg:copy "$parentAppName"::DATABASE_URL DATABASE_URL -a "$newAppName" --confirm "$newAppName"

Some of these configuration steps may not make sense for your specific application. Include any additional steps needed to configure your feature app in this file. While we use a pre-existing main app to copy over environment variables and testing data, you could include the testing configuration and seed data for the application in the source code itself to remove this dependency. This main app also does not need to be an app running the production version of your application as well, it could be a staging app itself where changes are vetted before being pushed to the production environment.

Destroying Apps

At this point we have a Pipelines configuration file and bash script that will create apps for us to test changes on whenever a pull request is opened, but as of now you will have to go in and manually delete the new apps once the PR from the branch is merged. To fix this we have implemented a script that uses the Bitbucket API to retrieve the pull request when it is merged, find the name of the created app for the branch, and subsequently delete that app. The only modifications needed are to the bitbucket-pipelines.yml file:

# bitbucket-pipelines.yml

# Change this to your project's node version
image: node:12.0.0

clone:
  depth: full


# Deploy the master branch to our app.
# Pulls the Heroku API Key and the name of the app from our
# configured environment variables.
herokuAppDeployment: &herokuAppDeployment
  name: Deploy Heroku App
  script:
    - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_NAME.git master -f

reviewAppDeployment: &reviewAppDeployment
  name: Deploy a Review App
  script:
    # Download the Heroku CLI
    - curl https://cli-assets.heroku.com/install.sh | sh
    # Format the name of the app that will be created.
    # EX: With a branch named 'chatbot-ui' and a repo called 'thinknimble', this will become 'thinknimble-chatbot-ui'.
    - NEW_APP_NAME=$BITBUCKET_REPO_SLUG-$BRANCH_NAME
    # Export the Heroku authentication token so it can be accessed in other scripts
    - export HEROKU_API_KEY
    # If an app exists for this branch, push the new code up.
    # Else, create a copy of the main app and push the new branch to it.
    - heroku apps:info -a $NEW_APP_NAME && git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$NEW_APP_NAME.git $BITBUCKET_BRANCH:master -f || /bin/bash copy-app.sh $HEROKU_APP_NAME $NEW_APP_NAME $BITBUCKET_BRANCH

reviewAppCleanup: &reviewAppCleanup
  name: Cleanup a Review App
  script:
    - curl https://cli-assets.heroku.com/install.sh | sh
    # Pull the commit that triggered this script, which for a pull request will be the merge commit. 
    # Retrieve the associated pull-request to get the review app name to delete.
    - BRANCH=`curl -X GET -G "https://$BITBUCKET_AUTH_STRING@api.bitbucket.org/2.0/repositories/thinknimble/$BITBUCKET_REPO_SLUG/pullrequests" --data-urlencode 'q=merge_commit.hash="'"$BITBUCKET_COMMIT"'"' | python -c "import json,sys;obj=json.loads(sys.stdin.read());value=obj.get('values', [{}])[0].get('source', {}).get('branch', {}).get('name', '');print(value)"`
    - BRANCH_NAME=`echo $BRANCH | egrep "(.*)" -o`
    - FEATURE_APP_NAME=$BITBUCKET_REPO_SLUG-$BRANCH_NAME
    # Poll Heroku to see if an app with the constructed name exists.
    # If it does than delete it, if it does not then just return.
    - heroku apps:info -a $FEATURE_APP_NAME && heroku apps:destroy -a $FEATURE_APP_NAME -c $FEATURE_APP_NAME || echo "No feature server to cleanup"

pipelines:
  # Whenever a pull-request is opened or updated, create/update a Heroku app for the pull-request.
  pull-requests:
    "*":
      - step: *reviewAppDeployment
  # Deploy the master branch to our main app whenever it is updated.
  # Then, update any Review Apps associated with the pull-request merged into `master`.
  branches:
    develop:
      - step: *herokuAppDeployment
      - step: *revewAppCleanup

Let's take a closer look at this review AppCleanup script.

reviewAppCleanup: &reviewAppCleanup
  name: Cleanup a Review App
  script:
    - curl https://cli-assets.heroku.com/install.sh | sh
    # Pull the commit that triggered this script, which for a pull request will be the merge commit. 
    # Retrieve the associated pull-request to get the review app name to delete.
    - BRANCH=`curl -X GET -G "https://$BITBUCKET_AUTH_STRING@api.bitbucket.org/2.0/repositories/thinknimble/$BITBUCKET_REPO_SLUG/pullrequests" --data-urlencode 'q=merge_commit.hash="'"$BITBUCKET_COMMIT"'"' | python -c "import json,sys;obj=json.loads(sys.stdin.read());value=obj.get('values', [{}])[0].get('source', {}).get('branch', {}).get('name', '');print(value)"`
    - BRANCH_NAME=`echo $BRANCH | egrep "(.*)" -o`
    - FEATURE_APP_NAME=$BITBUCKET_REPO_SLUG-$BRANCH_NAME
    # Poll Heroku to see if an app with the constructed name exists.
    # If it does than delete it, if it does not then just return.
    - heroku apps:info -a $FEATURE_APP_NAME && heroku apps:destroy -a $FEATURE_APP_NAME -c $FEATURE_APP_NAME || echo "No feature server to cleanup"

We utilize Bitbucket's REST API to retrieve the pull-request associated with a specific commit. This commit is provided by Pipelines, and when the pipeline was triggered by a pull-request being merged the commit will be the merge commit. Once we have the name of the pull-request, we construct the name of the created app, looking for the name that was constructed in the reviewAppDeployment script. We poll Heroku to see if an app with the constructed name exists. If it does, destroy it. Otherwise just print out that there is no app to cleanup and return.

Now you can merge you pull-request, check Heroku, and see that the created app has been taken down! We now have an automated pipeline to create apps for pull requests, and then destroy them when the branch is merged.

Security Considerations

Since you are destroying apps, you have to ensure that no important apps are accidentally destroyed. The script has a very specific naming convention for naming these review apps, so the risk of deleting an app would require an unfortunate naming error. In our organization we've nullified this risk by using a Heroku account that only has permission to delete apps from a separate Heroku team created solely for the purpose of housing these temporary review apps and other playground environments. You can modify the commands in the copy-app.sh command above to create apps only for a specific team by including the -t team-name flag to the commands to create the app and associated pipeline. The scripts in bitbucket-pipelines.yml will simply return if the user does not have access to delete or create an app. See the Heroku CLI documentation for more configuration options.

Looking Ahead

While an integrated solution such as Review Apps for Github would have been preferred, this combination of Bitbucket Pipelines and Heroku CLI commands does the job for our team. There are some known limitations and improvements we would like to make going forward.

Heroku app names are limited to a maximum of 30 characters, so if your constructed app name is longer than that the script will error out. We could store a shortened slug or just simply truncate names when they reach the character limit.
Some Heroku app configuration must be done manually, for instance copying over scheduled background tasks. This means that for a basic application in our stack this is fully automatic, but becomes more manual as app architecture becomes more complicated.

Conclusion

You can test this for your own application by including this Pipelines configuration file and bash script at the root of your repository, and opening up a pull request. You should now see a new app named after your branch! Let me know if you were able to use the information in this post to automate the creation and deletion of apps for Heroku.

# bitbucket-pipelines.yml

# Place this at the root of your repository

# Change this to your project's node version
image: node:12.0.0

clone:
  depth: full


# Deploy the master branch to our app.
# Pulls the Heroku API Key and the name of the app from our
# configured environment variables.
herokuAppDeployment: &herokuAppDeployment
  name: Deploy Heroku App
  script:
    - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_NAME.git master -f

reviewAppDeployment: &reviewAppDeployment
  name: Deploy a Review App
  script:
    # Download the Heroku CLI
    - curl https://cli-assets.heroku.com/install.sh | sh
    # Format the name of the app that will be created.
    # EX: With a branch named 'chatbot-ui' and a repo called 'thinknimble', this will become 'thinknimble-chatbot-ui'.
    - NEW_APP_NAME=$BITBUCKET_REPO_SLUG-$BRANCH_NAME
    # Export the Heroku authentication token so it can be accessed in other scripts
    - export HEROKU_API_KEY
    # If an app exists for this branch, push the new code up.
    # Else, create a copy of the main app and push the new branch to it.
    - heroku apps:info -a $NEW_APP_NAME && git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$NEW_APP_NAME.git $BITBUCKET_BRANCH:master -f || /bin/bash copy-app.sh $HEROKU_APP_NAME $NEW_APP_NAME $BITBUCKET_BRANCH

reviewAppCleanup: &reviewAppCleanup
  name: Cleanup a Review App
  script:
    - curl https://cli-assets.heroku.com/install.sh | sh
    # Pull the commit that triggered this script, which for a pull request will be the merge commit. 
    # Retrieve the associated pull-request to get the review app name to delete.
    - BRANCH=`curl -X GET -G "https://$BITBUCKET_AUTH_STRING@api.bitbucket.org/2.0/repositories/thinknimble/$BITBUCKET_REPO_SLUG/pullrequests" --data-urlencode 'q=merge_commit.hash="'"$BITBUCKET_COMMIT"'"' | python -c "import json,sys;obj=json.loads(sys.stdin.read());value=obj.get('values', [{}])[0].get('source', {}).get('branch', {}).get('name', '');print(value)"`
    - BRANCH_NAME=`echo $BRANCH | egrep "(.*)" -o`
    - FEATURE_APP_NAME=$BITBUCKET_REPO_SLUG-$BRANCH_NAME
    # Poll Heroku to see if an app with the constructed name exists.
    # If it does than delete it, if it does not then just return.
    - heroku apps:info -a $FEATURE_APP_NAME && heroku apps:destroy -a $FEATURE_APP_NAME -c $FEATURE_APP_NAME || echo "No feature server to cleanup"

pipelines:
  # Whenever a pull-request is opened or updated, create/update a Heroku app for the pull-request.
  pull-requests:
    "*":
      - step: *reviewAppDeployment
  # Deploy the master branch to our main app whenever it is updated.
  # Then, update any Review Apps associated with the pull-request merged into `master`.
  branches:
    develop:
      - step: *herokuAppDeployment
      - step: *revewAppCleanup

# copy-app.sh

# Place this at the root of your repository

#!/bin/bash

#
# Script to copy an existing app and deploy a specific branch to it.
# 1. Creates the new app
# 2. Copies over all environment variables from existing app
# 3. Provisions a new Postgres database and copies the rows over from the existing app
# 4. Deploys the specified branch to the new app
#
# USAGE:
#   ./copy-app.sh parentApp newApp branchName
#

# Defines the parameters passed in to the script. Requires the name of the
# parent app, the name of the new app, and the name of the branch to push to the new app to build.
set -e
parentAppName=$1
newAppName=$2
branchName=$3

# Create the new app
heroku create "$newAppName"

# Add the standard buildpacks for your app. Here we add NodeJS first, then python.
heroku buildpacks:add heroku/nodejs -a "$newAppName"
heroku buildpacks:add heroku/python -a "$newAppName"

# Copy environemnt variables from parent app to new app
configFile="config-temp.txt"
heroku config -s -a "$parentAppName" > "$configFile"
cat "$configFile" | tr '\n' ' ' | xargs heroku config:set -a "$newAppName"
rm "./$configFile"

# Remove the DATABASE_URL environment variable as it will be set when we provision a new database
heroku config:unset DATABASE_URL -a "$newAppName"

# Update the ALLOWED_HOSTS and CURRENT_DOMAIN environment variables
heroku config:set ALLOWED_HOSTS="$newAppName.herokuapp.com" CURRENT_DOMAIN="$newAppName.herokuapp.com" -a "$newAppName"

# Provision new Postgres database
heroku addons:create heroku-postgresql:hobby-dev -a "$newAppName"

# Push the source code of the branch up to the new app to build
git push https://heroku:"$HEROKU_API_KEY"@git.heroku.com/"$newAppName".git $branchName:master -f

# Copy rows from parent app's database
heroku pg:copy "$parentAppName"::DATABASE_URL DATABASE_URL -a "$newAppName" --confirm "$newAppName"

Top comments (1)

Alexey Yunoshev • Jun 12 '20

Thank you! That was really helpful

DEV Community