siegerts for Neon

Posted on Mar 8 • Originally published at neon.tech on Mar 6

Fullstack Serverless CI/CD in AWS Amplify Hosting with Postgres Database Branching

#awsamplify #javascript #typescript #webdev

This post will guide you through integrating AWS Amplify Hosting CI/CD with Neon Postgres with a focus on testing your application’s environment or feature branches using real data enabled by Neon’s database branching. This architecture automates your workflow and ensures each new feature or environment branch in your Amplify application has a corresponding, isolated database. This is especially useful if you’re building and deploying a Server Side Rendering (SSR) application using frameworks like Next.js, Nuxt, Astro, and SvelteKit.

Because both Neon and Amplify Hosting are serverless, the underlying infrastructure doesn’t require maintenance. Also, when not in use, both services scale down to zero. For Amplify Hosting, this involves serverless compute functions (i.e SSR), while for Neon, it pertains to the Postgres cluster compute.

Solution Overview

In this guide, we cover the deployment process of a serverless full-stack app using Amplify Hosting, leveraging its SSR compute capabilities. SSR enables running code and logic in a server environment, useful for tasks like using routing middleware, interfacing with third-party services, or connecting to a database.

To follow along, check out the YouTube video below that walks through in depth the process of deploying, and verifying the database branch creation in Neon when deploying a Nuxt SSR app that uses an Amplify Gen 2 backend with Amplify Auth. While the example features Nuxt for server-side APIs, the methodology applies to any SSR framework compatible with Amplify Hosting’s function-based compute.

For practical implementation, the scripts referenced in this guide (and video) are located in the Neon Branches with Amplify Hosting CI/CD repo on GitHub.

AWS Amplify Hosting

AWS Amplify Hosting is a fully managed CI/CD and hosting service that supports static sites, single-page applications (SPAs), and full-stack serverless SSR apps.

In Amplify Hosting, a typical workflow involves teams managing app branches that align with various environments, such as development, testing, and production. Each branch, like dev, test, and prod, corresponds to its respective environment and is accessible via a unique URL, which incorporates the first-level subdomain and the branch identifier. For example, https://<branch-name>.<amplify-app-id>.amplifyapp.com/.

Serverless Postgres

When developing an SSR app in AWS Amplify Hosting and requiring Postgres or an RDBMS, the choices for a serverless relational database are limited. It’s important that your database can handle your application traffic without manual adjustment since serverless SSR fullstack apps can scale out in response to bursts of request traffic. And then scale back down when not in use – similar to how Lambda (and cloud) functions operate.

Using Amazon RDS involves virtual private cloud (VPC) and architecture configurations, typically requiring a Lambda function or service proxy for database access to bridge the connection from Amplify Hosting to the VPC. Alternatively, directly exposing an RDS database online requires setting up Amazon RDS Proxy, essentially acting as PgBouncer, for scaling purposes.

Amazon DynamoDB (DDB) is another serverless database solution, but it doesn’t offer the relational capabilities inherent to RDBMS like Postgres, such as relational data models, complex joins, or transactions.

Neon is serverless Postgres with auto-scaling, easy setup, built-in connection pooling, and database branching. Neon’s branching capability allows developers to clone their entire Postgres cluster in seconds, creating isolated environments for testing new features without affecting the primary database branch. For these reasons, it pairs well with the serverless nature of Amplify Hosting and frontend fullstack SSR application architectures.

Amplify Hosting CI/CD Integration

This Amplify Hosting SSR app and Neon Postgres integration involves a custom Bash script (neon-ci.sh) to manage Neon database branches alongside Amplify app branches. This script dynamically updates the .env file’s DATABASE_URL environment variable with the database connection string for each branch. This .env file is then loaded into the application’s server-side runtime as environment variables. To interact with the Postgres database in serverless or edge contexts, the Neon Serverless Driver for JavaScript and TypeScript can be used. Additionally, database migrations can be integrated into the CI/CD pipeline since the connection string is available in the build time environment.

Here’s how the integration works

A new Git branch is created and pushed in a repo that is configured for fullstack continuous deployments in Amplify Hosting
During the CI/CD process, a new Neon Postgres branch will be created using the Neon CLI via a custom bash script
This branch’s connection string is injected into the DATABASE_URL parameter within a .env file.
The application is deployed and has access to the DATABASE_URL environment variable in server-side compute functions
Connect to your database using the Neon serverless driver

In order to set up this flow, you’ll need to complete the following steps

Set up your Amplify app and CI/CD pipeline as per your project requirements
Choose the Amazon Linux: 2023 build image and enable automatic service role creation (if you are not using another role)
Copy the neon-ci.sh script and amplify.yml build settings into the root of your app directory
Enable branch auto-detection for the app to create a new Amplify app branch for each new Git branch matching the configured pattern
Create a Neon project and create an API key
Create an SSM SecureString parameter for the Neon API key
Add the correct policy permissions to the Amplify app Service role for SSM Parameter Store and Amplify from the AWS CLI within the CI/CD build time
Modify neon-ci.sh invocation in your amplify.yml build settings with your specific Neon project ID, database name, and other parameters as needed
Push your changes and monitor the Amplify console for the deployment status – the Neon console will show the created branch(es)

Configuring Amplify Hosting Build Settings

To set up build settings in Amplify Hosting, first connect your code repository to AWS Amplify Hosting to activate the CI/CD pipeline. Amplify supports GitHub, Bitbucket, GitLab, or AWS CodeCommit as source control providers.

Create an Amplify app

Confirm the app settings

Choose the build image for your application (i.e. Amazon Linux: 2023) and enable automatic service role creation (if you are not using another role). This role will be updated to allow access to your Neon API key parameter in SSM Parameter Store.

Update the Amplify Hosting Build Settings

The app build settings are configured in the Amplify Hosting console or checked in as amplify.ymlwithin your code repository.

This amplify.yml file defines the CI/CD pipeline configuration for AWS Amplify Hosting. It is split into backend and frontend stages, with specific commands executed at different phases ( preBuild , build , and postBuild ) for both the backend and frontend stages.

Customize the build process by adjusting the amplify.yml file, which controls the actions taken during the CI/CD pipeline stages for both frontend and backend components.

version: 1
backend:
  phases:
    preBuild:
      commands:
        - sudo yum -y install jq
        - jq --version
        - npm i -g neonctl@v1
    build:
      commands:
        - nvm use 18
        - corepack enable
        - pnpm install
        - |
          if ["${AWS_BRANCH}" = "main"] || ["${AWS_BRANCH}" = "dev"]; then 
            pnpm amplify pipeline-deploy --branch $AWS_BRANCH --app-id $AWS_APP_ID
          else
            pnpm amplify generate config --branch main --app-id $AWS_APP_ID 
          fi

          bash neon-ci.sh create-branch --app-id $AWS_APP_ID --neon-project-id &lt;neon-project-id&gt; --branch-name $AWS_BRANCH --parent-branch main --api-key-param "&lt;ssm-param&gt;" --role-name &lt;neon-role&gt; --database-name &lt;neon-db-name&gt; --suspend-timeout 0
  cache:
    paths:
      - $(pnpm store path)
frontend:
  phases:
    preBuild:
      commands:
        - nvm use 18
        - corepack enable
        - npx --yes nypm i
    build:
      commands:
        - npm run build

  artifacts:
    baseDirectory: .amplify-hosting
    files:
      - "**/*"

Adjust as needed depending on your app framework requirements, and also make sure to update the neon-ci.sh input arguments. The following is a step-by-step breakdown of the sample amplify.ymlset up:

Backend:

Pre-Build Phase:
1. Installs jq for JSON processing
2. Installs the neonctl CLI tool globally using npm
Build Phase:
1. Sets the Node version using nvm and enables corepack for package manager version management
2. Based on the branch (main or dev), it deploys the backend Amplify app and then invokes the neon-ci.sh script to manage Neon database branches

Frontend:

Sets up Node, installs dependencies, and executes the build process for the frontend

Cache Configuration:

Caches the pnpm store path to enhance the speed of future builds

Creating a new Postgres Database in Neon

Visit the Neon Console, sign up, and create your first project by following the prompts in the UI. Then, create an API key for your Neon account. This key should be stored in SSM Parameter Store and accessed by your Amplify build scripts for authentication with Neon for database branching. Whether you’re working with an existing project or starting a new one, this project will host the database linked to your Amplify SSR app. When you create a new branch in the Amplify app, a corresponding database branch is automatically created in your specified Neon project.

Add the Neon CI bash script

Include the Neon bash CI script in your application directory. This will be referenced during the backend build stage of the CI/CD pipeline. This script can be modified, if needed, to account for your specific use case but the primary function is to create a branch for each Amplify app branch and retrieve the connection string.

The script has the following commands and options:

/neon-ci.sh &lt;command&gt; [options]

Commands
create-branch:
    Creates a new Neon database branch.

options:
    --app-id &lt;app-id&gt;
    --neon-project-id &lt;project-id&gt;
    --parent-branch-id &lt;parent-branch-id&gt;
    --api-key-param &lt;api-key-param&gt;
    --role-name &lt;role-name&gt;
    --database-name &lt;database-name&gt;
    --suspend-timeout &lt;suspend-timeout&gt;

cleanup-branches:
    Cleans up Neon database branches that no longer
    have corresponding Amplify app branches.

options:
    --app-id &lt;app-id&gt;
    --neon-project-id &lt;project-id&gt;
    --api-key-param &lt;api-key-param&gt;

These commands, specifically create-branch, are used during the backend build phase of the CI/CD process.

Update the Amplify Hosting app backend build role with the correct policy statements

Update the Amplify Hosting backend build role with the policies below to allow access to SSM Parameter Store and Amplify using the AWS CLI within the CI/CD build environment. The neon-ci bash script calls out to SSM and Amplify which will require the service role to have the correct permissions to access.

An example of this from the bash script:

# neon-ci.sh
...

function set_neon_api_key {
    export NEON_API_KEY=$(aws ssm get-parameter --name $NEON_API_KEY_PARAM --with-decryption --query Parameter.Value --output text)
    if [[-z "${NEON_API_KEY}"]]; then
        echo "ERROR: NEON_API_KEY is not set. Exiting."
        exit 1
    fi
}

...

To update the role, add the below inline policies to the Amplify app backend build role in AWS Identity and Access Management (IAM). Scope these resources as required by your app.

For SSM Parameter Store

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAmplifySSMCalls",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParametersByPath",
        "ssm:GetParameters",
        "ssm:GetParameter"
      ],
      "Resource": ["arn:aws:ssm:*:*:parameter/&lt;name-of-your-parameter&gt;"]
    }
  ]
}

For AWS Amplify access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Action": "amplify:ListBranches",
      "Resource": "arn:aws:amplify:*:*:apps/*/branches/*"
    }
  ]
}

Enable branch auto-detection for the app to create a new Amplify app branch

Update the applications repository settings to enable branch auto-detection. Once enabled, a new app branch will automatically create new Amplify app branches when a new Git branch is created and pushed to the repo.

Post build database branch clean up

If you want to automate database branch cleanup after each build, then you can include logic to use the clean up branches function in the Neon CI bash script. The cleanup-branches function will pull all of your Amplify Hosting app branch references with the aws cli and then compare those with the branch names in your Neon project.

_ Note: Make sure to test any destructive branch actions on test apps and branches first before deploying to production CI/CD pipelines._

...

postBuild:
        commands:
          - # EXAMPLE: only run the cleanup-branches command if you have tested 
          - |
            if ! ["${AWS_BRANCH}" = "main"] &amp;&amp; ! ["${AWS_BRANCH}" = "dev"]; then
              # bash neon-ci.sh cleanup-branches --app-id $AWS_APP_ID --neon-project-id &lt;neon-project-id&gt; --api-key-param "&lt;ssm-param&gt;"
            fi

...

Connecting to Postgres with Nuxt SSR Server Routes

Now, you can use the Neon Serverless Driver to connect to your database. For example, using Nuxt SSR in Amplify Hosting with server routes, you can query your database branch using a similar pattern as below:

// ./server/routes/api/shows.ts

import { neon } from "@neondatabase/serverless";
const { databaseUrl } = useRuntimeConfig();
const sql = neon(databaseUrl!);

export default defineEventHandler(async (event) =&gt; {

  const shows = await sql`SELECT * FROM netflix_shows limit 10`;

  return {
    shows,
  };
});

The environment variable is populated into Nuxt with useRuntimeConfig().

// nuxt.config.js

export default defineNuxtConfig({
  devtools: { enabled: true },
  runtimeConfig: {
    // The private keys which are only available within server-side
    databaseUrl: process.env.DATABASE_URL,

    // Keys within public, will be also exposed to the client-side
    public: {
      //
    },
  },
});

Best Practices and Considerations

Securely manage your Neon API key. Do not commit this key to git. Use a SSM Parameter Store SecureString to store and access your Neon API key securely within your build environment.
Implement logging within your build scripts to track the creation and deletion of database branches. This will be useful for troubleshooting and auditing your CI/CD process.
Integrate database migration scripts into your CI/CD pipeline to update the schema of your database branches automatically when necessary. Drizzle and Prisma can be integrated into your build process for this using the database branch connection string environment variable.

Conclusion

Integrating Neon database branches with Amplify Hosting CI/CD provides a flexible way to automate environment isolation and database creation – in a truly serverless way.

To get started with incorporating Serverless Postgres into your Amplify Hosting SSR apps, sign up and try Neon for free. Follow us on Twitter and join us in Discord to share your experiences, suggestions, and challenges.

DEV Community