Solving Database Connection Issues in a Serverless World Using Prisma’s Data Proxy

With Serverless a lot of headaches is now offloaded and outsourced to Cloud providers. We pay for what we use, scaling is no more our concern. Provisioning is more or less elastic and automatic. But among all good things, tradeoffs are a bitter reality. When we try to solve one problem we often introduce another problem that was non-existent in the first place.

With all the amazing benefits, Serverless comes with its own challenges. Local testing is one of them. Database connection management is another one and we will talk about it in more detail in this post.

The Problem

To query a database we need to first establish a TCP connection with the database server. We re-use this connection for subsequent queries instead of reconnecting every time. In practice, we actually use a pool of these connections and reuse them. This works well with always-on servers.

But in a Serverless world, we might face scaling issues. For example: In a project that I was working on, we had close to 40 Lambda functions. Each of them will read from/ write to the same database. With the increase in load, multiple instances of these Lambdas will run in parallel. Each of these ephemeral compute instances requires a separate connection.

But there is a limit on how many simultaneous connections you can have with the database.

In our case, we were using an AWS RDS instance. A t3 micro RDS instance allows only 80-90 connections, t3 large instance provides around 700 simultaneous connections. We were already hitting the connection limit in our developer environment with just 4 developers using t3 micro for development and testing purposes.

The Solution

We needed an external connection pooler. But doing it by yourself requires effort and expertise. Me being lazy looked outside. We were already using Prisma as ORM and Prisma offers a built-in solution: Prisma Data Proxy.

Prisma is an open-source next-generation ORM that provides a typesafe query builder, hassle-free migrations and a GUI client to view and edit your data.

In case you are unaware of Prisma: check out this webinar that we recently did on AWS RDS and Prisma.

Coming back to Prisma Data Proxy, it is very simple to use. Every database has a database connection URL. In exchange for this URL, Prisma Data Proxy provides a proxy URL. To query the database we will use this proxy URL. For migrations, we will use the original database connection URL.

Let's look at the details to use Prisma Data Proxy:

Step 1: Add your project to the Prisma Data Platform

Go to Prisma Data Platform. Sign in with your Github profile and select your project repository. The Platform will automatically detect the path of schema.prisma file.

Step 2: Provide your database URL

Provide your database URL. As of now, Data Proxy supports two regions only. select the region closer to your database.

Step 3: Get the Prisma Data Proxy URL

Get a Proxy URL in exchange for your database URL.

Step 4: Replace the URL in your environment file.

DATABASE_URL="prisma://aws-us-east-1.prisma-data.com/?api_key=j_5xpbBnfdcMfRH-HA7bBazLKirYR2FtwTankit0ItsYE6HhzZ8YoCSnVd5KyyeKks"

Step 5: Change the generator client in your schema.prisma file by adding previewFeatures flag.

generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["dataProxy"]
}

Step 6: Generate the client

To generate the client run the following command

PRISMA_CLIENT_ENGINE_TYPE='dataproxy' npx prisma generate

and you are done. No more connection issues. All your queries will use the connections from the connection pool provided by the Prisma Data Proxy.

Before we say Goodbye

I've found Prisma ORM very productive and I can't recommend you enough to try it. In case you need any help, they have a humble and helpful community on slack as well.

And you might want to check out AntStack. We at Antstack help you embrace Serverless stack by designing your business applications. We also let you make architectural decisions keeping cost and best practices in mind.

I hope you enjoyed reading the article as much as I enjoyed writing it.🤗