How to set up Amazon RDS Proxy for Amazon Aurora (Serverless) database cluster and connect AWS Lambda function to it

#aws #serverless #postgres #java

Introduction

In this article we'd like to explore how to connect Lamdba function to the Aurora database through the RDS Proxy using Java JDBC capabilities. You can explore RDS Proxy capabilities in the Using Amazon RDS Proxy article. We will use Aurora Serverless v2 PostgreSQL database in this article but the same will also work Amazon Aurora PostgreSQL-Compatible Edition, Amazon Aurora MySQL-Compatible Edition, Amazon RDS for PostgreSQL, Amazon RDS for MySQL, Amazon RDS for MariaDB, and Amazon RDS for SQL Server as well. And it's easy to re-write Lambda function using different programming language as the basic concepts remain the same.

Solution

What we'd like to achieve is described in the following architecture diagram:

We'll focus on the part where Lambda connects through RDS Proxy to Aurora database. As the code sample we will use what we have created as part of the Data API for Amazon Aurora Serverless v2 with AWS SDK for Java series, and the project can be found on my GitHub repository. The relevant part of the Infrastructure as a Code can be found in this AWS SAM template.

First of all, we need to create VPC to put Lambda, RDS Proxy and Aurora Cluster to, so that Lambda is able to talk to RDS Proxy which in turn connects to Aurora Cluster. For that, please define your own VPC Id and Subnet list in the "Parameter" section of the SAM template like this :



  VpcId:
    Type: String
    Default: vpc-950cd6fd
    Description: VpcId of your existing Virtual Private Cloud (VPC)
  Subnets:
    Type: CommaDelimitedList  
    Default: subnet-0787be4d, subnet-88dc46e0
    Description: The list of SubnetIds, for at least two Availability Zones in the
      region in your Virtual Private Cloud (VPC)

Then we need to define Lambda Security Group which references the VPC Id:



  LambdaSecurityGroup:
      Type: AWS::EC2::SecurityGroup
      Properties:
        GroupDescription: SecurityGroup for Serverless Functions
        VpcId: 
          Ref: VpcId

and then VPC Security Group itself:



  VPCSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for RDS DB Instance.
      VpcId: 
         Ref: VpcId 
      SecurityGroupEgress: 
        - CidrIp: '0.0.0.0/0'
          Description: lambda RDS access over 5432
          FromPort: 5432
          IpProtocol: TCP
          ToPort: 5432
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '5432'
          ToPort: '5432'
          SourceSecurityGroupId:
               Ref: LambdaSecurityGroup

VPC Security Group defines Security Group engress and igress roules which enable Lambda to talk to RDS Proxy via the port number 5432 (default port for PostgreSQL) and references the defined Lambda Security Group.

Now let's create Aurora Serverless v2 PostgreSQL database cluster. But before the let's create SecretsManager to store the database user and password:



  DBSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Ref UserSecret
      Description: RDS database auto-generated user password
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DBMasterUserName}"}'
        GenerateStringKey: "password"
        PasswordLength: 30
        ExcludeCharacters: '"@/\'

Then we need to define DB Subnet Group:



  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: Subnets available for the RDS DB Instance
      SubnetIds:
       Ref: Subnets

and then create the Aurora Cluster with the engine "aurora-postgresql" itself:



 AuroraServerlessV2Cluster:
    Type: 'AWS::RDS::DBCluster'
    DeletionPolicy: Delete
    Properties:
      DBClusterIdentifier: !Ref DBClusterName
      Engine: aurora-postgresql
      Port: 5432
      EnableHttpEndpoint: true
      MasterUsername: !Join ['', ['{{resolve:secretsmanager:', !Ref DBSecret, ':SecretString:username}}' ]]
      MasterUserPassword: !Join ['', ['{{resolve:secretsmanager:', !Ref DBSecret, ':SecretString:password}}' ]]
      DatabaseName: !Ref DatabaseName
      ServerlessV2ScalingConfiguration:
        MinCapacity: 0.5
        MaxCapacity: 1
      DBSubnetGroupName:
        Ref: DBSubnetGroup
      VpcSecurityGroupIds:
      - !Ref VPCSecurityGroup

In the last lines we reference already created DB Subnet Group and Vpc Security Group Ids. With ServerlessV2ScalingConfiguration we define the scaling behaviour of our Aurora Serverless V2 Cluster.

After it we need to create database instance of the Aurora (Serverless v2) cluster :



  AuroraServerlessV2Instance:
    Type: 'AWS::RDS::DBInstance'
    Properties:
      Engine: aurora-postgresql
      DBInstanceClass: db.serverless
      DBClusterIdentifier: !Ref AuroraServerlessV2Cluster
      MonitoringInterval: 1
      MonitoringRoleArn: !GetAtt EnhancedMonitoringRole.Arn
      PubliclyAccessible: false
      EnablePerformanceInsights: true
      PerformanceInsightsRetentionPeriod: 7

Now let's take care of the creation of the RDS Proxy itself :



  RDSProxy:
    Type: AWS::RDS::DBProxy
    Properties:
      Auth:
        - { AuthScheme: SECRETS, SecretArn: !Ref DBSecret}
      DBProxyName: 'rds-proxy'
      RoleArn: !GetAtt RDSProxyRole.Arn
      EngineFamily: 'POSTGRESQL'
      IdleClientTimeout: 120
      RequireTLS: true
      DebugLogging: false
      VpcSecurityGroupIds:
      - !Ref VPCSecurityGroup
      VpcSubnetIds: !Ref  Subnets

We use EngineFamily 'POSTGRESQL' and AuthScheme connected to the created SecretsManager secret for authentication in our scenario. There are also other authentication possibilities like IAM authentication offered. We also connect RDS Proxy to the already created Vpc Security Group and Vpc Subnet Id.

We also need to create RDS Proxy Role to allow RDS Proxy to fetch secrets from the SecretsManager:



  RDSProxyRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Action: [ 'sts:AssumeRole' ]
            Effect: Allow
            Principal:
              Service: [ rds.amazonaws.com ]
      Policies:
        - PolicyName: DBProxyPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Action:
                  - secretsmanager:GetSecretValue
                Effect: Allow
                Resource:
                  - !Ref DBSecret

As the final step we need to create DB Proxy Target Group:



  ProxyTargetGroup:
    Type: AWS::RDS::DBProxyTargetGroup
    Properties:
      DBProxyName: !Ref RDSProxy
      DBClusterIdentifiers: [ !Ref AuroraServerlessV2Cluster]
      TargetGroupName: default
      ConnectionPoolConfigurationInfo:
        MaxConnectionsPercent: 5
        MaxIdleConnectionsPercent: 4
        ConnectionBorrowTimeout: 120

DB Proxy Target Group connects RDS Proxy with the Aurora V2 Cluster and defines Connection Pool Configuration. Please refer to Configure Connection Settings documentation for the extended explanation of this configuration.

Now let's connect Lambda function "GetProductByIdViaAuroraServerlessV2WithRDSProxy" to the RDS Proxy.
The relevant part is this one:



      Policies:
        - Statement:
            - Sid: AllowDbConnect
              Effect: Allow
              Action:
                - rds-db:connect
              Resource:
                - !Sub arn:aws:rds-db:${AWS::Region}:${AWS::AccountId}:dbuser:!Select [6, !Split [":", !GetAtt RDSProxy.DBProxyArn]]/*

        - VPCAccessPolicy: {}
      VpcConfig:
        SecurityGroupIds:
          - Fn::GetAtt: LambdaSecurityGroup.GroupId
        SubnetIds: !Ref  Subnets

In the VPCAccessPolicy part we reference already created Security Groups and Subnets. In the Policies part we allow Lambda to connect to our RDS Proxy instance by defining the exact resource ARN by knowing how the resource schema name works but extracting the appropriate RDSProxy ARN part from it. For the exact explanation please visit Creating and using an IAM policy for IAM database access.

The word of caution: only for the demonstration purpose I passed the database name and password as the Lambda environment variables to connect to RDS Proxy which introduces the security risk.



      Environment:
        Variables:
          DB_USER_PASSWORD: !Join ['', ['{{resolve:secretsmanager:', !Ref DBSecret, ':SecretString:password}}' ]]
          DB_USER_NAME: !Join ['', ['{{resolve:secretsmanager:', !Ref DBSecret, ':SecretString:username}}' ]]
          RDS_PROXY_ENDPOINT: !GetAtt RDSProxy.Endpoint

The proper solution is to use the stored database name and password in Amazon Secret Manager and then retrieve the in the Lambda function itself.

That it on the Infrastructure as a Code level.

Everything else will be done in the Lambda handler itself to connect to the RDS Proxy. In my case I wrote GetProductByIdViaAuroraServerlessV2RDSProxyHandler in Java and used JDBC for the connection with the PostgreSQL database driver in the pom.xml like this.



     <dependency>
       <groupId>org.postgresql</groupId>
       <artifactId>postgresql</artifactId>
       <version>42.5.4</version>
    </dependency>

Each programming languge in which Lambda function can be implemented offers its own capabilities to connect to the database.

Conclusion

In this article we explored how to connect Lamdba function to the Aurora database through the RDS Proxy and demonstrated the example of the corresponding Infrastructure as a Code part and the Lambda function written in Java which uses JDBC to connect to RDS Proxy.