Automating RDS CA (Certificate Authority) Using AWS Lambda

In the last few days, we have dealt with the challenge of updating all our RDS CA services, running more than three hundred instances. Therefore, the AWS guidance is to update it manually as a matter of production critical behavior; however, manually doing it is frustrating. That's why we came up with the idea of deploying a Lambda code to handle it.

‼️ Disclaimer: As the data is sensitive, the examples will be explained using fictitious scenarios. I hope you grasp the main idea. If not, please reach out. It will be a pleasure to hear from you.

Why this is essential

Managing SSL/TLS certificates is crucial for maintaining the security of any database system. AWS RDS uses CA certificates to secure connections, and these certificates need regular updates to prevent service disruptions. Manually updating these certificates across multiple instances can be time-consuming and error-prone. Automating this process with AWS Lambda can save time and reduce the risk of human error. We hope to guide you through creating a Lambda function to automate RDS CA certificate updates.

Concept

The primary goal is to provide a step-by-step guide to automate the update of RDS CA certificates using an AWS Lambda function. The solution will iterate over specific RDS instances, verify their eligibility for updates, and apply the new CA certificate. This ensures that your RDS instances remain secure with minimal manual intervention.

Challenges

1. Handling Multiple Instances: Many environments consist of numerous RDS instances spread across different namespaces. Updating each instance manually is impractical.
2. Rate Limiting (Throttling): AWS imposes rate limits on API calls. When updating multiple instances, exceeding these limits can cause errors.
3. Ensuring Idempotency: The function must handle retries gracefully and ensure that updates are applied correctly without causing inconsistencies.
4. Namespace and Identifier Matching: Properly identifying which instances need updates based on their identifiers and namespaces requires careful string handling.

Step-by-Step Guide

Prerequisites

• An AWS account with appropriate permissions.
• Basic understanding of AWS Lambda and IAM roles.
• Python and Boto3 library installed.

Setting Up IAM Roles

Create an IAM role for your Lambda function with permissions to:

• Describe RDS instances.
• Modify RDS instances.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds:DescribeDBInstances",
        "rds:ModifyDBInstance"
      ],
      "Resource": "*"
    }
  ]
}

Writing the Lambda Function

Create a new Lambda function and attach the IAM role created in the previous step. Use the following code to automate the CA certificate updates:

import boto3
import logging
import time

# Logging configuration
logger = logging.getLogger()
logger.setLevel(logging.INFO)

# AWS region where your RDS instances are located
REGION_NAME = "<REGION>"

# New CA certificate ID
NEW_CA_CERTIFICATE_ID = "<CERTIFICATE>"

# Create an RDS client
rds_client = boto3.client('rds', region_name=REGION_NAME)

def lambda_handler(event, context):
    namespaces = event.get('namespaces', [])
    if not namespaces:
        logger.error("The event does not provide a namespace.")
        return {
            'statusCode': 400,
            'body': 'The event does not provide a namespace.'
        }

    for namespace in namespaces:
        logger.info(f"Updating instances for namespace: {namespace}")
        paginator = rds_client.get_paginator('describe_db_instances')
        page_iterator = paginator.paginate()

        for page in page_iterator:
            for instance in page['DBInstances']:
                instance_id = instance['DBInstanceIdentifier']

                # Check if instance_id ends with -0, -1, -2, etc. or serverless
                if instance_id.startswith(namespace) and (instance_id.split('-')[-1] in ['0', '1', '2']:
                    try:
                        rds_client.modify_db_instance(
                            DBInstanceIdentifier=instance_id,
                            CACertificateIdentifier=NEW_CA_CERTIFICATE_ID,
                            ApplyImmediately=True
                        )
                        logger.info(f"Successfully updated CA certificate for instance: {instance_id}")
                    except rds_client.exceptions.ThrottlingException as e:
                        logger.warning(f"Throttling encountered for instance: {instance_id}. Retrying...")
                        time.sleep(5)  # Wait for a few seconds before retrying
                        try:
                            # Retry the operation
                            rds_client.modify_db_instance(
                                DBInstanceIdentifier=instance_id,
                                CACertificateIdentifier=NEW_CA_CERTIFICATE_ID,
                                ApplyImmediately=True
                            )
                            logger.info(f"Successfully updated CA certificate for instance: {instance_id} after retry")
                        except Exception as retry_e:
                            logger.error(f"Error updating CA certificate for instance: {instance_id} after retry")
                            logger.error(retry_e)
                    except Exception as e:
                        logger.error(f"Error updating CA certificate for instance: {instance_id}")
                        logger.error(e)

    return {
        'statusCode': 200,
        'body': 'CA certificate update completed.'
    }

‼️ Detailed Code Explanation

• Initialization: Set up logging, region, and the RDS client.
• Handler Function: The lambda_handler function reads namespaces from the event, iterates over them, and processes each RDS instance.
• Namespace Iteration: For each namespace, the function paginates through RDS instances and checks if their identifiers match the required pattern (-0, -1, -2). > In our use case, the instances are called, for example, namespace-mock-0 | namespace-mock-1 | namespace-mock-2 -- and so on.
• Updating CA Certificates: For each matching instance, the function attempts to update the CA certificate. If throttling errors occur, it retries after a delay.
• Logging and Error Handling: Logs are used extensively to track progress and errors. The function handles throttling by retrying the operation after a short delay.

Configuring the Lambda Function

• Memory: Allocate enough memory to handle the load. Start with 512MB and adjust based on performance.
• Timeout: Set an appropriate timeout value. A higher timeout (e.g., 5 minutes) may be necessary for environments with many instances.
• Environment Variables: Use environment variables to store configuration values such as REGION_NAME and NEW_CA_CERTIFICATE_ID.

Setting Up Event Triggers

You can trigger the Lambda function manually or set up an automated trigger using Amazon CloudWatch Events to run periodically (PS: in our use case, running it manually was the correct option.)

Testing the Function

Create a test event in the Lambda console with the following JSON payload:

{
  "namespaces": ["namespace-mock-test", "namespace-mock-stg", "namespace-mock-prd"] # and so on.
}

Run the test and monitor the logs in CloudWatch to verify that the instances are being updated as expected.

At the end of the day

Please remember the following: "Automating maintenance tasks, such as updating RDS CA certificates, can greatly improve how efficiently things run and reduce the chance of mistakes made by people. AWS Lambda, with its serverless architecture, is a great way to automate tasks like these. The solution in this article automates the update process and handles common issues like throttling, making sure that the update process is strong and reliable. Using this kind of automation can help you easily keep your database environments secure and reliable."