DEV Community

Ehi Enabs for AWS Community Builders

Posted on

Deploying Amazon Managed Service for Apache Kafka (Amazon MSK) with CloudFormation

Amazon Managed Service for Apache Kafka (Amazon MSK) helps developers easily build scalable and resilient streaming applications by abstracting the complexities of Kafka infrastructure management.

Developed initially by LinkedIn, Apache Kafka has become a significant technology for real-time data processing and event streaming. Kafka is a distributed streaming platform which uses a publish-subscribe messaging model. Known for its scalability, reliability, fault tolerance, and durable storage capabilities, Kafka facilitates the seamless ingestion, processing, and delivery of massive volumes of data in real-time.

Whether it's processing website clickstreams, tracking sensor data from IoT devices, or powering real-time analytics, Kafka's versatility makes it a go-to solution for building robust streaming architectures.

Deploying Amazon Managed Service for Apache Kafka (Amazon MSK) with CloudFormation

AWS CloudFormation is a great tool designed to streamline infrastructure provisioning through code. With CloudFormation, developers can describe their AWS infrastructure using a simple, declarative template format, encompassing everything from EC2 instances to S3 buckets and IAM roles. This template is a blueprint for resource creation and configuration, ensuring consistency and reproducibility across environments. CloudFormation templates, typically written in JSON or YAML, articulate the desired state of the infrastructure, abstracting away the procedural steps needed to achieve that state.

The following is a CloudFormation Template for deploying Apache Kafka (Amazon MSK)

AWSTemplateFormatVersion: "2024-09-09"
Description: "Amazon MSK Deployment Example"

    Type: AWS::MSK::Cluster
      ClusterName: MyKafkaCluster
      KafkaVersion: "2.8.0"
      NumberOfBrokerNodes: 3
        InstanceType: kafka.m5.large
          - !Ref SubnetIds
          - !Ref SecurityGroupId
          InCluster: true
          DataVolumeKMSKeyId: !Ref KmsKeyId
            LogGroup: !Ref LogGroupName

    Description: "Subnet IDs for Kafka cluster"
    Type: List<AWS::EC2::Subnet::Id>
    Description: "Security Group ID for Kafka cluster"
    Type: AWS::EC2::SecurityGroup::Id
    Description: "KMS Key ID for data encryption at rest"
    Type: AWS::KMS::Key::Id
    Description: "Log Group Name for Kafka cluster logs"
    Type: String

Enter fullscreen mode Exit fullscreen mode

After creating the template, you can deploy your Amazon MSK with was cli by using the following command;

aws cloudformation create-stack --stack-name MyKafkaStack --template-body file://msk-deployment.yaml \
--parameters ParameterKey=SubnetIds,ParameterValue="subnet-12345678,subnet-23456789" \
ParameterKey=SecurityGroupId,ParameterValue="sg-12345678" \
ParameterKey=KmsKeyId,ParameterValue="arn:aws:kms:us-east-1:123456789012:key/abcd1234-abcd-1234-abcd-123456789012" \

Enter fullscreen mode Exit fullscreen mode


Deploying Amazon Managed Service for Apache Kafka (Amazon MSK) with AWS CloudFormation streamlines the process of setting up a Kafka cluster for real-time data streaming. By abstracting the complexities of infrastructure management, Amazon MSK enables teams to focus on building innovative streaming applications with ease.

Top comments (0)