DEV Community

Cover image for Getting Started With Apache Kafka
Clivern
Clivern

Posted on

Getting Started With Apache Kafka

#microservices #kafka #broker #docker

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Let's first run a single kafka node with docker and docker-compose.

  • Create a docker-compose.yaml file 
---
version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.1.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:6.1.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
Enter fullscreen mode Exit fullscreen mode
  • After installing docker and docker-compose, Start kafka and zookeeper containers with the following command 
$ docker-compose up -d
Enter fullscreen mode Exit fullscreen mode
  • Get kafka container id to run some commands inside the container. 
$ docker ps

$ docker exec -it $ID bash
Enter fullscreen mode Exit fullscreen mode

To create a kafka topic clivern

$ cd /bin
$ kafka-topics --create \
  --bootstrap-server localhost:9092 \
  --topic clivern
Enter fullscreen mode Exit fullscreen mode

To run a consumer

$ cd /bin
$ kafka-console-consumer \
    --bootstrap-server localhost:9092 \
    --topic clivern
Enter fullscreen mode Exit fullscreen mode

To run a producer

$ cd /bin
$ kafka-console-producer \
    --bootstrap-server localhost:9092 \
    --topic clivern
Enter fullscreen mode Exit fullscreen mode

To list all topics

$ cd /bin
$ kafka-topics --bootstrap-server localhost:9092 --list
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

granitebps profile image

Keeping My Dev Machine Clean with Docker

Granite Bagas -

wimadev profile image

5 costly mistakes when deploying Docker containers (and how to dodge them like a pro 😎)

Lukas Mauser -

sidyguedes profile image

PHP/MySQL/Nginx/Redis Application Docker

Sidgley de Almeida Guedes -

vivekyadav200988 profile image

Understanding the ETL Process with Real-Time Data: Extraction, Transformation, Loading, and Visualization

Vivek Yadav -