In a recent post I showed a setup to run Kafka on Kubernetes locally using Persistent Volumes(PV) and Persistent Volume Claims(PVC), this post covers the setup using Kubernetes Storage Class(SC) and leveraging the default one provisioned automatically by Kind which is the Rancher local-path-provisioner.

This setup is simpler and uses less code than the previous one with the trade off of having a bit less control over the path of data externalized to the host machine while requiring some internal knowledge of Kind to set it up. In general I favor this approach for local development over the one from my previous post.

Strimzi is an awesome simpler alternative to achieve the same, check it out. My goal with this setup here is for learning and to have a more "realistic" Kubernetes setup on local development machine so I opted to not use Strimzi or Helm charts.

I have created and tested these approaches on a Linux Development machine. It should work for Mac and Windows with some minimal adjustments also but I have never tried it.

You can get the full source from Github repo where you will find the files and Quick Start for both aforementioned approaches. To clone the repo git clone git@github.com:mmaia/kafka-local-kubernetes.git.

Pre-reqs, install:

• Docker
• Kubectl
• Kind

The setup using Storage class

If you checked out the repo described above the setup presented here is under storage-class-setup folder. You will find multiple Kubernetes declarative files in this folder, please notice that you could also combine all files in a single one separating them with a line containing triple dashes ---, if combining them is your preference you can open a terminal and from the storage-class-setup folder run for each in ./kafka-k8s/*; do cat $each; echo "---"; done > local-kafka-combined.yaml this will concatenate all files in a single one called local-kafka-combined.yaml.

The main thing to notice in this setup below compared to the previous one is that you don't have any PV or PVC configuration files this time because we're leveraging the default Rancher local-path-provisioner provided by Kind automatically through it's default Storage class.

I keep them separate to explicitly separate each type in this case and because it's convenient as you can just run kubectl pointing to the directory as described below in the "Running it" section.

kind-config.yaml - This file configures Kind to expose the kafka and schema-registry ports to the local machine host so you can connect your application while developing from your IDE or command line and connect with Kafka running on Kubernetes.

kind-config.yaml - This file configures Kind to expose the kafka and schema-registry ports to the local machine host so you can connect your application while developing from your IDE or command line and connect with Kafka running on Kubernetes it also maps the default path of the Rancher storage provisioner from Kind container to your local host machine.

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
  - role: worker
    extraPortMappings:
      - containerPort: 30092 # internal kafka nodeport
        hostPort: 9092 # port exposed on "host" machine for kafka
      - containerPort: 30081 # internal schema-registry nodeport
        hostPort: 8081 # port exposed on "host" machine for schema-registry
    extraMounts:
      - hostPath: ./tmp
        containerPath: /var/local-path-provisioner
        readOnly: false
        selinuxRelabel: false
        propagation: Bidirectional

kafka-network-np.yaml - Sets up the internal Kubernetes network used by the setup.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: kafka-network
spec:
  ingress:
    - from:
        - podSelector:
            matchLabels:
              network/kafka-network: "true"
  podSelector:
    matchLabels:
      network/kafka-network: "true"

kafka-service.yaml - This file defines the mappings between the internal containers and ports that are exposed, called NodePorts by defaul in Kubernetes nodeports can be used in the range 30000 to 32767.

apiVersion: v1
kind: Service
metadata:
  labels:
    service: kafka
  name: kafka
spec:
  selector:
    service: kafka
  ports:
    - name: internal
      port: 29092
      targetPort: 29092
    - name: external
      port: 30092
      targetPort: 9092
      nodePort: 30092
  type: NodePort

kafka-ss.yaml - This is the definition of Kafka in this setup, this time we use a Stateful Set.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    service: kafka
  name: kafka
spec:
  serviceName: kafka
  replicas: 1
  selector:
    matchLabels:
      service: kafka
  template:
    metadata:
      labels:
        network/kafka-network: "true"
        service: kafka
    spec:
      enableServiceLinks: false
      containers:
      - name: kafka
        imagePullPolicy: IfNotPresent
        image: confluentinc/cp-kafka:7.0.1
        ports:
          - containerPort: 29092
          - containerPort: 9092
        env:
          - name: CONFLUENT_SUPPORT_CUSTOMER_ID
            value: "anonymous"
          - name: KAFKA_ADVERTISED_LISTENERS
            value: "INTERNAL://kafka:29092,LISTENER_EXTERNAL://kafka:9092"
          - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
            value: "true"
          - name: KAFKA_BROKER_ID
            value: "1"
          - name: KAFKA_DEFAULT_REPLICATION_FACTOR
            value: "1"
          - name: KAFKA_INTER_BROKER_LISTENER_NAME
            value: "INTERNAL"
          - name: KAFKA_LISTENERS
            value: "INTERNAL://:29092,LISTENER_EXTERNAL://:9092"
          - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
            value: "INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT"
          - name: KAFKA_NUM_PARTITIONS
            value: "1"
          - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
            value: "1"
          - name: KAFKA_LOG_CLEANUP_POLICY
            value: "compact"
          - name: KAFKA_ZOOKEEPER_CONNECT
            value: "zookeeper:2181"
        resources: {}
        volumeMounts:
          - mountPath: /var/lib/kafka/data
            name: kafka-data
      hostname: kafka
      restartPolicy: Always
  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

The remaining files are declarative Kubernetes configurations files to schema-registry and zookeeper.

schema-registry-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service: schema-registry
  name: schema-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      service: schema-registry
  strategy: {}
  template:
    metadata:
      labels:
        network/kafka-network: "true"
        service: schema-registry
    spec:
      enableServiceLinks: false
      containers:
        - env:
            - name: SCHEMA_REGISTRY_HOST_NAME
              value: "schema-registry"
            - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
              value: "kafka:29092"
            - name: SCHEMA_REGISTRY_LISTENERS
              value: "http://0.0.0.0:30081"
          image: confluentinc/cp-schema-registry:7.0.1
          name: schema-registry
          ports:
            - containerPort: 30081
          resources: {}
      hostname: schema-registry
      restartPolicy: Always

schema-registry-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    service: schema-registry
  name: schema-registry
spec:
  ports:
    - port: 30081
      name: outport
      targetPort: 30081
      nodePort: 30081
  type: NodePort
  selector:
    service: schema-registry

zookeeper-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    service: zookeeper
  name: zookeeper
spec:
  ports:
    - name: "2181"
      port: 2181
      targetPort: 2181
  selector:
    service: zookeeper

zookeeper-ss.yaml - Again the main difference this time is the usage of Stateful Set.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    service: zookeeper
  name: zookeeper
spec:
  serviceName: zookeeper
  replicas: 1
  selector:
    matchLabels:
      service: zookeeper
  template:
    metadata:
      labels:
        network/kafka-network: "true"
        service: zookeeper
    spec:
      enableServiceLinks: false
      containers:
        - name: zookeeper
          imagePullPolicy: IfNotPresent
          image: confluentinc/cp-zookeeper:7.0.1
          ports:
            - containerPort: 2181
          env:
            - name: ZOOKEEPER_CLIENT_PORT
              value: "2181"
            - name: ZOOKEEPER_DATA_DIR
              value: "/var/lib/zookeeper/data"
            - name: ZOOKEEPER_LOG_DIR
              value: "/var/lib/zookeeper/log"
            - name: ZOOKEEPER_SERVER_ID
              value: "1"
          resources: {}
          volumeMounts:
            - mountPath: /var/lib/zookeeper/data
              name: zookeeper-data
            - mountPath: /var/lib/zookeeper/log
              name: zookeeper-log
      hostname: zookeeper
      restartPolicy: Always
  volumeClaimTemplates:
    - metadata:
        name: zookeeper-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 250Mi
    - metadata:
        name: zookeeper-log
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 250Mi

Running it

1. Open a terminal and cd to the storage-class-setup folder.
2. Create a folder called "tmp", this is where the storage will be automatically provisioned by the default Kind storage class.
3. Run kind specifying configuration: kind create cluster --config=kind-config.yml. This will start a Kind kubernetes control plane + worker.
4. Run kubernetes configuration for kafka kubectl apply -f kafka-k8s
5. When done stop kubernetes objects: kubectl delete -f kafka-k8s and then if you want also stop the kind cluster which will also delete the storage on the host machine: kind delete cluster.

After running the kubectl apply command(step 4 above) check your local tmp folder where you will find the automated storage mapped to your local host disk, notice that those folders will be deleted when you shutdown the Kind cluster but they will persist over pod restarts of Kafka and zookeeper.

Connecting a Kafka client

I decided to add this section because I commonly see developers struggling to connect their Kafka clients from the IDE or local dev machine to a local Kafka running on docker, docker-compose or kubernetes, mostly because the client gives an error that it can't find host kafka when running.

There are many ways to solve that I will explain a few ways here.

1. The simplest way to do it is to add kafka and schema-registry to your /etc/hosts file in the host machine, so when the broker configures LISTENER_EXTERNAL://kafka:9092 which will be advertised to the kafka client the client will resolve kafka to localhost and it will just work. In some configurations you do the same with schema-registry. Some people don't like to do this so there are options.

2. Change the configuration on your compose file or kubernetes kafka config file to LISTENER_EXTERNAL://localhost:9092, so now when in your local client you specify kafka broker address to localhost:9092 it will just work as it will match the advertised address the client received from the broker. The downside of doing this is that if you decide to package your application to run locally in a container inside the docker-compose or kubernetes network the call will fail as localhost will then point to the application container / pod instead of kafka.

3. When using Kubernetes use a tool like Telepresence to expose and route kubernetes ports.

4. When using docker-compose you might leverage the docker internal address mappings as describe nicely in this article.

You can find a detailed explanation on why this happens in this Confluent blog post.

That's it, it's done, you have a functional local Kafka +
Schema Registry running on Kubernetes that you can reach from your application running on your developer machine or IDE.

Photo by Fotis Fotopoulos on Unsplash