Announcing Azure Functions Kafka extension Public Preview

We're excited to announce today the public preview of the Kafka Extension for Azure Functions. With this new extension you can now have functions trigger in response to messages in Kafka Topics, or write to a Kafka Topic through the output binding. The extension is supported when hosting functions in the Premium plan enabling it to elastically scale and trigger on Kafka messages. You can also use Kafka with Azure Functions containers in Kubernetes alongside KEDA

How to get started with the Kafka Trigger

To get started with using the Kafka trigger, you need to include the extension in your function project.

.NET Functions

For .NET functions, you can pull in the Kafka NuGet extension.

dotnet add package Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

JavaScript, TypeScript, Python, Java, and PowerShell Functions

For other functions, you need to install the extension into your project using the Azure Functions core tools

func extensions install Microsoft.Azure.WebJobs.Extensions.Kafka --version 2.0.0-beta

You can then create a function that can activate and run whenever a message is dropped in a Kafka topic.

Example C# Trigger using Confluent Cloud Managed Kafka

public static class kafka_example
    {
        [FunctionName("kafkaApp")]
        public static void ConfluentCloudStringTrigger(
             [KafkaTrigger(
                "BootstrapServer",
                "users",
                ConsumerGroup = "<ConsumerGroup>",
                Protocol = BrokerProtocol.SaslSsl,
                AuthenticationMode = BrokerAuthenticationMode.Plain,
                Username = "<APIKey>",
                Password = "<APISecret>",
                SslCaLocation = "confluent_cloud_cacert.pem")]
        KafkaEventData<string> kafkaEvent,
        ILogger logger)
        {       
            logger.LogInformation(kafkaEvent.Value.ToString());
        }
    }

Example host.json

{
  "version": "2.0",
  "extensions": {
    "kafka": {
      "maxBatchSize": 100
    }
  }
}

You can find more settings documented here

Example functions.json

Note: This is auto-generated for C#

{

  "bindings": [
    {
      "type": "kafkaTrigger",
      "consumerGroup": "azfunc",
      "protocol": "saslSsl",
      "authenticationMode": "plain",
      "username": "<KafkaAPIKey>",
      "password": "<KakfaSecret>",
      "sslCaLocation": "<RootCertificateFile>",
      "topic": "<KafkaTopicName>",
      "brokerList": "<Server>.eastus.azure.confluent.cloud:9092",
      "name": "kafkaEvent"
    }
  ],

}

Please find a complete end to end walkthrough and sample app using the Confluent Cloud in this GitHub repo Kafka extension sample with Confluent Cloud.

This extension is being developed in the open, please contribute, try out and post any issues on the Azure Functions Kafka extension GitHub repo.

Along with the Rabbit MQ Trigger that we announced last year that is now also supported in the Premium Plan, we continue to innovate to bring in more Cloud Native and Open Source event sources. We are looking forward to see what applications you build with this !