Learn how to use Azure Event Hubs with the Dapr framework

This blog uses an example to walk you through how to use Azure Event Hubs integration with Dapr, a set of distributed system building blocks for microservices development.

Azure Event Hubs will be used as a "binding" within the Dapr runtime. This will allow services to communicate with Azure Event Hubs without actually knowing about it or being coupled to it directly (via SDK, library etc.), using a simple model defined by the Dapr runtime.

Sounds too good to be true? 😉 Read on...

You will:

• Learn the basics of Dapr and how to set it up for local development
• Deploy your first Dapr app(s)
• Setup Event Hubs on Azure and try out an example to see how to communicate with it transparently using Dapr integration

Azure Event Hubs is a fully managed Platform-as-a-Service (PaaS) for streaming and event ingestion. It also provides Apache Kafka support enabling clients and applications to talk to Event Hubs without need to set up, configure, and manage your own Kafka clusters!

Dapr? What's that?

Dapr stands for Distributed Application Runtime. You can get all the scoop in this announcement blog, but here is a (buzzword compliant!) gist to get you started.

It is an open source, portable runtime to help developers build resilient, microservice stateless and stateful applications. It does so by codifying the best practices for building such applications into independent components. The capabilities exposed by the runtime components can be accessed over HTTP or gRPC, making it completely agnostic and allows it to work with any language and/or framework.

Dapr: modus operandi

Dapr adopts a sidecar architecture and runs either as a separate container or as a process. This means that the application itself does not need to include Dapr runtime as a dependency. This allows for easy integration as well as providing separation of concerns.

Dapr also includes language specific SDKs for Go, Java, JavaScript, .NET and Python. These SDKs expose the functionality in the Dapr building blocks, such as saving state, publishing an event or creating an actor, through a typed, language API rather than calling the http/gRPC API.

Dapr is platform agnostic. You can run your applications locally, on any Kubernetes cluster and other hosting environments that Dapr integrates with.

Dapr components

At the time of writing, Dapr is in alpha state and supports the following distributed systems building blocks which you can plug into your applications - Service invocation, State Management, Pub/Sub messaging, Resource Bindings, Distributed tracing and Actor pattern

For a comprehensive overview, please refer to the Dapr documentation

As mentioned earlier, the example application in this post makes use of the Dapr bindings for Azure Event Hubs. Here is a peek at what "bindings" are.

Dapr Resource Bindings

Bindings provide a common way to trigger an application with events from external systems or invoke an external system with optional data payloads. These "external systems" could be anything: a queue, messaging pipeline, cloud-service, filesystem, etc.

Currently supported bindings include Kafka, Rabbit MQ, Azure Event Hubs etc.

In a nutshell, Dapr bindings allow you to focus on business logic rather than integrating with individual services such as databases, pub/sub systems, blob storage etc.

• you can leverage off the shelf bindings to integrate with different systems without depending on or including specific SDKs or libraries
• you can also swap/replace bindings without changing your code

Alright, enough theory!

Pre-requisites

You can run Dapr anywhere, including Kubernetes, thanks to its first class Operator based support. Since this is an introductory blog post, let's focus on the end to end flow, keep things simple and run Dapr locally as a standalone component.

If you're itching to run Dapr on Kubernetes, check out this getting started guide!

For the sample app, you will need:

• Docker
• Go 1.12 or above
• Azure Event Hubs

Setup Dapr

Start by installing the Dapr CLI which allows you to setup Dapr on your local dev machine or on a Kubernetes cluster, provides debugging support, launches and manages Dapr instances.

For e.g. on your Mac, you can simply use this to install Dapr to /usr/local/bin

curl -fsSL https://raw.githubusercontent.com/dapr/cli/master/install/install.sh | /bin/bash

Refer to the docs for details

You can use the CLI to install Dapr in standalone mode. All you need is a single command

dapr init

.. and that's it!

Setup Azure Event Hubs

If you don't already have a Microsoft Azure account, go ahead and sign up for a free one! Once you're done you can quickly set up Azure Event Hubs using either of the following quickstarts:

• Azure portal - here is a step-by-step guide
• Azure CLI or Azure Cloud shell (in your browser!) - here is a step-by-step guide

You should now have an Event Hub instance with a namespace and associated Event Hub (topic). As a final step you need to get the connection string in order to authenticate to Event Hubs - use this guide to finish this step.

Receive Event Hubs data using Input Bindings

An Input Binding in Dapr represents an event resource that Dapr uses to read events from and push to your application. Let's run an app that uses this to receive data from Azure Event Hubs.

Run the app with Dapr

Start by cloning the repo and change into the correct directory

git clone https://github.com/abhirockzz/dapr-eventhubs-bindings
cd input-binding-app

Update components/eventhubs_binding.yaml to include Azure Event Hubs connection string in the spec.metadata.value section. Please note that you will have to append the name of the Event Hub to end the connection string i.e. ;EntityPath=<EVENT_HUBS_NAME>. This is what the value for connectionString attribute should look like:

Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<EVENT_HUBS_KEY>;EntityPath=<EVENT_HUBS_NAME>

Start the Go app which uses the Azure Event Hubs Input Bindings

export APP_PORT=8080
dapr run --app-port $APP_PORT go run app.go

You should see the logs

== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="starting Dapr Runtime -- version 0.1.0 -- commit 4358565-dirty"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="log level set to: info"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="standalone mode configured"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="dapr id: foobar"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="loaded component messagebus (pubsub.redis)"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="loaded component timebound (bindings.azure.eventhubs)"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="loaded component statestore (state.redis)"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="application protocol: http. waiting on port 8080"
== DAPR == time="2019-10-29T06:32:07+05:30" level=info msg="application discovered on port 8080"
.....

Run Azure Event Hubs producer application

This app uses the Azure Event Hubs native Go client to send messages.

Set the required environment variables:

export EVENT_HUBS_NAMESPACE="<EVENT_HUBS_NAMESPACE>"
export EVENT_HUBS_KEY="<EVENT_HUBS_KEY>"
export EVENT_HUB_NAME="<EVENT_HUB_NAME>"

Please ensure that the name of the Event Hub is the same as what you configured for the connection string in the input binding configuration

Run the producer app - it will send five messages to Event Hubs and exit

cd eventhubs-producer
go run producer.go

Confirm

Check Dapr application logs, you should see the messages received from Event Hubs.

== APP == time from Event Hubs 'Thu Oct 31 16:57:45 2019'
== APP == time from Event Hubs 'Thu Oct 31 16:57:49 2019'
== APP == time from Event Hubs 'Thu Oct 31 16:57:52 2019'
== APP == time from Event Hubs 'Thu Oct 31 16:57:54 2019'
== APP == time from Event Hubs 'Thu Oct 31 16:57:56 2019'

Behind the scenes

Here is a summary of how it works:

Input Binding

The eventhub_binding.yaml config file captures the connection string for Azure Event Hubs.

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: eventhubs-input
spec:
  type: bindings.azure.eventhubs
  metadata:
  - name: connectionString
    value: Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<KEY>;EntityPath=<EVENT_HUBS_NAME>

The key attributes are:

• metadata.name - name of the Input binding component
• spec.metadata.name - Event Hubs connection string

Notice that the connection string contains the information for the broker URL (<EVENT_HUBS_NAMESPACE>.servicebus.windows.net), primary key (for authentication) and also the name of the topic or Event Hub to which your app will be bound and receive events from.

Using the binding in the app

The Go app exposes a REST endpoint at /eventhubs-input - this is the same as the name of the Input Binding component (not a coincidence!)

func main() {
    http.HandleFunc("/eventhubs-input", func(rw http.ResponseWriter, req *http.Request) {
        var _time TheTime
        err := json.NewDecoder(req.Body).Decode(&_time)
        if err != nil {
            fmt.Println("error reading message from event hub binding", err)
            rw.WriteHeader(500)
            return
        }
        fmt.Printf("time from Event Hubs '%s'\n", _time.Time)
        rw.WriteHeader(200)
    })
    http.ListenAndServe(":"+port, nil)
}

Dapr runtime does the heavy lifting of consuming from Event Hubs and making sure that it invokes the Go application with a POST request at the /eventhubs-input endpoint with the event payload. The app logic is then executed, which in this case is simply logging to standard output.

Send data to Event Hubs data with Output Bindings

An output binding represents a resource that Dapr will use invoke and send messages to. Let's use an output binding to send data to Event Hubs.

Run the app with Dapr

Change to the correct directory

cd output-binding-app

Update components/eventhubs_binding.yaml to include Azure Event Hubs connection string in the spec.metadata.value section. Please note that you will have to append the name of the Event Hub to end the connection string i.e. ;EntityPath=<EVENT_HUBS_NAME>. This is what the value for connectionString attribute should look like:

Endpoint=sb://<EVENT_HUBS_NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<EVENT_HUBS_KEY>;EntityPath=<EVENT_HUBS_NAME>

Start the Go app which uses the Azure Event Hubs Output Bindings. It will send five messages to the Dapr binding HTTP endpoint in quick succession and exit.

dapr run --port 3500 go run app.go

In the app, you will see logs:

== APP == sent message {"data": {"time":"2019-10-31 17:18:07.602377 +0530 IST m=+0.000569729"}}
== APP == response 200 OK
== APP == sent message {"data": {"time":"2019-10-31 17:18:12.375237 +0530 IST m=+4.773393815"}}
== APP == response 200 OK
== APP == sent message {"data": {"time":"2019-10-31 17:18:14.504914 +0530 IST m=+6.903055209"}}
== APP == response 200 OK
== APP == sent message {"data": {"time":"2019-10-31 17:18:16.637992 +0530 IST m=+9.036116365"}}
== APP == response 200 OK
== APP == sent message {"data": {"time":"2019-10-31 17:18:18.763691 +0530 IST m=+11.161800122"}}
== APP == response 200 OK
== APP == finished sending messages... use ctrl+c to exit

This means that the messages were sent to Azure Event Hub via the Dapr output binding

In the receiver (input bindings) app, you should see

== APP == time from Event Hubs '2019-10-31 17:18:07.602377 +0530 IST m=+0.000569729'
== APP == time from Event Hubs '2019-10-31 17:18:12.375237 +0530 IST m=+4.773393815'
== APP == time from Event Hubs '2019-10-31 17:18:14.504914 +0530 IST m=+6.903055209'
== APP == time from Event Hubs '2019-10-31 17:18:16.637992 +0530 IST m=+9.036116365'
== APP == time from Event Hubs '2019-10-31 17:18:18.763691 +0530 IST m=+11.161800122'

To keep things simple, we used the same Event Hub topic for input and output binding example. In reality these can be used independently i.e. an application which needs to be triggered by Event Hubs in an event-driven manner can subscribe to a topic (via Dapr) and another app which needs to push data to Event Hubs (to a different topic) can use the Dapr binding HTTP endpoint to POST event data.

Behind the scenes

The output binding configuration remains the same as Input binding.

The Go app sends a message to Event Hubs via the output binding. It does so by sending a POST request to the Dapr HTTP endpoint http://localhost:<dapr_port>/v1.0/bindings/<output_binding_name>.

....
const daprURL = "http://localhost:3500/v1.0/bindings/eventhubs-output"
go func() {
    ....
    for i := 1; i <= 5; i++ {
        body := fmt.Sprintf(format, time.Now().String())
        resp, err := http.Post(daprURL, contentType, strings.NewReader(body))
        .....
        fmt.Println("sent message", body)
        fmt.Println("response", resp.Status)
        time.Sleep(2 * time.Second)
    }
    done <- true
}()
....

In this example, the name of the output binding is eventhubs-output and the Dapr HTTP port is 3500, the endpoint URL is:

http://localhost:3500/v1.0/bindings/eventhubs-output

As was the case with Input Binding, the Dapr runtime takes care of sending the event to Event Hubs. Since we have the Input binding app running, it receives the payload and it shows up in the logs.

Summary

In this blog post, you saw how to use Dapr "bindings" to connect integrate your applications via Azure Event Hubs without having to know about it! Instead, you connected through the Dapr runtime (just a sidecar) using the Dapr HTTP API.

It is also possible to do it using gRPC or language specific SDKs (as previously mentioned)

As the time of writing, Dapr is in alpha state (v0.1.0) and gladly accepting community contributions 😃 Vist https://github.com/dapr/dapr to dive in!

If you found this article helpful, please like and follow 🙌 Happy to get feedback via Twitter or just drop a comment.