Recently I had to work with Spring Cloud Data Flow, which is a microservice streaming and batch platform based on Cloud Foundry and Kubernetes. As I am using it with Kubernetes, I will not talk about Cloud Foundry.
I am using Azure Kubernetes Service, and I would like to use Azure Event Hubs as Spring Cloud Data Flow Binder.
We (me and my teammates) spent a lot of time reading documentation (Spring Cloud Data Flow configuration and also Spring Cloud Stream configurations ) and we've found that we should define binders configuration when deploying the stream, and not at a server level.
Imagine the following stream:
stream create --name 'uppercase' --definition 'http | transform --expression=payload.toUpperCase() | log'
This stream will get some data from an http endpoint, then the data will be transformed to uppercase then logged, an easy stream, good for make some tests.
Note: Each application (http/transform/log) need to have a dependency to the specific binder (and only one), it means that if you want to use the same application with different binders you will have to build different versions.
When deploying the stream, you can pass some configuration properties to each application :
stream deploy uppercase --properties "deployer.http.kubernetes.createNodePort=true"
This for example will tell the "deployer" (ie: Skipper) to create a NodePort service pointing to the http application in Kubernetes (so it will be possible to send a message from outside the cluster)
We want that when the application is created by Spring Cloud Data Flow, it passes some properties directly to the applications (and all applications), for example, we do not want to define the credential information for each application when deploying the stream, it may be injected by Spring Cloud Data Flow itself.
In order to do that we will have to edit the ConfigMap used for the Dataflow Server, to define a set of binders (with all needed configuration), and then it will be possible to reference those binders when deploying our stream.
Here is the new ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: scdf-server
labels:
app: scdf-server
data:
application.yaml: |-
spring:
cloud:
dataflow:
applicationProperties:
stream:
spring:
cloud:
azure:
eventhub:
# this is the "default" binders as it is needed by
# The Azure Event Hub Binder
connection-string: BINDER_ENDPOINT_CONNECTION_STRING
checkpoint-storage-account: BINDER_STORAGE_ACCOUNT
checkpoint-access-key: BINDER_STORAGE_ACCOUNT_ACCESS_KEY
checkpoint-container: BINDER_STORAGE_ACCOUNT_CONTAINER
stream:
# create other binders if needed
binders:
binder-one:
type: eventhub
defaultCandidate: false
environment:
spring:
cloud:
azure:
eventhub:
connection-string: BINDER_ONE_ENDPOINT_CONNECTION_STRING
checkpoint-storage-account: BINDER_ONE_STORAGE_ACCOUNT
checkpoint-access-key: BINDER_ONE_STORAGE_ACCOUNT_ACCESS_KEY
checkpoint-container: BINDER_ONE_STORAGE_ACCOUNT_CONTAINER
binder-two:
type: eventhub
defaultCandidate: false
environment:
spring:
cloud:
azure:
eventhub:
connection-string: BINDER_TWO_ENDPOINT_CONNECTION_STRING
checkpoint-storage-account: BINDER_TWO_STORAGE_ACCOUNT
checkpoint-access-key: BINDER_TWO_STORAGE_ACCOUNT_ACCESS_KEY
checkpoint-container: BINDER_TWO_STORAGE_ACCOUNT_CONTAINER
datasource:
url: jdbc:sqlserver:
username: your_username
password: your_password
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
testOnBorrow: true
validationQuery: "SELECT 1"
Then now when deploying the stream you can use the following command :
stream deploy uppercase --properties "deployer.http.kubernetes.createNodePort=true,app.http.spring.cloud.stream.bindings.output.binder=binder-one,app.transform.spring.cloud.stream.bindings.input.binder=binder-one,app.transform.spring.cloud.stream.bindings.output.binder=binder-two,app.log.spring.cloud.stream.bindings.input.binder=binder-two"
This is it!
I hope this post helped you.
Note, I originally posted this article on medium : https://medium.com/@rgoyard/how-to-run-spring-cloud-data-flow-with-azure-event-hub-as-a-binder-f5e888d0719
Top comments (1)
Hi, I tried your solution but I have an error on the stream :
I'm using bitnami's helm chart, so I don't know how to add dependencies on the server.