DEV Community

Cover image for Streaming Azure Data Pipelines into real-time dashboards
Denise Schlesinger
Denise Schlesinger

Posted on • Updated on

Streaming Azure Data Pipelines into real-time dashboards

Sometimes organizations are required to respond to events in real-time, allowing them to detect and respond to changes such as security threats, customer-impacting operational issues, business KPIs, system monitoring, etc. Using real-time dashboards can help them make data-driven decisions fast.

What is a real-time dashboard?
A real-time dashboard is a type of visualization that is automatically updated with the most current data available.

Which data sources can be used for real-time dashboards?
The sources of streaming data can be IOT sensors, social media posts, monitoring metrics, or any other time-sensitive data being collected.

Which technology can process streaming events?
There are many technologies that allow producing and consuming streams of events, such as Apache Kafka, Spark and others.
We will be using Azure technologies that will help us hit the road running, in no-time.

In this post I will show you how to create synthetic data, stream it using Azure Events hub, process it with Azure Stream Analytics and showcase it in Power BI in a live dashboard using "streaming datasets".

Let's start

What are we building?

We will:

  • Run a python script creating synthetic events and sending them to Azure Events hub
  • Use Azure Stream Analytics to consume the events from Events hub and sending them to Power BI streaming dataset
  • Visualize in Power BI the events in real time with a dashboard.

Here is the architecture
Image description

The recipe
I assume you already have an azure account (click here if you don't)

Create an events hub
In the azure portal create an Events Hub
Image description

Click on Create
Fill all the relevant fields
Click "review and create"
Click "create"
Image description

Go to your newly created events hub
Create a Shared Access Policy

Image description

Image description

Copy the Shared Access Policy primary key and the events hub name and paste it into the python script you downloaded from (also see below) github

import datetime
import time
import random
import socket
import struct
import json
from azure.eventhub import EventHubProducerClient, EventData
import psutil
import pyodbc
import pandas as pd

eventHubConnString = "<paste here the events hub shared access signature primary key>"
eventHubName = "<paste here the events hub name>"
producer = EventHubProducerClient.from_connection_string(conn_str=eventHubConnString, eventhub_name=eventHubName)
hostname = socket.gethostname()

def generateRandomIP():
    ip = socket.inet_ntoa(struct.pack('>I', random.randint(1, 0xffffffff)))
    return ip

def getRandomBrowser():
    browsersList = ["Edge", "Chrome", "Safari", "Firefox"]
    browser = random.choice(browsersList)
    return browser

def getRandomMethod():
    methods = [ "GET", "POST", "PUT", "DELETE" ]
    method = random.choice(methods)
    return method

def getRandomURL():
    urls = ["/page1.html", "/page2.html", "/page3.html", "/page4.html", "/page5.html" ,"/page6.html"  ]
    url = random.choice(urls)
    return url

def getCurrentDate():
    today = datetime.datetime.now()
    current = today.strftime("%Y-%m-%d")
    return current

def getCurrentTime():
    today = datetime.datetime.now()
    current = today.strftime("%H:%M:%S")
    return current

def getRandomTemperature():
    return random.randint(0,99)

def generateClickEvent():
    click_event = {
        "tableName" : "clicks",
        "date" : getCurrentDate(),
        "time": getCurrentTime(),
        "ip": generateRandomIP(),
        "browser": getRandomBrowser(),
        "method": getRandomMethod(),
        "url" : getRandomURL(),
        "productID": getRandomProductId()
    }
    return click_event

def getRandomProductId():
    return str(random.randint(0,99999))

def sendToEventsHub(jsonEvent):
    eventString = json.dumps(jsonEvent)
    print(eventString) 

    event_data_batch = producer.create_batch() 
    event_data_batch.add(EventData(eventString)) 
    producer.send_batch(event_data_batch)

def generateClicks():
    try:
        while True:
            clickEvent = generateClickEvent()    
            sendToEventsHub(clickEvent)
            # time.sleep(10)

    except KeyboardInterrupt:
        producer.close()
        conn.close()


if __name__ == "__main__":
    generateClicks()
Enter fullscreen mode Exit fullscreen mode

Create a stream analytics job
Image description

Image description

Fill the details and click review and create and then Click create
Click on Inputs - Add stream input
Image description

Choose your subscription, the events hub we created above, and settings as shown below
Image description

Click Outputs and add a Power BI output
Image description

Fill all the details as shown below
Please notice that this will authorize the stream analytics job to access your power BI account (I recommend for demo purposes to use the same account as the one logged in into the Azure Portal.
Image description

Click on Query
Copy and paste the query snippet below
Image description

SELECT 
    DateAdd(s,-3,System.TimeStamp) AS windowstart,
    System.TimeStamp AS windowend,
    browser,
COUNT(*) AS entries
INTO
[powerbi]
FROM
[bddemo-clicks] TIMESTAMP BY EventProcessedUtcTime
GROUP BY browser, TumblingWindow(s, 3)
Enter fullscreen mode Exit fullscreen mode

Save the query
Create a real time dashboard in Power BI
Login to power BI with the same credentials you provided the stream analytics job above
Create a dashboard
Image description
Add a tile to your dashboard
Image description

Click on Custom Streaming Data
Image description

Choose the stream we created in the stream analytics job
Image description

Choose the visualization type
We will choose the Line chart and fill the details
Image description

Choose a tile name and click Apply

We are done preparing everything!

Let's run our demo

Run the python script provided
Make sure you are authenticated to run your local code and connect to Azure (if not click here and install powershell)

Perform Azure authentication

$ az login 
Enter fullscreen mode Exit fullscreen mode

Run the python script

python streamingClickEvents.py
Enter fullscreen mode Exit fullscreen mode

Now our code is running and sending events to the events hub
Image description

Start the stream analytics job
Image description

After 1-2 minutes the job status will be "running" and the events will start flowing
Image description

In Power BI you will see the events flowing in your dashboard
Image description

MAKE SURE to shut down your script and stop the stream analytics job to avoid unnecessary costs.

That's all folks!

I hope you enjoyed this tutorial.
Cheers!
Denise

Photo by Jachan DeVol on Unsplash

Top comments (0)