An event is the starting point of any ETL pipeline. So then the question becomes, how do we handle the event such that the proper orchestration and handling produce the desired outcome we are looking for?
Let's walk through a "Hello, world!" example of a pub/sub pipeline using some of the tools the Azure platform provides. We will create two Functions using VS Code + the Azure plugin. The first will be a webhook that publishes an event to the Event Grid. The second Function will receive the message from Event Grid and log a response to the logs.
0. Prerequisites
1. Hello World HTTP Azure Function
Install the Visual Studio Code plugin "Azure Functions". Once completed, an Azure icon will appear on your left-hand menu, click on it. Once in Azure Functions menu, click "Sign in to Azure...". After signing in, click the "Create New Project..." icon from the top menu.
Once you choose a location, make the following selections:
-
Select a language for your function project
- Python
-
Select a template for your project's first function
- HTTP trigger
-
Provide a function name
- publish-event
-
Authorization level
- Anonymous
-
Select how you would like to open your project
- Open in current window
-
Enter the alias or full path of the python "3.6x" executable to use
- python3
You should be able to run the function locally pressing F5. Once it is up and running, in a web browser you should be able to navigate to:
http://localhost:7071/api/publish-eventhttp://localhost:7071/api/publish-event?name=dev3l
NOTE: At this point, I had some issues with Python 3.7 where the plugin was looking for 3.6.x. Refer to this Stack Overflow if you run into a similar issue, then recreate the function.
2. Hello World Event Grid Azure Function
From the Azure Functions menu, click the thunderbolt icon from the top menu to create a new function.
Make the following selections:
-
Select a template for your project's first function
- Azure Event Grid trigger
-
Provide a function name
- subscribe-event
We have two functions that run locally. I was trying to find an easy way to test the subscription event locally, but could not find any straightforward solutions. Let's have faith that it is running.
3. Push Up to Azure
From the Azure Functions menu, click the up arrow icon from the top menu.
Make the following selections:
-
Select Function App in Azure
- Create new Function App in Azure...
- pub-sub-function
-
Select a location for new resources.
- US East
4. Test in Azure
Log in to Azure Portal. From "All Resources" click on the newly created "pub-sub-function" resource.
For the webhook, it can be easily hit through its web URL. For my function, it is: https://pub-sub-function.azurewebsites.net/api/publish-event.
Clicking on each shows the code and ability to "Run". Run each and validate that each completes successfully. For the Event Grid function, after running wait a few seconds and view the "Monitor" section to validate the function was called.
My Event Grid function threw an exception about JSON serialization. We will be able to test this functionality better using Postman in a few steps.
5. Create Event Grid Topic
Click the + icon to create a new resource. Search for "Event Grid Topic" and fill out the details like the image below with a name of "pub-sub-grid".
Navigate to the pub-sub-grid resource and be sure to copy the Subscription ID and the Resource group to be used later with Postman.
6. Create Function Event Grid subscription
Head back to the "pub-sub-function" Function resource. From the "subscribe-event" function, click the "Add Event Grid subscription" link.
Be sure to select the newly created "pub-sub-grid" from the Event Grid Topics selection.
7. Validate pub-sub-function is called from pub-sub-grid topic
For some more in-depth detail about calling Event Grid, refer to the bottom of this blog post.
In Azure Portal navigate to the pub-sub-grid resource. From here we will need to retrieve the Event Grid Topic API URL and an Access Key.
Using postman set the method to POST and the URL to the Event Grid Topic API URL, https://pub-sub-grid.eastus-1.eventgrid.azure.net/api/events in my case. Select Headers and add a key aeg-sas-key with a value of your Access Key. In the Body of the post select raw and paste the following:
[
{ "id": "1",
"eventType": "pub-sub",
"subject": "pubsub/event",
"eventTime": "2019-09-28T12:00:00+00:00",
"data": {
"name": "dev3l"
},
"dataVersion": "1.0",
"metadataVersion": "1",
"topic": "/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.EventGrid/topics/pub-sub-grid"
}
]
Find and replace the following:
- subscription_id: Subscription ID copied above from pub-sub-grid
- resource_group: Resource group copied above from pub-sub-grid
Validate that you receive a 200 response and then view the Monitor of the pub-sub-function subscribe-event function.
8. Publish Event in publish-event Function
Back to Visual Studio Code to make some changes to our functions. First, add "requests" to the requirements.txt file. Click the "Terminal" to start a new Terminal session inside your virtual environment. Type pip install requests to get the library installed in your local environment for testing.
Update the contents of publish-event/init.py to be:
import logging
import requests
import azure.functions as func
url = "{event_grid_topic_url}"
payload = """[{
"id": "1",
"eventType": "pub-sub",
"subject": "pubsub/event",
"eventTime": "2019-09-28T12:00:00+00:00",
"data": {
"name": "|NAME|"
},
"dataVersion": "1.0",
"metadataVersion": "1",
"topic": "/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.EventGrid/topics/pub-sub-grid\"
}]"""
headers = { 'aeg-sas-key': "{access_key}" }
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
# PUBLISH EVENT
requests.request("POST", url, data=payload.replace("|NAME|", name), headers=headers)
return func.HttpResponse(f"Hello {name}!")
else:
return func.HttpResponse(
"Please pass a name on the query string or in the request body",
status_code=400
)
Find and replace the following:
- event_grid_topic_url: Event Grid Topic URL for pub-sub-grid
- subscription_id: Subscription ID copied above from pub-sub-grid
- resource_group: Resource group copied above from pub-sub-grid
- acces_key: Event Grid Topic Access Key for pub-sub-grid
Granted we are hardcoding the URL, but this opens the door to being fancy later. Hit F5 to run the project and navigate to the URL http://localhost:7071/api/publish-event?name=pubsub.
By doing so, you should receive a webpage response and the publish event should be sent to the event grid. You can once again validate this by viewing the Monitor logs of the subscribe-event function. Push up the changes and let's next test it works in Azure!
9. Bringing It All Together in Azure
Once you have pushed the updated Azure Functions, navigate back to your web function URL. For me, the address is: https://pub-sub-function.azurewebsites.net/api/publish-event?name=dev3lsolutions. Where name is query string parameter that is displayed on the next page AND sent as a publish event to the pub-sub-grid Resource, which then invokes the subscribe-event Function!
The source code for this tutorial can be found here in GitHub!
Conclusion
Microsoft Azure has allowed us to create a serverless pub/sub architecture using Python. In this example of what can be done, we explored several tools and resource types of the Azure of the ecosystem. Functions can publish events in a variety of ways to be consumed by any number of subscriber Functions. The availability and ability to scale in this cloud solution make it a clear choice of traditional in house message queuing and processing systems used by business ETL processes.
Top comments (0)