Make a Dashboard for Your Twitter Stats With Python And Anvil

Welcome to the last part of our tutorial on building your own Twitter Analytics dashboard with Python, Azure Functions and Anvil.

A Quick Recap

Let us summarise what we have done so far.

1. In the first part of our series, we got to know the Twitter API and learned how to retrieve Tweet stats and Follower/Unfollower data.

2. In the second part, we used Azure Functions to periodically download and store these data to Azure Blobs

3. In the previous blog post, we used Azure Blog Storage triggers to compare our point-in-time snapshots to their predecessors automatically. Thus, we compiled an aggregated result of our Twitter Followers over time and a consolidated data structure for our Tweet stats.

What we will learn today

There is little value in any statistic unless it can be made visible. We still need a way to plot graphs of our Twitter performance over time. That's why we will build a simple dashboard today. To make this article short and easy to follow, we will just focus on new and lost follower data over time. However, you will see how easy adding more graphs and insights into this dashboard will be.

How it will work

To make a modern dashboard, we will build a web-based application. This means that you will be able to access your dashboard from within your browser. You have seen these kinds of applications quite frequently. Twitter, Gmail, Google Analytics,… – all of those run in your browser interactively.

Web-based applications consist of multiple layers: One part of the dashboard runs in the browser. Browsers can understand HTML, CSS and JavaScript. This is just intended to control how the dashboard will look and feel. Then, there is a server-side responsible for all the heavy lifting like data gathering, preparation, authorisation, etc.

The good news is that we will be able to create our dashboard in pure Python. No HTML, no CSS, no JavaScript. We will use a tool called Anvil which will bridge our Python code to modern web technologies. To put it in simple terms, Anvil will translate our Python code so that the browser can understand it.

Getting started with Anvil

You have two options to start programming with Anvil:

1. You can download the open-source version and run everything on your own machine, or
2. You can sign up for an Anvil account and use their service for your development.

I have outlined both variants in the Getting Started with Anvil Guide

Part I: Creating A Dashboard with Anvil's Open Source version

Create a hello-world app

Before we start, we create a skeletton app with Anvil. To do this, we create a blank Anvil app to start with

create-anvil-app hello-world MyTwitterDashboard

This will create a new Anvil app in a directory called MyTwitterDashboard. Of course, you can name it whatever you want.

We can check whether the creation of the app has been successful and whether anvil starts up correctly by using the anvil-app-server command:

anvil-app-server --app MyTwitterDashboard

If you now head over to you browser at localhost:3030 and see an empty page, everything worked fine.

Code for the Server Side

Now, we create a server side function in anvil which is responsible for connecting to Azure and download our data. Head over to the ServerModule1.py file in the server_code directory. Here we can re-use the get_blob_service_client function which we coded in the previous part of this series and expose two functions, get_follower_data and get_follower_data to Anvil's server.

The decorator @anvil.server.callable will instruct Anvil to expose this function over HTTP, so that we can call it from the front end.

Note that we rely on Anvil's secret storage. In this storage we place a variable called AZURE_CONNECTION_STRING to access our Azure Connection String. This way you don't have to hard-code your sensitive information into the code.

Here is the full source code of ServerModule1.py:

import json
import os
import gzip
import anvil.server
import anvil.secrets
from azure.storage.blob import BlobServiceClient


def get_blob_service_client():
    AZURE_CONNECTION_STRING = anvil.secrets.get_secret("AZURE_CONNECTION_STRING")
    blob_service_client = BlobServiceClient.from_connection_string(AZURE_CONNECTION_STRING)
    return blob_service_client


def get_data_from_azure(blob_service_client, blob_name, container_name="output", default=None):
    try:
        blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
        data = blob_client.download_blob().readall()
        uncompressed = gzip.decompress(data)
        return json.loads(uncompressed)
    except:
        return default

@anvil.server.callable
def get_follower_data():
    data = get_data_from_azure(get_blob_service_client(), "NEW_FOLLOWERS.json.gz")
    return [dict(dt=k, n=len(v)) for k,v in data.items()]

@anvil.server.callable
def get_unfollower_data():
    data = get_data_from_azure(get_blob_service_client(), "LOST_FOLLOWERS.json.gz")
    return [dict(dt=k, n=len(v)) for k,v in data.items()]

Create the Front End

The front end is the visible part of your Anvil application, i.e. the part that will appear in your browser. To start, we just add two Plots, one for our followers and one for our unfollowers over time to our Form. Open the _template.py file in the client_code/Form directory and add two plot components to the end of the init_components method like so:

def init_components(self, **properties):
    ...
    self.followers_plot = Plot()
    self.add_component(self.followers_plot)
    self.unfollowers_plot = Plot()
    self.add_component(self.unfollowers_plot)

The full Form/_template.py would look like this:

from anvil import *

class Form1Template(HtmlPanel):

  def init_components(self, **properties):
    # Initialise HtmlPanel
    super().__init__() 
    # Set the html template for the app
    self.html = '@theme:standard-page.html'

    # Add a GridPanel to the Form
    self.content_panel = GridPanel()
    self.add_component(self.content_panel)

    # Add a FlowPanel to accept NavBar links
    self.nav_links = FlowPanel()
    self.add_component(self.nav_links, slot="nav-right")

    # Add a title to the app
    self.title_label = Label(text="Bas.codes – Twitter Analytics")
    self.add_component(self.title_label, slot="title")

    # Add a sidebar to the app. Comment out the following two rows if you don't want a sidebar in your app.
    self.left_nav = ColumnPanel()
    self.add_component(self.left_nav, slot="left-nav")

    self.followers_plot = Plot()
    self.add_component(self.followers_plot)
    self.unfollowers_plot = Plot()
    self.add_component(self.unfollowers_plot)

Nothing interesting has happened so far. We just created and initialized to member variables, self.followers_plot and self.unfollowers_plot and set both of them to an empty plot.

Now, it's time to populate these plots with data. Luckily, these plots are not specific to Anvil, but are good old Plotly graphs.
So, we can simply use the go functions from the plotly.graph_objects module.

We can build the followers graph very simple like so:

def build_followers_graph(self):
  followers = anvil.server.call('get_follower_data')
  scatter = go.Scatter(x = [x['dt'] for x in followers],
                        y = [x['n'] for x in followers],
                        fill = 'tozeroy',
                        line=dict(color='#2196f3'))
  self.followers_plot.data = scatter
  self.followers_plot.layout.title = "FOLLOWERS"

And here, something interesting happens: The first line (followers = anvil.server.call('get_follower_data')) will instruct Anvil to call the server side function get_follower_data we just created over the network. This way, your browser does not directly talk to Azure, but to your Anvil server which in turn fetches the follower data from Azure. This is important as you might want to share your Anvil app but not the logic behind it. You can thus easily hide the storage part in the server.

The next lines are pretty straight forward: A scatter plot is created with the data we got from Azure and this scatter plot is assigned to the still empty Plot object of our _template.py file.

Now, we just have to ensure that our graphs are build when the application is started. We do this by calling our build_followers_graph method in the constructor of our Form.

For the unfollowers, we repeat the steps above.

The whole code of the Form/__init__.py file now looks like this:

from _template import Form1Template
from anvil import *
import anvil.server
import plotly.graph_objects as go
from datetime import datetime

class Form1(Form1Template):

  def __init__(self, **properties):
    # Set Form properties and Data Bindings.
    self.init_components(**properties)

    # Any code you write here will run when the form opens.
    self.build_followers_graph()
    self.build_unfollowers_graph()


  def build_followers_graph(self):
    followers = anvil.server.call('get_follower_data')
    scatter = go.Scatter(x = [x['dt'] for x in followers],
                         y = [x['n'] for x in followers],
                         fill = 'tozeroy',
                         line=dict(color='#2196f3'))
    self.followers_plot.data = scatter
    self.followers_plot.layout.title = "FOLLOWERS"

  def build_unfollowers_graph(self):
    unfollowers = anvil.server.call('get_unfollower_data')
    scatter = go.Scatter(x = [x['dt'] for x in unfollowers],
                         y = [x['n'] for x in unfollowers],
                         fill = 'tozeroy',
                         line=dict(color='#2196f3'))
    self.unfollowers_plot.data = scatter
    self.unfollowers_plot.layout.title = "UNFOLLOWERS"

If you now run

anvil-app-server --app MyTwitterDashboard --secret AZURE_CONNECTION_STRING=<Your Azure Connection String>

again, refresh your browser at localhost:3030, you will see your follower/unfollower dashboard which should look like this:

Note that you have to export the AZURE_CONNECTION_STRING environment variable, so that Anvil can pick it up and use it in your server code.

Part II: Using the Anvil Hosted Service

Let's recap what we have done so far. With only very few lines of code we created a full-fledged web application with a responsive layout (yes, your application will work on your smartphone, too!). On the way, we connected to Azure, processed the results and drew graphs. As if this would not be impressive enough, we can even simplify the whole process even further.

By using the hosted Anvil.works environment, we can interactively design our app. Let's do that!

If you haven't already, sign up at anvil.works and log into your account.

First, click on New Blank app and choose Material Design

Next, you should see your Anvil Development Environment which should look like this:

From the Toolbox Menu on the right, drag and drop the Plot component right into the application canvas in the middle

Make sure, that you re-name self.plot_1 and self.plot_2 to followers_plot and unfollowers_plot respectively.

Now, head over to the Server Code section in the left toolbox. Click Add new Server Module and paste the server code of the previous section here:

It should look like this:

The last step is to connect our Plot components with our server code just the same way we did it in the Open Source edition of the previous section.

Click on Form1 in the left menu. Switch to the Code view in the middle of your editor. Then, you can paste in the missing parts of your code like so:

Now, there is only one thing left: Our server code does not know about your AZURE_CONNECTION_STRING. In Anvil you do have Application Secrets instead of environment variables which are specifically designed to store access secrets like our connection string.

You can create an App secret by clicking the + Sign on the SERVICES headline in the left toolbox. Now, click on App Secret and create the AZURE_CONNECTION_STRING secret.

Inside your server code, you have to add this line to the top:

import anvil.secrets

As a last step, change the line

AZURE_CONNECTION_STRING = os.getenv('AZURE_CONNECTION_STRING')

in your server code to this line:

AZURE_CONNECTION_STRING = anvil.secrets.get_secret("AZURE_CONNECTION_STRING")

If you now hit the Run button, you should see the exact same app - but this time it's hosted on Anvil.works.

Limitations of the free edition

Running other peoples' Python code is a very challenging task to do. Anvil has to work out security and performance considerations to provide a secure and responsive platform for everyone. Unfortunately, this means that the free plan of hosted Anvil is restricted to avoid overloading of the platform.

In our sample code we use the azure module as well as the gzip module. These modules are not available in the free plan. Luckily, Anvil.works provides a free trial of their professional version. Also, the Open Source version is unrestricted. Should you decide to proceed with your Twitter Dashboard project, you will can always use the Open Source version on your own machine for free.

Conclusion and Outlook

We have built Azure Functions to Download Data from our Twitter Account, have them processed on the Azure Platform and stored into Azure Blob Storage. Then, we went on and built a frontend for it in Anvil.

All of this happened with nothing else than Python and it did not require us to buy or rent a server. Of course, all the components we discussed are way more powerful than our small example app might make you think. Of course, you can think of extending your Twitter stats dashboard with more plotly graphs or other frontend elements provided by Anvil.

I hope you've enjoyed building a Twitter Follower and Unfollower Dashboard with me! Let me know how this did work out for you and how you want to extend this app. If you like stuff like this, make sure to follow me on Twitter so that you don't miss future projects we can build together!