Build a website monitor with docker + flask + twilio + vue + mongodb (part 1)

I have been eager to learn docker for quite some time and finally got some quality (self-isolation) time to try it out. Turns out it is really as good as the internet says, and let's jump into it!

I have recently built a website but sometimes it may go offline due to a wrong deployment build or whatever reason. There are some online services that claims to notify you when your site is down, but I don't know how often they check your site etc.. So let's build our own website monitor!

Step 1: Python whatsapp messaging service by twilio

• The basic idea is simple, we query our website, check the response, if the response is not as expected, we notify ourselves via sms or email (the traditional way) or via whatsapp (the better way).

• Twilio offer free whatsapp messaging service, and who still uses SMS in 2020?? Sign up an account here: https://www.twilio.com/whatsapp and this video explained everything you need https://www.youtube.com/watch?v=98OewpG8-yw

• Basically you need 1. twilio credentials, 2. link your whatsapp number to the twilio sender and 3. install the twilio module for python. 
  To do it more properly, let's also use virtual environment for python.
  

mkdir backend
pip install pipenv
pipenv shell
pipenv install twilio

• You also need to set up few environment variables before it can work properly (and this is better for our docker app later than hardcode). Credentials can be found here: https://www.comm100.com/livechat/knowledgebase/where-do-i-find-the-twilio-account-sid-auth-token-and-phone-number-sid.html. ADMIN_PHONE_NUMBER is your phone number.

export TWILIO_ACCOUNT_SID='abcd'
export TWILIO_AUTH_TOKEN='1234'
export ADMIN_PHONE_NUMBER='+441234'

• Alright, the whatsapp messaging service can be configured like this:

# backend/TwilioWhatsapp.py

import os
from twilio.rest import Client

# check document here if needed
# https://www.twilio.com/docs/libraries/reference/twilio-python/index.html
def send_whatsapp_message(to_whatsapp_number, message_body):
    client = Client() # without specifying credentials, it uses environment variables

    from_whatsapp_number = "whatsapp:+14155238886"

    to_whatsapp_number="whatsapp:" + to_whatsapp_number

    client.messages.create(body=message_body, from_=from_whatsapp_number, to=to_whatsapp_number)

if __name__ == "main":
    send_whatsapp_message(os.environ['ADMIN_PHONE_NUMBER'], "hello")

1. We can test it by python TwilioWhatsapp.py, you should get a whatsapp message if you have set up the whatsapp sandbox properly.

Step 2: check whether our site is live using requests + beautifulsoup

• install beautifulsoup4 and lxml library
  pipenv install beautifulsoup4 lmxl

• Our python code would be
  

# backend/URLChecker.py

import requests
from bs4 import BeautifulSoup

chrome_headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36'}

def find_text_from_URL(URL, text_to_find):
    response = requests.get(URL, headers=chrome_headers)
    soup = BeautifulSoup(response.content, "lxml")

    if text_to_find in soup.text:
        return True
    else:
        return False


def extract_text_from_URL(URL):
    response = requests.get(URL, headers=chrome_headers)
    soup = BeautifulSoup(response.content, "lxml")

    return soup.text


if __name__ == '__main__':
    print(extract_text_from_URL("https://google.com/"))
    print(find_text_from_URL("https://google.com/", 'Google'))

• Basically we can use extract_text_from_URL() to see what your site normally returns and then use find_text_from_URL() to find whether the text is available. As far as I find, the vue.js rendered website does not return the proper response from requests, but it is anyway enough to use those texts.

Step 3: create a service that checks our website conditions every so often.

• Assuming we will have multiple website to monitor in the future, we can create a simple site_to_monitor.json to store the information we need. When we actually want to allow other users to add new sites, we probably should migrate to a database.

# backend/site_to_monitor.json
[{
    "URL": "https://google.com/",
    "text_to_find": "Google",
    "phone_number": "+441234567"
}]

• Now our monitor_service app

# backend/MonitorService.py

from TwilioWhatsapp import send_whatsapp_message
from URLChecker import  find_text_from_URL, extract_text_from_URL
import json
import time
import os

def monitor_service():
    while True:
        # every one hour check the service itself is working.
        send_whatsapp_message(os.environ['ADMIN_PHONE_NUMBER'], "From Downotifier: the monitoring service is fully functioning")

        for i in range(60):
            with open("site_to_monitor.json") as site_info:
                site_info = json.load(site_info)

            for site in site_info:
                if not find_text_from_URL(site['URL'], site['text_to_find']):
                    send_whatsapp_message(site['phone_number'], "Your site {} is down, please check it".format(site['URL']))
                    print(site['URL'] + " is down.")

            time.sleep(60)

if __name__ == "__main__":
    monitor_service()

Small wrap up: Now the monitoring service is ready, we shall prepare our backend API to allow user adding new website to monitor, see which text is currently returned by our service and also remove the old records if it is no longer necessary.

Part 2 of this article is here: