Sending out SMS messages via Twilio

In the previous post, we scrapped a job posting site and were able to get a couple of jobs. Rememeber, I mentioned in Part 1 we will scrap a webpage and sent jobs to a user via SMS. Talking about SMS you should definitely checkout this article.

Before we jump right into creating the SMS functionality, let's first of all store our scrapped data into a very simple SQLite database. I will be creating a new file called data.py. This is where the code handling our data storage will reside.
We will create a class which will handle all operations in relation to the data base from creating to adding to deleting data. For now we will implement a few methods to be sure the basic functionality at least works. When the class will be instantiated, a SQLite connection will be established and both the database and the table will be created if they don't exist. We will have 2 methods to save the data to the database and also retrieve data (for now). When storing our data, we will make sure the URL is the primary key to make sure duplicate data won't be saved. With time we will implement error handling and refine the class.

import sqlite3

class JobsDataBase:
    def __init__(self):
        self.conn = sqlite3.connect("jobs_data.sqlite")
        self.cur = self.conn.cursor()
        self.cur.execute('CREATE TABLE IF NOT EXISTS jobs (id VARCHAR, description VARCHAR, PRIMARY KEY (id))')
        self.conn.commit()

    def save(self, id, body):
        self.cur.execute(f'INSERT OR IGNORE INTO jobs (id, description) values ("{id}", "{body}")')
        self.conn.commit()
        print("done")

    def getjobs(self):
        self.conn = sqlite3.connect("jobs_data.sqlite")
        self.cur.execute('SELECT * FROM jobs')
        data = self.cur.fetchall()
        return data

Now let's create a function called save_data within scrapper.py. Don't forget to import data.py.

def save_data(jobs):
    jobs_data = data.JobsDataBase()
    for j in jobs:
        jobs_data.save(j[0], j[1])

The get_job_details function in scrapper.py has been modified to return a list of lists containing job URLs and descriptions.

def get_job_details(urls):
    job_details = []
    for url in urls:
        page = requests.get(url)
        soup = BeautifulSoup(page.content, "html.parser")

        job = soup.find("div", class_="job-description")
        job_details.append([url, job.find("p").text])
    return job_details

Now let's dive into implementing our SMS functionality. After savind data into our database, we want to send out an SMS to the user. Considering the fact that there won't be duplicte data stored in the database, there are zero chances of sending out an SMS more than once (unless there's a bug in the code)

We will be using Twilio to implement the SMS functionality. wilio is a cloud-based communications company that offers tools for making and receiving calls, sending messages, and more. Twilio's products include voice, video, messaging, authentication, and more.

Before we dive into the code, we will start by creating a Twilio account and download its python [helper library].(https://github.com/twilio/twilio-python). Run the following command.

pip3 install twilio

Make sure you setup your Twilio console and get your account SID and Auth token. Create the variables account_sid and auth_token in your code and assign them their values. Be sure to also generate your Twilio phonenumber which you will use to send SMS messages.

Now lets start by importing our Twilio client.

from twilio.rest import Client

Now we will create a send_sms function. In the fuction you can see we when conposing the message to be sent we limited the characters. That's because there's a character limit and for now we won't dwell on hacking that limitation as our main focus is sending out an SMS via Twilio.

def send_sms(url, job):
    client = Client(account_sid, auth_token)
    text = f"{job[:500]}\n\nVisit site: {url}"
    message = client.messages.create(
    from_='twilio-phone-number',
    to='receiver-phone-number',
    body=text
    )

We will be sending out an SMS after saving any job into the database. So we will add an extra line in our save_data function.

def save_data(jobs):
    jobs_data = data.JobsDataBase()
    for j in jobs:
        jobs_data.save(j[0], j[1])
        send_sms(j[0], j[1])

Running the save_data function will send out SMS messages depending on the number of jobs. In our next articles, we will refine the data base, send out SMS messages with all the details, etc