In this series, I will show how to create a sentiment analysis app and perform analysis on any hashtag. The series is divided into 3 parts
- Building the classifier
- Building the backend
- Building the frontend
In the final post, we'll bring everything together to make the app. We'll use tools like nltk, docker, streamlit, fastAPI and the link to the code will be provided.
Final product
Building the classifier
We'll be using a pre-trained model that I trained using open-source code. This is not a SOTA(State-of-the-Art) model, but for our task, this should be fine.
Let's begin by making a project directory.
mkdir sentwitter && cd sentwitter
mkdir backend
mkdir frontend
Download the trained model to backend/models directory
wget https://raw.githubusercontent.com/amalshaji/sentwitter/master/backend/models/sentiment_model.pickle
mv sentiment_model.pickle /backend/models
Install required libraries
python3 -m pip install nltk
python3 -m nltk.downloader punkt
python3 -m nltk.downloader wordnet
python3 -m nltk.downloader stopwords
python3 -m nltk.downloader averaged_perceptron_tagger
Let's write a function to pre-process the input(tweet)
# backend/classify.py
import re, string
from nltk.tag import pos_tag
from nltk import WordNetLemmatizer
from nltk.tokenize import word_tokenize
def remove_noise(tweet_tokens, stop_words=()):
cleaned_tokens = []
for token, tag in pos_tag(tweet_tokens):
token = re.sub(
"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+#]|[!*\(\),]|"
"(?:%[0-9a-fA-F][0-9a-fA-F]))+",
"",
token,
)
token = re.sub("(@[A-Za-z0-9_]+)", "", token)
# remove all the links and special characters
if tag.startswith("NN"):
pos = "n"
elif tag.startswith("VB"):
pos = "v"
else:
pos = "a"
lemmatizer = WordNetLemmatizer()
token = lemmatizer.lemmatize(token, pos)
# pos tagging
if (
len(token) > 0
and token not in string.punctuation
and token.lower() not in stop_words
):
cleaned_tokens.append(token.lower())
# remove stopwords and punctuations
return cleaned_tokens
Write a helper function to load the model.
# backend/utils.py
import pickle
def load_model():
f = open("./models/sentiment_model.pickle", "rb")
classifier = pickle.load(f)
f.close()
return classifier
Test out classifier
# backend/test.py
from utils import load_model
from classify import remove_noise
from nltk.tokenize import word_tokenize
model = load_model()
while True:
_input = input("Enter a sample sentence: ")
custom_tokens = remove_noise(word_tokenize(_input))
result = model.classify(dict([token, True] for token in custom_tokens))
print(f"{_input}: {result}")
Output
❯ python .\test.py
Enter a sample sentence: I am awesome
I am awesome: Positive
Enter a sample sentence: I hate you
I hate you: Negative
Enter a sample sentence: I have a gun
I have a gun: Positive
Enter a sample sentence: I like your hair
I like your hair: Negative
This isn't the best model😂, gun labeled as Positive and hair sentence as Negative. Feel free to build your own model or try with the heavy ones using transformers( huggingface).
In the next post, we'll be building the backend to serve predictions through an API.
References
- The
open-sourcecode used to train the model was used a long time ago for a project, I can't find the source. So if you do, or you are the author, please comment the link. - Article Cover by MorningBrew
- Series Cover by Ravi Sharma
Top comments (0)