Test Driving the Popular Sentiment Analysis Services

Think about the following situations:

1. There are millions of products on Amazon. How does Amazon know when its customers are unhappy?
2. Tons of comments poured into social media before the presidential elections. How did the campaigns measure the public opinion? The answer is: Sentiment Analysis.

In this article, we’re going to talk about the problem that sentiment analysis solves, why it is popular, and a high level view of how it works. Then we will explore the following three services: Amazon comprehend, Google Cloud Natural Language, and IBM Watson™ Natural Language Understanding, and see how they perform on challenging examples.

Sentiment Analysis Usage and Approach

Why is sentiment analysis popular these days? Because it’s fast and insightful. We’re living in a world that is flooded with information from social media. Since nowadays people are spending a lot of time on the internet, social media becomes an extremely valuable source for business or politicians to find out people’s opinions. A typical sentiment detection application will be able to process a large amount of data and extract critical information - people’s love or hate about your product, service or brand in real-time. Furthermore, it can even quantify the degree of emotions to let you identify the most angry customers.

But how does sentiment analysis work? There are two base approaches to detect sentiment in text, lexicon/rule-based approach, and machine learning approach. Currently, there are already a lot of NLP and sentiment analysis services in the market. Many of the services are using a hybrid of the two and using large data sets to train their model.

Next we are going to explore some popular sentiment analysis technologies.

Sentiment Analysis Services

Amazon Comprehend is a natural language processing (NLP) service within the Amazon Web Service infrastructure. In order to use the Comprehend API, you need an AWS account and IAM user, then install the AWS SDK for the programming language you’re using. The following example is the usage in Golang.

Create a comprehend client:

comprehendClient = comprehend.New(sess, aws.NewConfig().WithRegion(“us-east-1"))

Create input and set detection language:

input := &comprehend.DetectSentimentInput{}
input.SetLanguageCode("en")
input.SetText()
output, err := comprehendClient.DetectSentiment(input)

The API will return a sentiment type (Positive, Neutral, Negative, Mixed), and a score indicates the confidence/likelihood score for each sentiment type. The following example is a response from DetectSentiment operation:

{
"SentimentScore": {
        "Mixed": 0.030585512690246105,
        "Positive": 0.94992071056365967,
        "Neutral": 0.0141543131828308,
        "Negative": 0.00893945890665054
    },
    "Sentiment": "POSITIVE",
    "LanguageCode": "en"
}

Google Cloud Natural Language API is another cloud natural language understanding service, which is part of the Cloud Machine Learning API. It has client library, currently available in C#, GO, Java, NODE.JS, PHP, Python, and Ruby. Or you can directly use the API by the GCLOUD command line tool. The sentiment analysis functionality currently supports 10 languages.

IBM Watson™ Natural Language Understanding can take input as text, HTML, or a public URL. The sentiment detection currently supports 10 languages (some of them are supported with less accuracy, like Spanish). An interesting feature of this API is it support emotion detection, like Joy, Sadness, Fear.

Sentiment Analysis Challenge

If you use the services to analyze some messages from social media (try some examples from celebrities on Twitter), you will find it works very well. But it would be much more interesting to see how they work with some challenging text. Currently, the most common types of challenges in sentiment analysis are Negations and Sarcasm.

Let us start with negations. Negators in a sentence, like “no, not”, will directly change the polarity of sentiment. I tried some examples with negation: “I am not very good”. “The show was not interesting”. All services will give Negative. So looks like the services can handle this case decently.

But things get really interesting when text input contains sarcasm and irony. We know that sarcasm is when people express negative feelings with positive words, so it can be a disaster for sentiment detection. For example, in the sentence "I love working on Sunday", the word “love” will be detected as a sentiment-bearing word that expresses intense positive sentiment, which leads the overall sentiment of the sentence to be positive.

In order to see how well these services work with sarcasm, I crowdsourced some tough examples from my coworkers.

Here are the awesome examples:

“A day will come when you think yourself safe and happy, and suddenly your joy will turn to ashes in your mouth, and you'll know the debt is paid.”
IBM: Positive
AWS: Positive
Google Cloud: Positive

(if you use the IBM emotion detection, the result seems to be a little more meaningful: Emotion: Joy 0.53, Sadness 0.17, Fear 0.22 )

“I’ll wipe that big happy smile off your face, and I’ll be laughing and grinning while doing so.”
All three services gave Positive sentiment results.

Looks like these examples are too hard for our services, so I’m going to use some simple but common type of sarcasm - the contrast between a positive sentiment and negative situation. All the services failed all tests except for one: “Oh, how I love being ignored!” (AWS and IBM successfully detect it as Negative while GCP give Positive)

Now we can see that currently the sentiment analysis providers AWS, Google Cloud, or IBM Watson are all not very competent to handle sarcasm, and their sentiment results are far from perfect. In order to deal with prevalent challenging texts from social media, you may want to consider applying filters that working dedicated to sarcasm before sending the text to the services.

Overall, sentiment analysis is a very useful and intriguing technology that can help you enhance your product or business routine. Also, the services in the market are doing a great job providing convenient integration and decent results. So go ahead to try and enjoy your journey on sentiment analysis!