DEV Community

Cover image for Sentiment Analysis for Tweets in a Chat Bot - Part 2 Integrations
Albert Bennett
Albert Bennett

Posted on • Updated on

Sentiment Analysis for Tweets in a Chat Bot - Part 2 Integrations

#azure #twitter #webdev #csharp

If you liked what you read feel free to connect with me on linkedin or follow me on dev.to :)

Hi and welcome to part 2 of a series of posts that I am writing all about how you can use Azure, luis.ai and the twitter API to create a chat bot that can be used to search and do sentiment analysis on some tweets.

In this part I'll be going on to the coding and integration side of things using a function app.
You can check out the repos for this project by clicking on the links below:
https://github.com/Albert-Bennett/TwitterSentimentAnalysisFunctionApp
https://github.com/Albert-Bennett/TwitterSentimentAnalysisChatBot

The function app
To start we will need to create a new function app. Mine is setup with a http trigger and Open API. The Open API part isn't really required for this project but, I like to have automated documentation where I can just in case I need to refer back to the functionality or various other details of the function app as I go along.
new function app
It's also worth noting that I will be using .net 6.0 for the project.

The request and response
So, the plan for the function app is that it accepts a get request that defines a set of hash tags and a number defining the max number of tweets to search for in the twitter API.
To do this we will make sure that the http trigger accepts a get request and defines variables based on query parameters.

           int maxTweets = 0;

            if (!int.TryParse(req.Query["max_tweets"], out maxTweets))
            {
                return new BadRequestObjectResult("max_tweets must be a number greater than 0 and less than 100");
            }

            string hashtagQueryParam = req.Query["hashtags"];

            string[] hashtags = hashtagQueryParam.Split('\u002C');
Enter fullscreen mode Exit fullscreen mode

With the query parameters defined we need to decide what will be returned from the function app on a successful response. I think the best would be to return the top X most popular tweets from the ones that we got back from twitter, as well as details on the sentiment analysis of the found tweets. To do this we will structure the response object as such:

    public enum TweetSentiment
    {
        Neutral = 0,
        Positive = 1,
        Negative = 2,
InConclusive = 3
    }

    internal class FNResponse
    {
        public TweetData[] MostPopularTweets { get; set; } 
        public int NumberOfTweetsFound { get; set; }
        public Dictionary<TweetSentiment, int> TweetSentimentAnalysis { get; set; } 
    }
Enter fullscreen mode Exit fullscreen mode

We are using a enum to manage the response of the sentiment analysis as it is possible for there to be no results for one or more of the analysis categories (positive, negative, neutral, in conclusive). The 'InConclusive' results is only returned for tweets that the analysis can't\ isn't done on or if there was an error when doing the sentiment analysis on that tweet.

Twitter API
Next we are going to need to check what endpoint we need to call and what the response for that action is, on the Twitter API. We need these details so we know what we have to call in the API and how to model our data objects.
From looking at the documentation we can see that the endpoint that we need to call is: https://api.twitter.com/2/tweets/search/recent
To this we will need to add the URL encoding for the hashtag symbol that being %23. This is because the # is a special character and can cause issues when adding it to a URL, also the standard Twitter API doesn't have an option to search for a specific hashtag just a content search so to search for hastags we need to add it to the search query.
Next looking at the response object from the docs and testing out the endpoint through Postman we can see that we need to model our data as such.

    public class TwitterSearchResult
    {
        public TweetData[] data { get; set; }
        public TwitterSearchResultMetaData meta { get; set; }
    }

    public class TweetData
    {
        public TweetPublicMetrics public_metrics { get; set; }
        public string text { get; set; }
    }

    public class TweetPublicMetrics
    {
        public int retweet_count { get; set; }
        public int reply_count { get; set; }
        public int like_count { get; set; }
        public int quote_count { get; set; }

        /// <summary>
        /// This method is just going to add up all of the public metrics to get an arbitrary popularity metric
        /// </summary>
        /// <returns></returns>
        public int GetPopularity()
        {
            return retweet_count + reply_count + like_count + quote_count;
        }
    }

    public class TwitterSearchResultMetaData
    {
        public string next_token { get; set; }
    }
Enter fullscreen mode Exit fullscreen mode

We don't need the ID from the tweet object as we won't be using it but, we do need to know the public metrics (likes, replies, etc) and of course the text from the tweet. There is other data the gets returned from the twitter API when calling that endpoint but, we don't need so so I did model the data for it.
With all of this info we can build up a good idea of what we want to send to twitter and what we want to get back.

The Twitter Service
Super! with all of this information we can now start building up the services that we need to get the function app to do what we want.
For starters we need our TwitterService.

    public class TwitterService : ITwitterService
    {
        readonly IHttpClientFactory _httpClientFactory;

        readonly string bearerToken;
        readonly string baseUrl;

        public TwitterService(IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;

            bearerToken = Environment.GetEnvironmentVariable("TwitterBearerToken");
            baseUrl = Environment.GetEnvironmentVariable("TwitterBaseUrl");
        }

        public async Task<TweetData[]> FindTweetsByHashtag(FNRequestBody request)
        {
            int maxResults = request.MaxTweets;
            string paginationToken = null;
            string hashtagQuery = GetHashtagQuery(request.HashTags);

            List<TweetData> tweetsFound = new();

            while (tweetsFound.Count < maxResults && (paginationToken != null || tweetsFound.Count == 0))
            {
                var twitterSearchResult = await GetNextSetOfTweets(paginationToken, hashtagQuery);

                if (twitterSearchResult != null)
                {
                    paginationToken = twitterSearchResult.meta.next_token;

                    if (tweetsFound.Count + twitterSearchResult.meta.result_count > maxResults)
                    {
                        var diff = maxResults - tweetsFound.Count;

                        for (int i = 0; i < diff; i++)
                        {
                            tweetsFound.Add(twitterSearchResult.data[i]);
                        }
                    }
                    else
                    {
                        tweetsFound.AddRange(twitterSearchResult.data);
                    }
                }
                else
                {
                    return tweetsFound.ToArray();
                }

            }

            return tweetsFound.ToArray();
        }

        string GetHashtagQuery(string[] hashTags)
        {
            string query = string.Empty;

            foreach (string hashTag in hashTags)
            {
                var tag = hashTag.StartsWith('#') ?  hashTag : $"#{hashTag}";
                query = string.IsNullOrEmpty(query) ? tag : query + $" {tag}";
            }

            return query;
        }

        async Task<TwitterSearchResult> GetNextSetOfTweets(string paginationToken, string query)
        {
            string paginationSubQuery = string.IsNullOrEmpty(paginationToken) ? string.Empty : $"&next_token={paginationToken}";

            string url = HttpUtility.UrlEncode($"{baseUrl}?query={query}&tweet.fields=public_metrics{paginationSubQuery}");

            var message = new HttpRequestMessage(HttpMethod.Get, url)
            {
                Headers =
                {
                    { "Authorization", $"Bearer {bearerToken}" }
                }
            };

            using (var client = _httpClientFactory.CreateClient())
            {
                var response = await client.SendAsync(message);

                if (response.IsSuccessStatusCode)
                {
                    using var contentStream = await response.Content.ReadAsStreamAsync();

                    return await JsonSerializer.DeserializeAsync<TwitterSearchResult>(contentStream);
                }
            }

            return null;
        }
Enter fullscreen mode Exit fullscreen mode

Simply put, the twitter service connects to twitter. It creates the endpoint and sends on the query request to the twitter API. It then sends back data to us that we can use with luis to do the sentiment analysis. The only gotcha moment with the API call is that the twitter API only sends back 10 results at a time. To get around this they also send on a pagination token. We can use that token to get the next set of results on and on until we have the max amount that has been requested.

luis Data Modeling
This next part is a little easier as we can test out the luis prediction endpoint that we gathered from the previous step in Postman. With this we can see what the returned object is from the luis API.
luis prediction response
With this info we can model our data as such:

    public class LuisQueryResult
    {
        public LuisPrediction prediction { get; set; }
    }

    public class LuisPrediction
    {
        public LuisSentiment sentiment { get; set; }
    }

    public class LuisSentiment
    {
        public string label { get; set; }
    }
Enter fullscreen mode Exit fullscreen mode

Although the response object from luis was much more complex and there are query parameters that we can add to get more data, I'd like to keep the responses back as lean as possible. Besides for our project all we really need is the sentiment analysis portion of the response.

The luis Service
This is going to be much the same as the twitter service. We need to build up a query and send that as a get request to the endpoint and then transform the results.

    public class LuisService : ILuisService
    {
        readonly IHttpClientFactory _httpClientFactory;

        readonly string subscriptionKey;
        readonly string appId;
        readonly string baseUrl;
        readonly string endpoint;

        public LuisService(IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;

            subscriptionKey = Environment.GetEnvironmentVariable("SubscriptionKey");
            appId = Environment.GetEnvironmentVariable("AppId");
            baseUrl = Environment.GetEnvironmentVariable("LuisBaseUrl");
            endpoint = Environment.GetEnvironmentVariable("LuisEndpoint");
        }

        public async Task<Dictionary<TweetSentiment, int>> GetSentimentAnalysisOnTweets(TweetData[] tweets)
        {
            Dictionary<TweetSentiment, int> sentimentAnalysis = new Dictionary<TweetSentiment, int>();

            foreach (TweetData tweet in tweets)
            {
                var analysis = await GetSentimentAnalaysis(tweet.text);

                if (sentimentAnalysis.ContainsKey(analysis))
                {
                    sentimentAnalysis[analysis]++;
                }
                else
                {
                    sentimentAnalysis.Add(analysis, 1);
                }
            }

            return sentimentAnalysis;
        }

        async Task<TweetSentiment> GetSentimentAnalaysis(string text)
        {
            string url = $"{baseUrl}{appId}{endpoint}?subscription-key={subscriptionKey}&query={HttpUtility.UrlEncode(text)}";

            using (var client = _httpClientFactory.CreateClient())
            {
                var response = await client.GetAsync(url);

                if (response.IsSuccessStatusCode)
                {
                    using var contentStream = await response.Content.ReadAsStreamAsync();

                    var result = await JsonSerializer.DeserializeAsync<LuisQueryResult>(contentStream);

                    switch (result.prediction.sentiment.label)
                    {
                        case LuisConstants.NegativeResult:
                            return TweetSentiment.Negative;

                        case LuisConstants.PositiveResult:
                            return TweetSentiment.Positive;

                        default:
                            return TweetSentiment.Neutral;
                    }
                }
            }

            return TweetSentiment.InConclusive;
        }
    }
Enter fullscreen mode Exit fullscreen mode

The only gotcha ya' in this service was the addition of an 'InConclusive' sentiment. this is for when the luis API returns back an unsuccessful response from something like a bad request of for querying the API too often in a short time span.

Putting it all together
Finally on to gluing it all together. What we need to do is use the two services to build up the response body and use the popularity metric that we added to the public_metric object to return the X most popular tweets, with the sentiment analysis done.

        public async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest req)
        {
            int maxTweets = 0;

            if (!int.TryParse(req.Query["max_tweets"], out maxTweets))
            {
                return new BadRequestObjectResult("max_tweets must be a number greater than 0 and less than 100");
            }

            if (maxTweets <= 0)
            {
                return new BadRequestObjectResult("max_tweets must be greater than 0");
            }
            else if (maxTweets > 100)
            {
                return new BadRequestObjectResult("max_tweets must be less than 100");
            }

            string hashtagQueryParam = req.Query["hashtags"];

            string[] hashtags = hashtagQueryParam.Split('\u002C');

            if (hashtags == null || hashtags.Length == 0)
            {
                return new BadRequestObjectResult("You must include hashtags to search in the request body");
            }

            var foundTweets = await _twitterService.FindTweetsByHashtag(hashtags, maxTweets);

            if (foundTweets != null)
            {
                var maxNumberOfPopularTweets = int.Parse(Environment.GetEnvironmentVariable("MaxNumberOfPopularTweets"));
                var mostPopularTweets = foundTweets.OrderByDescending(x => x.public_metrics.GetPopularity()).Take(maxNumberOfPopularTweets).ToArray();

                var fnResponse = new FNResponse
                {
                    MostPopularTweets = mostPopularTweets,
                    NumberOfTweetsFound = foundTweets.Length,
                    TweetSentimentAnalysis = await _luisService.GetSentimentAnalysisOnTweets(foundTweets)
                };

                return new return new OkObjectResult(fnResponse);
            }

            return new BadRequestObjectResult($"No tweets found searching for the following hashtags: {hashtags}.");
        }
    }
Enter fullscreen mode Exit fullscreen mode

There is really not much to our entry point just, some data validation and then the assembly of the various data points.
With that all done, the result should look something like this.
function app output
The last last part I'm going to go over is the local setting file where I have placed all of the configuration for the function app in case you want to try and get the bot working by yourself.

{
    "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "TwitterBearerToken": "[your twitter bearer token]",
    "TwitterBaseUrl": "https://api.twitter.com/2/tweets/search/recent",
    "MaxNumberOfPopularTweets": 3,
    "SubscriptionKey": "[your azure account subscription key]",
    "AppId": "[your luis app id]",
    "LuisBaseUrl": "https://[the name of your luis app].cognitiveservices.azure.com/luis/prediction/v3.0/apps/",
    "LuisEndpoint": "/slots/production/predict"
  }
}
Enter fullscreen mode Exit fullscreen mode

For demo purposes placing the config here is fine but, in a more production environment these should be held in something like KeyVault. Unless you are testing locally, in which case it's also fine to keep them locally so long as the local.settings file doesn't get committed to the repo and you are using non production settings.

With this series of posts I'm just going over the key parts to the application. There are other code tasks that I had done and can be seen in the repos for this project. If you need a crash course in it or on function apps I have a few posts where I got over it that can be found below:

With that thanks for reading my post and I'll see you in part 3 for the finale and the bot side of this chat bot.

Top comments (0)

Read next

emregursanc profile image

What is SEO (Search Engine Optimization)?

Emre Gürsanç -

goodylili profile image

OAuth Tutorial with Go and the Spotify API

Ukeje Chukwuemeriwo Goodness -

harman_diaz profile image

Benefits of Azure Advanced Threat Protection You Need to Know

Harman Diaz -

ariflogs profile image

Building an UI library that does not look generic!

Arif Hossain -