Despite reports of application crashes and buffering streams, the FIFA World Cup 2022 was a tremendous triumph, generating $7.5 billion for FIFA. Let's examine its twitter data and gauge the general public's sentiments regarding specific news events, such as "Argentina winning the FIFA World Cup."
The underlying foundation of our social media analytics pipeline is composed of diverse platforms and relies heavily on the cloud infrastructure provided by Amazon Web Services (AWS). The objective is to assess the general public's feelings regarding recent news events. The system processes data through three successive stages to derive meaningful insights from newly published tweets:
- Initial data ingestion
- Analysis phase
- Data visualization phase
Data Ingestion
The process of Data Ingestion involves the use of a Java application that connects to the Twitter Streaming API via the Twitter4j library to gather real-time tweets. The application communicates with a MySQL relational database through the JDBC library and stores tweets in the database immediately after publication. The application runs on an AWS Linux EC2 instance, continually ingesting tweets, including those with relevant keywords for sentiment analysis, apart from those containing the phrase "Argentina wins the FIFA World Cup."
Data Analysis
Data Analysis is carried out using the AWS CloudWatch service, which triggers a serverless Lambda function at regular intervals to filter and analyze English tweets for the positive, negative, neutral, and mixed sentiment. The Lambda function reads unprocessed tweets from the MySQL database and conducts sentiment analysis, keyword extraction using AWS Comprehend, and location matching using the Python geo-text package.
Data Visualization
Data Visualization is achieved using AWS QuickSight, which creates a dashboard presenting data visualization analytics about the breakdown of tweet sentiments, most prevalent terms, and user regions. It updates data continually, providing users with a real-time experience.
The data pipeline for streaming Twitter data comprises the following processes.
- Extract tweets that include particular keywords using the Twitter API and put these tweets into the Kinesis firehose deployed on AWS EC2.
- Create an S3 bucket for storing processed data.
- Set up an Amazon Redshift cluster.
- Configure AWS Glue to continuously ingest data from Kinesis.
- Connect Tableau to Amazon Redshift as a data source and see the graphical representation of data.
Social media data analysis is just one of the many use cases of data engineering pipelines. Billion-dollar companies like Netflix and Samsung have deployed data pipelines to achieve their goals.
Top comments (0)