DEV Community

Derrick Sherrill for WayScript

Posted on

Scrape Web App Data & Add it to Google Sheets with WayScript


Scraping data from Twitter is very useful when you want to make sentiment analysis or understand a phenomenon taking place online. However it requires a good level of coding and a better comprehension of changing websites structures. Using Wayscript, we can bypass these problems, and seamlessly integrates Twitter and Google Sheets. Today we’ll create a script to automatically scrape Twitter for social mentions add to Google Sheets.


No prerequisites but some content you might find useful:

Building our script

Our workflow will look like this:

  • Add a time trigger to periodically run the Twitter Scraper module
  • Use the list returned by the module and loop through it to get each item
  • Use important data about each item and add it to sheet files

You can work with any trigger you want like a text message trigger or Gmail trigger, but here we’ll use a time Trigger.

Adding Time Trigger

The Twitter Module

The next step is to scrap Twitter according to a special mention or search item. For this purpose, we’ll use Twitter Module provided by Wayscript to get the latest 50 tweets about cryptocurrency.

Add Twitter Trigger

Set the mode to Search for word or #hashtag. In advanced input parts, you can select options such as ignoring retweets, replies and choose the result type (mixed or popular).

Setting input search

We've selected the Tweet outputs to get a list. It will be more easier to work with. You'll notice some changes in variables panel.

Variables panel

Add to Google Sheet

The Twitter module will return a list of 50 tweets, each item of the list containing a lot of information. Here, we will only use the username, the name, the content, and the date.

But first, we must add a loop step.

Adding loop step

Then in the loop add the Google Sheet module and set it to Write Mode.

Google Sheet configuration

Now let’s add the variables. Understand that here each variable is a column.

Adding variables

Final Step

Now turn on the time trigger. The automation of this script is handled by Wayscript. Then, the google sheet will be updated every day at 11.00 am.

Turn on Time Trigger

Questions or concerns?

Questions about this script or anything else? Join our discord. We're always around to help. If you want to work the full script template, just find it here.

Top comments (0)