Contents: intro, what will be scraped, process, code, links, outro.
Intro
This blog post is a continuation of Googles' web scraping series. Here you'll see examples of how you can scrape Twitter Results from Google organic search using Python with beautifulsoup, requests libraries. An alternative API solution will be shown.
What will be scraped
Process
Selecting author info container, title, link, displayed link.
Same process goes for selecting tweet snippet, tweet published date, tweet link.
Code
import requests, lxml
from bs4 import BeautifulSoup
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
params = {
"q": "garyvee twitter",
"hl": "en",
"gl": "us"
}
response = requests.get("https://www.google.com/search", headers=headers, params=params)
soup = BeautifulSoup(response.text, 'lxml')
for result in soup.select('[jscontroller=yz368b]'):
title = result.select_one('g-link .a-no-hover-decoration .NsiYH').text
link = result.select_one('g-link .a-no-hover-decoration')['href']
displayed_link = result.select_one('g-link .a-no-hover-decoration .V0XQK').text
print(f'{title}\n{link}\n{displayed_link}\n')
for tweet in result.select('g-scrolling-carousel.rQgnxe .dHOsHb g-inner-card'):
tweet_link = tweet.select_one('.h4kbcd')['href']
tweet_snippet = tweet.select_one('.xcQxib').text
tweet_published = tweet.select_one('.kLhEKe .f:nth-child(3)').text
print(f'{tweet_link}\n{tweet_snippet}\n{tweet_published}')
-------------------
'''
Gary Vaynerchuk (@garyvee) · Twitterhttps://twitter.com/garyvee
https://twitter.com/garyvee?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor
https://twitter.com/garyvee
https://twitter.com/garyvee/status/1420238044259565569?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet
98% of NFT projects will not be good investments .. always spend on things you love or with $ u can afford to lose, this is a very high risk asset in the earliest of days
3 hours ago
https://twitter.com/garyvee/status/1420211442490486789?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet
Kind intent always wins in the end
5 hours ago
https://twitter.com/garyvee/status/1420207827210153987?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet
feels like ETH is the winner ... so many projects but one common theme
5 hours ago
'''
Using Google Twitter Results API
SerpApi is a paid API with a free plan.
Why use it over a custom-created code? No need to maintain the parser, and quicker access to the data, plus it scrapes the thumbnail of the tweet image (if it appears).
import json
from serpapi import GoogleSearch
params = {
"api_key": "YOUR_API_KEY",
"engine": "google",
"q": "garyvee twitter",
"hl": "en"
}
search = GoogleSearch(params)
results = search.get_dict()
title = results['twitter_results']['title']
link = results['twitter_results']['link']
displayed_link = results['twitter_results']['displayed_link']
tweets = results['twitter_results']['tweets']
print(f'{title}\n{link}\n{displayed_link}\n{json.dumps(tweets, indent=2)}\n')
----------------------
'''
Gary Vaynerchuk (@garyvee) · Twitter
https://twitter.com/garyvee?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor
https://twitter.com/garyvee
[
{
"link": "https://twitter.com/garyvee/status/1420238044259565569?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet",
"snippet": "98% of NFT projects will not be good investments .. always spend on things you love or with $ u can afford to lose, this is a very high risk asset in the earliest of days",
"published_date": "3 hours ago"
}
...
]
'''
Links
Code in the online IDE • Google Twitter Results API
Outro
If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.
Yours,
Dimitry, and the rest of SerpApi Team.
Top comments (0)