Contents: intro, imports, what will be scraped, process, code, links, outro.
Intro
This blog post contains visual info about how to scrape Bing Related Searches using Python.
Imports
from bs4 import BeautifulSoup
import requests
import lxml
from serpapi import GoogleSearch
import os # used for creating an environment variable on replit.com
What will be scraped
Process
Choosing the right CSS selectors. I'll show two approaches:
- Using Chrome Dev tools to see the HTML layout and test selectors with SelectorGadget extension
- Using solely SelectorGadget extension
From there either of selectors from two approaches will work:
Code
from bs4 import BeautifulSoup
import requests, lxml
headers = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
html = requests.get('https://www.bing.com/search?q=lion king&hl=en', headers=headers)
soup = BeautifulSoup(html.content, 'lxml')
for related_search in soup.select('.b_rs ul li'):
title = related_search.text
link = f"https://www.bing.com{related_search.a['href']}"
print(f'{title}\n{link}')
# part of the output:
'''
lion
https://www.bing.com/search?q=lion&FORM=QSRE1
jeremy irons
https://www.bing.com/search?q=jeremy+irons&FORM=QSRE2
'''
Using Bing Related Searches API
SerpApi is a paid API with a free trial of 5,000 searches.
Here's how you can get the results:
from serpapi import GoogleSearch
import os
params = {
"api_key": os.environ["API_KEY"], # pycharm environment
"engine": "bing",
"q": "lion king"
}
search = GoogleSearch(params)
results = search.get_dict()
for result in results['related_searches']:
query = result['query']
link = result['link']
print(f'{query}\n{link}')
# part of the output:
'''
elton john circle of life
https://www.bing.com/search?q=elton+john+circle+of+life&FORM=QSRE1
lion king theatre
https://www.bing.com/search?q=lion+king+theatre&FORM=QSRE2
'''
Links
Code in the online IDE • Bing Related Searches API
Outro
If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.
Yours,
Dimitry, and the rest of SerpApi Team.
Top comments (2)
I have been reading all you shared here. In this you explained everything very well. If i want any further guideline we will contact you here dev.to/dmitryzub/scrape-bing-related 49spredictions.com/
The Information you shared above is great. I have been reading all you shared here. In this you explained everything very well. If i want any further guideline we will contact you here dev.to/dmitryzub/scrape-bing-relat... floridaloteryresults.com/