DEV Community

Cover image for Scrape Google Inline Videos with Python
Dmitriy Zub ☀️
Dmitriy Zub ☀️

Posted on • Updated on

Scrape Google Inline Videos with Python

Contents: intro, imports, what will be scraped, process, code, links, outro.


This blog post is a continuation of Google's web scraping series. Here you'll see examples of how you can scrape Inline Videos from Google Search using Python using beautifulsoup, requests and lxml libraries. An alternative API solution will be shown.


import requests, lxml
from bs4 import BeautifulSoup
from serpapi import GoogleSearch
Enter fullscreen mode Exit fullscreen mode

What will be scraped



Selecting Container. Link lays directly in the container under href attribute.

Selecting Title, Channel name, Platform, Date, Duration CSS selectors.


import requests, lxml
from bs4 import BeautifulSoup

headers = {
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"

response = requests.get(" last of us 2 reviews", headers=headers)
soup = BeautifulSoup(response.text, 'lxml')

for result in'.WpKAof'):
    title = result.select_one('.p5AXld').text
    link = result['href']
    channel = result.select_one('.YnLDzf').text.replace(' · ', '')
    video_platform = result.select_one('.hDeAhf').text
    date = result.select_one('.rjmdhd span').text
    duration = result.select_one('.MyDQSe span').text

The Last of Us 2 Review
Jun 12, 2020
Enter fullscreen mode Exit fullscreen mode

Using Google Inline Videos API

SerpApi is a paid API that provides a free trial of 5,000 searches.

The main differences is you don't have to maintain the parser, e.g. if layout/selectors is changed there's no need for debugging since it already done for the end-user, because at times it could annoying...

import json # used for pretty print output
from serpapi import GoogleSearch

params = {
  "api_key": "YOUR_API_KEY",
  "engine": "google",
  "q": "the last of us 2 review",
  "gl": "us",
  "hl": "en"

search = GoogleSearch(params)
results = search.get_dict()

for results in results['inline_videos']:
    print(json.dumps(results, indent=2, ensure_ascii=False))

  "position": 1,
  "title": "The Last of Us 2 Review",
  "link": "",
  "thumbnail": "",
  "channel": "IGN",
  "duration": "8:01",
  "platform": "YouTube",
  "date": "Jun 12, 2020"
Enter fullscreen mode Exit fullscreen mode


Code in the online IDEGoogle Inline Videos API


If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.

Dimitry, and the rest of SerpApi Team.

Top comments (0)