DEV Community

macnux
macnux

Posted on • Edited on

Python Web Scraping

In this post, we will talk about Python web scraping and how to scrape web pages using multiple Python scraping libraries such as Beautifulsoup, Selenium, and some other magic tools like PhantomJS.

You’ll learn how to scrape static web pages, Ajax loaded content, iframes, how to handle cookies and much more stuff.

1 What is Web Scraping
2 Benefits of Web Scraping
3 How to Use Beautifulsoup
4 Your First Python Web Scraper
4.1 Handling HTTP Exceptions
4.2 Handling URL Exceptions
5 Beautifulsoup Find by Class
6 Beautifulsoup findAll Examples
7 Find nth Child Using Beautifulsoup
8 Find Tags using Regex
9 Scraping JavaScript
9.1 Chrome Web Scraper Using Selenium
10 Selenium Web Scraping
10.1 Selenium page_source
11 Get iframe Content Using Selenium
12 Get iframe Content Using Beautifulsoup
13 Handle Ajax Calls Using (Selenium+ PhantomJS)
14 Wait for Ajax Calls to Complete Using PhantomJS
15 Handling Cookies
16 Scraping VS Crawling

Article link
Thanks in advance.

Top comments (2)

Collapse
 
microworlds profile image
Caleb David

Great post

Collapse
 
m_waqarsikandar profile image
Software Craftsman

Check out I'm using Scrapy tool in python!!!
fiverr.com/m_waqarsikandar