This is an overview of a blog post I recently wrote about how to scrap web pages using Python BeautifulSoup and Requests libraries.
Web scraping is the process of automatically extracting information from a website. Web scraping, or data scraping, is useful for researchers, marketers and analysts interested in compiling, filtering and repackaging data.
Web scraping in Python is a breeze. There are number of ways to access a web page and scrap its data. I have used Python and BeautifulSoup for the purpose.
In this example, I have scraped college footballer data from ESPN website.
- Install requests and beautifulsoup libraries
- Fetch the web page and store it in a BeautifulSoup object.
- Set a parser to parse the HTML in the web page. I have used the default html.parser
- Extract the player name, school, city, playing position and grade.
- Appended the data to a list which will be written to a CSV file at later stage.