loading...
Cover image for What is the best way for web scraping?

What is the best way for web scraping?

nishantwrp profile image Nishant Mittal ・1 min read

Hi Guys,

I know that Scrapy can be used to scrape data. But also I want the code to be presentable on Github. I want to know what are best practices for web scraping using Python.

Also, if you guys know any web scraping project on Github please provide me the link to it.

Discussion

pic
Editor guide
Collapse
patarapolw profile image
Pacharapol Withayasakpunt

Indeed, in the past, I used Python.

  • Concurrency -- try ThreadPoolExecutor, or some kinds of coroutines. It may speed up things a lot.
  • GET the content. I guess requests is OK.
  • Locating the content. I now prefer lxml to BeautifulSoup.

As you have noticed in some of the comments, you might try Node.js, where you can use Cheerio, which is jQuery-ish; but has no problem with CORS. (You may still need to fetch with axios or Node-fetch, though.)

Collapse
nishantwrp profile image
Nishant Mittal Author

Yup, but still I'm wondering why almost no one is in favour of Scrapy.

Collapse
peter_jachim profile image
Peter Jachim

I really like to use requests to make http requests, with bs4 to parse data. I am generally able to get what I need in about 4 lines of code, or about 10 to iterate through links that meet specific criteria. I think it generally looks pretty pythonic and does everything I need it to.

If there are a lot of tables, you can use pandas to read them in as dataframes, and if you need to click through pop ups or fill out forms you can use selenium (which is less presentable, but still super interpretable).

I actually used all of those techniques in this paper: arxiv.org/abs/2006.13990 (though the code is not available, so not super helpful for you).

Collapse
nishantwrp profile image
Nishant Mittal Author

Thanks! Looks like beautiful soup is a common choice!

Collapse
leonlafa profile image
Leon Lafayette

I've found puppeteer and cheerio to be a good combo.

Collapse
nishantwrp profile image
Nishant Mittal Author

This is the first time I've heard about cheerio. It looks nice but honestly I'm not a big fan of jquery.

Collapse
leonlafa profile image
Leon Lafayette

Me either.

I was pushed for time tbf and found cheerio made things a little less verbose allowing me to get things done quickly :D

Thread Thread
nishantwrp profile image
Collapse
john9088 profile image
Jason Britto

I would suggest Selenium, its very easy with few methods, which can be used to explore DOM and fetch data.
I myself have made few projects with the help of youtube.
Try any project tutorial from youtube provided if you are alright familiar with basic python you will understand without any problem

Collapse
nishantwrp profile image
Nishant Mittal Author

Thanks for the suggestion but I don't think selenium would be the best fit for me.

Collapse
eruizdechavez profile image
Erick Ruiz de Chavez

I do not understand this:

But also I want the code to be presentable on Github.

What do you want to be "presentable" on GitHub, your code, or the code you scrape from other websites?

Collapse
nishantwrp profile image
Nishant Mittal Author

Actually both!

Collapse
vigneshwaranchandrasekaran profile image
Vigneshwaran Chandrasekaran

use puppeteer js

Collapse
nishantwrp profile image
Nishant Mittal Author

Ok that's new! Maybe you can point towards a already written code or a tutorial for web scraping using puppeteer.

Collapse
jldohmann profile image
Jesse

BeautifulSoup is great, and I've had a good experience with it. lxml (XPath) is my go-to though, and I like it!

Collapse
nishantwrp profile image
Nishant Mittal Author

Thanks for sharing!

Collapse
mxdws profile image
Martin Dawson

There are more than one ways to scrape with Python, but Beautiful Soup is definitely a stable, well documented, tried and tested library to use. I made a video about how to use it if that might help. Also wrote an article too 😀

Collapse
nishantwrp profile image
Nishant Mittal Author

Thanks! Will definitely take a look!

Collapse
sunilaleti profile image
Sunil Aleti

You can check my tutorial dev.to/aletisunil/demystify-the-we...
Hope it helps

Collapse
nishantwrp profile image
Nishant Mittal Author

Thanks! Will check it out!

Collapse
maheshthedev profile image
Mahesh Sv

I created a Telegram automated Bot. I used Webscraping in the project. You can check the project here github.com/maheshthedev/DataScienc...

Collapse
nishantwrp profile image
Nishant Mittal Author

I see you've used BeautifulSoup. Thanks for the code!

Collapse
mrakonja profile image
Mladen Milosavljevic

Scrapy is fastests but hardest to master.
Beautiful Soup and Selenium are better for beginners.

Collapse
nishantwrp profile image
Collapse
andrewbrown profile image
Andrew Brown 🇨🇦

The best way is to not get caught 🙃
I just use Nokogiri

Collapse
nishantwrp profile image
Collapse
webscrapingx profile image
RedAndGreen Web Scraping

github.com/RGGH

Scrapy repos x 6

Scrapy has steeper learning curve, but that means it's better once you've learned it!

Collapse
nishantwrp profile image
Nishant Mittal Author

Thanks for sharing your code!

Collapse
thomasstep profile image
Thomas Step

I used to use something called beautiful soup. I'm not sure what the standard is now though since that was a few years back.

Collapse
nishantwrp profile image
Nishant Mittal Author

It looks like everybody is using it!