Hi Guys,
I know that Scrapy can be used to scrape data. But also I want the code to be presentable on Github. I want to know what are best practices for web scraping using Python.
Also, if you guys know any web scraping project on Github please provide me the link to it.
Indeed, in the past, I used Python.
requestsis OK.As you have noticed in some of the comments, you might try Node.js, where you can use Cheerio, which is jQuery-ish; but has no problem with CORS. (You may still need to fetch with axios or Node-fetch, though.)
Yup, but still I'm wondering why almost no one is in favour of
Scrapy.I've found puppeteer and cheerio to be a good combo.
This is the first time I've heard about cheerio. It looks nice but honestly I'm not a big fan of jquery.
Me either.
I was pushed for time tbf and found cheerio made things a little less verbose allowing me to get things done quickly :D
Oh, ok.
I really like to use requests to make http requests, with bs4 to parse data. I am generally able to get what I need in about 4 lines of code, or about 10 to iterate through links that meet specific criteria. I think it generally looks pretty pythonic and does everything I need it to.
If there are a lot of tables, you can use pandas to read them in as dataframes, and if you need to click through pop ups or fill out forms you can use selenium (which is less presentable, but still super interpretable).
I actually used all of those techniques in this paper: arxiv.org/abs/2006.13990 (though the code is not available, so not super helpful for you).
Thanks! Looks like beautiful soup is a common choice!
BeautifulSoup is great, and I've had a good experience with it. lxml (XPath) is my go-to though, and I like it!
Thanks for sharing!
use puppeteer js
Ok that's new! Maybe you can point towards a already written code or a tutorial for web scraping using puppeteer.
I would suggest Selenium, its very easy with few methods, which can be used to explore DOM and fetch data.
I myself have made few projects with the help of youtube.
Try any project tutorial from youtube provided if you are alright familiar with basic python you will understand without any problem
Thanks for the suggestion but I don't think selenium would be the best fit for me.
There are more than one ways to scrape with Python, but Beautiful Soup is definitely a stable, well documented, tried and tested library to use. I made a video about how to use it if that might help. Also wrote an article too 😀
Thanks! Will definitely take a look!
You can check my tutorial dev.to/aletisunil/demystify-the-we...
Hope it helps
Thanks! Will check it out!
Scrapy is fastests but hardest to master.
Beautiful Soup and Selenium are better for beginners.
I agree.
I created a Telegram automated Bot. I used Webscraping in the project. You can check the project here github.com/maheshthedev/DataScienc...
I see you've used BeautifulSoup. Thanks for the code!
I used to use something called beautiful soup. I'm not sure what the standard is now though since that was a few years back.
It looks like everybody is using it!
The best way is to not get caught 🙃
I just use Nokogiri
Haha
github.com/RGGH
Scrapy repos x 6
Scrapy has steeper learning curve, but that means it's better once you've learned it!
Thanks for sharing your code!
Actually both!