DEV Community

Discussion on: Web Scraping Walkthrough with Python

Collapse
 
westerdal profile image
Jay Westerdal • Edited

You can always work around them banning your IP by using spider.com. They have millions of IPs and allow you to crawl anything and not get blocked.

A terms of service is not the law, there is nothing illegal about scraping a website. Read: eff.org/deeplinks/2018/01/ninth-ci...

Collapse
 
thebouv profile image
Anthony Bouvier

It is not illegal (in the US, but keep in mind not everyone on this site is US-based nor are the companies that might get targeted by a spider written here).

However, it may be unethical.

"Please don't do this to our site and our property and our data."

"Yeah, well, screw you. I'm doing it anyway."