DEV Community


How to Build Fail-Safe Web Scrapers

Mohan Ganesan
Proxies API has been built to make Data mining more easier and efficient, to support the Online Crawling Apps & Software
Originally published at ・1 min read

Here is a list of places that your web crawler will probably fail at. You will need to build in checks for each and also expect them to happen. Send yourself alerts by having portions of the scripts check for unexpected behavior.

Discussion (0)