The 6 commandments of web scraping

The 6 commandments of web scraping are:

1. Though shalt always use a framework.
2. Though shalt use CSS selectors or XPath for web scraping.
3. Though shalt build monitors that check for expected results in your crawler.
4. Though shalt imitate humans at every turn.
5. Though shalt use a rotating proxy service.
6. [Optional] Though shalt use these 4 external services to stabilize your web crawler.