DEV Community

Discussion on: Building a Polite Web Crawler

Collapse
 
turnerj profile image
James Turner

Yep, you can supply any user agent in the crawl settings (see example). Providing a user agent like that, while will work perfectly fine, circumvents sites giving direction into what content is accessible or not via the "robots.txt" file.