DEV Community


Discussion on: Building a Polite Web Crawler

turnerj profile image
James Turner Author

Yep, you can supply any user agent in the crawl settings (see example). Providing a user agent like that, while will work perfectly fine, circumvents sites giving direction into what content is accessible or not via the "robots.txt" file.

Forem Open with the Forem app