DEV Community

Mohan Ganesan
Mohan Ganesan

Posted on • Originally published at

How to Set User Agent String while Web Scraping

Are you getting IP blocked repeatedly when web scraping at scale?

Is this happening even after passing it through a couple of free proxy servers?

It might be because the target website’s algorithm might be picking up on who you are by the User-Agent-String signature that your curl request or any other library you might be using is sending. Even the absence of it is telling.

Discussion (0)