DEV Community

Discussion on: Secure your Data API from Web Scrapers

Collapse
 
mxldevs profile image
MxL Devs

Thanks for the feedback. Session tokens are a fantastic tool and so common that I forgot about it, given that most apps require some form of authentication and therefore my activity can be easily logged and flagged.

What do you mean by sequential requests?

Collapse
 
kremnari profile image
Nick Smith

Basically checking the last N requests to see what their coverage (especially how broad of indicated content) would be. Depending on user privilege (an auditor, for example, would be an exception), the queries themselves probably shouldn't prompt a wide range of content. For example, sequential(ish) requests would be if someone requested all content for one month, then the previous month, then the 3rd. Having such broad requests could be used to detect a scraper. A user would likely be a bit more specific in what they are looking for. An occasional prompt (helpful hint?) could be provided to the user to be more specific, or even offer a suggestion. If such a prompt is ignored too often, it could again, be a +1 of the suspicious-o-meter.

This could also appear with something like TikTok, Instangram...ect, whereby a user can just scroll through a never-ending list. Each list is still governed by some criteria, even if it's generally handled server side, but the user can select something to view another user/category/tag/ect. The server can keep track of the hits/changes, and limit how much content is being provided by how fast the requests are being switched. A robot could be searching multiple queues at once, but a person is going to "enjoy" their content.

Ultimately, I agree with your philosophy... A strong goal is to make displaying further content take equivocally more time, both to protect server resources, and server content.

Thread Thread
 
mxldevs profile image
MxL Devs

Oh I see what you mean. User behaviour definitely is a good indicator based on how you've described it.

Devs probably will be using some sort of analytics framework to try and understand how users use the app. This can also be used to establish "regular" usage vs "irregular" usage, so it can serve multiple purposes!