DEV Community

Jordan Scrapes SteamDB

Jordan Hansen on September 23, 2019

Demo code here A request except I did agree to it This request comes straight at you from u/Jimmyxavi. Looks like he’s working on a pr...
Collapse
 
tonjohn profile image
Burton

Nice! If algolia has all the data used to render the page, you can directly call the algolia APIs instead of navigating to each page and scraping it. Should be more performant, simpler code, and less likely to get rate limited / blocked by SteamDB since you only hit them once to inject your javascript payload.

Collapse
 
aarmora profile image
Jordan Hansen

I looked at calling aloglia directly and it DID work but it's limited to just 1,000 results. I'm only a little familiar with algolia but I believe the algolia engine here is built for filtering. They don't expect anyone to want more than 1,000 results so there is no way for me to just "get all" without a filter.

I like your thinking, though!