loading...

re: Jordan Scrapes SteamDB VIEW POST

FULL DISCUSSION
 

Nice! If algolia has all the data used to render the page, you can directly call the algolia APIs instead of navigating to each page and scraping it. Should be more performant, simpler code, and less likely to get rate limited / blocked by SteamDB since you only hit them once to inject your javascript payload.

 

I looked at calling aloglia directly and it DID work but it's limited to just 1,000 results. I'm only a little familiar with algolia but I believe the algolia engine here is built for filtering. They don't expect anyone to want more than 1,000 results so there is no way for me to just "get all" without a filter.

I like your thinking, though!

code of conduct - report abuse