DEV Community

loading...

Discussion on: What was your win this week?

Collapse
dazfuller profile image
Darren Fuller

A bit of Python which asynchronously downloads data from an API. But which also uses queues and worker process functions to throttle the query rate so as to not overload their systems (and so I don’t end up with a load of 429 responses). At the end it parses all the XML response data into csv. Used it to download 2.5 years worth of data (can only query one day at a time) to unblock a colleague.

I’ve used the aiohttp library before a lot, but this is the first time I’ve been happy with the throttling process as well.

Collapse
dazfuller profile image
Darren Fuller

Oh, and it’s the first time I’ve blogged in a few years, so happy to be trying to get back into writing more content.

aizoo.info/post/bringing-azure-dat...

Collapse
hunterpp profile image
Hunter Peress

All your data are belong to us ;-) Congratulations!

Forem Open with the Forem app