Amazon Simple Storage Service (Amazon S3) is a very popular object storage service that offers industry-leading scalability, data availability, security, and performance. It allows you to virtually store unlimited number of files (objects). AWS provides JavaScript SDK to interact with S3 APIs. You can use aws-sdk to do variety of things like getting the list of objects in an S3 bucket, upload an object, download the object etc.
While working with it recently, I figured it would be handy to use the Iterator pattern to fetch files from s3 in a loop using for await...of. So came up with a basic implementation below. Please provide your feedback
Downside of this pattern or the code above is that only 1 file is processed at a time as we are awaiting the download of the file and processing it before moving to the next iteration. We could use Promise.all with regular loops. Do you guys think its doable in this pattern?
One improvement I definitely want to see is to only load next set of keys when we are running out of objects in the keysList instead of loading all (currently using loadKeys() function). Can someone fork and build that out? 😉 ciao!!
Top comments (0)