A Full Stack Developer specializes in Python (Django, Flask), Go, & JavaScript (Angular, Node.js). Experience designing, planning, and building complete web applications with backend API systems.
Hi,
Glad that you liked the post and it helped you in your use-case.
With this process of streaming the data, you have to keep retrieving the file chunk from S3 until you reach the total file size. I would recommend to clone this repo and compare with your local code to identify if you missed something 😉
Optionally, I would recommend to also check out the sequel to this post for parallel processing 😁
Thank you! I found out what I was missing, I made the start_byte = end_byte + 1. Losing one row per chunk. Your next article was exact what I was looking for for the next step of my program.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Hi,
Glad that you liked the post and it helped you in your use-case.
With this process of streaming the data, you have to keep retrieving the file chunk from S3 until you reach the total file size. I would recommend to clone this repo and compare with your local code to identify if you missed something 😉
Optionally, I would recommend to also check out the sequel to this post for parallel processing 😁
Parallelize Processing a Large AWS S3 File
Idris Rampurawala ・ Jun 25 ・ 6 min read
Thank you! I found out what I was missing, I made the start_byte = end_byte + 1. Losing one row per chunk. Your next article was exact what I was looking for for the next step of my program.