I had a question and wanted to get some software architecture ideas to see how others would go about designing this system out.
The basic idea/flow of the system is as followed:
- User uploads a file to a server, which will store it on some cloud storage, like AWS S3.
- Once file has been transferred to S3 it needs to get processed by a specific on premise server. If that server is currently busy processing another file it needs to wait in a queue.
- The on premise server doing the processing needs to read the files from the queue and process them.
- Once a file is finished processing the results need to be uploaded back to the storage, and we need to notify the user that everything is done.
- Throughout all of this some status updates to the user would be nice to see. For example: "Waiting to be processed..." "File is being processed..." "Results Ready."
I guess some general notes: The on premise machine doing the processing would be a Linux box.
I'm just looking for some general ideas on what some others would be looking at. Any ideas are great. Thanks!