I can advice this approach to make your system more resilient and prevent duplication:
Each url should have a sate enum [waiting, processing , terminated]
The worker should read and update the state inside a lock, the lock will prevent that two or more worker take the same job because all of them see it at a same time as available.
in rails you can do it by
Have a Timeout on the processing time, if the url remains in processing too long, this can be because the worker or the server crashed unexpectedly. so the watcher should reset the task in waiting
Hi!
Thank you for your Post.
I can advice this approach to make your system more resilient and prevent duplication:
Each
url
should have a sate enum [waiting
,processing
,terminated
]The worker should read and update the state inside a
lock
, the lock will prevent that two or more worker take the same job because all of them see it at a same time as available.in rails you can do it by
Timeout
on the processing time, if the url remains inprocessing
too long, this can be because the worker or the server crashed unexpectedly. so the watcher should reset the task inwaiting
Thanks 🙏 This is brilliant!