I'm working on a message queueing system in a Rails application and trying to figure out whether to use delayed_job or sidekiq. Reading articles online it seems sidekiq is better performant for the job but wondering if for my use case if delayed_job should be considered.
Basically it's a message system that will have messages that can be immediately delivered or could be scheduled for a future date. Should also be able to edit the queued message as well as delete them from the queue.
This application will run on Heroku if there are any technical considerations around that choice.
Considering AWS' SQS as well but thinking it'll be more overhead to go down this route.
Top comments (8)
I have never used Delayed Job but I did recently move all of our background jobs to Sidekiq and here is an article I wrote about why. We have been switched over for almost a year and no regrets yet! Actually lots of "thank god we did this" moments.
Switching From Resque to Sidekiq
Molly Struve (she/her) γ» Jun 26 '19 γ» 7 min read
Thanks Molly for sharing your experiences on switching to Sidekiq. Was insightful.
Not exactly an answer here but if you rely on Rail's ActiveJob as your job interface/adapter, you can easily swap between delayed_job and sidekiq in the future. My vote is for sidekiq, but if you start off right be depending on ActiveJob, there's no wrong answer :)
I really like this approach in making it future proof in case we need to swap infrastructure that it'll be an easier swap. Thanks Mac!
In a previous gig I switched a system from Delayed job to Resque and then to Sidekiq (we were wary about switcthing to Sidekiq directly because we had some thread safety issues that were resolved).
My personal suggestion: if you use Rails you can distance yourself a bit from the underlying system with ActiveJob.
Regarding charateristics: DJ stores the queue in the DB and to use it or not to use it really depends on which kind of job consuming patterns you have.
We were building a push notification delivery system for enteprise customers which meant that in moments we had 0 jobs on the queues and in others 300 thousand to process really quickly. If your jobs are time sensitive (as push notifications are) DJ is a bad idea.
DEV runs on delayed job even, the reason why it works it's because those jobs don't have to be completed nearly simultaneously.
Resque and Sidekiq (see @molly_struve's post about it) are far better but they enlarge the surface by introducing Redis in your toolbox, which in some cases might be more than you want.
I personally think it's worth learning about Redis (and you can use it as a SaaS) in general.
Sidekiq is also quite faster than Resque (Rails is not great at multi processing concurrency) and faster than DJ.
I remember how happy we were when, after switching to Sidekiq, we went to the PMs and said: "well, now we can send those notifications literally 30 times faster"
So my advise is: start with delayed job and if that's enough for your needs stay there, otherwise migrate to sidekiq
Don't forget to configure Redis for persistency, you don't want to lose your queues in case of restart ;-)
By DelayedJob do you mean DelayedJob backed by ActiveRecord/DB? I assume so from how the question is worded.
I'm a big fan of DB backed queues for most situations, especially to start out. One big benefit is you don't need additional infrastructure, assuming you are already using a DB.
DB backed queues can definitely have scaling issues, but depending on the size of your queue and the size of your DB you can put off that issue for almost ever. For example at my job we average over 4 million jobs a day via our DB backed queue.
In general I think DB backed queues are my default, unless the specifics of the project call for something else!
Thanks for your insights Corey. That's correct, delayed_job in using ActiveRecord and DB based.
I've used DelayedJob in a customer project, but, we moved to Sidekiq. DelayedJob is simple and you don't need the Redis, but today is common uses Redis. I prefer the Sidekiq.