In the realm of Ruby on Rails and background job processing, Solid Queue is the newest contender. Solid Queue is a robust alternative to well-established solutions like good_job and Sidekiq. This guide explores Solid Queue, delving into its distinctive features and comparing it with its counterparts.
What is Solid Queue?
Solid Queue is an open-source background processing library tailored for Ruby on Rails applications. Its primary objective is to efficiently manage asynchronous tasks, ensuring your application's performance remains rock-solid even during peak traffic and demanding workloads.
Key Features of Solid Queue:
- Reliability: Solid Queue places a premium on reliability and data integrity. It guarantees that jobs are processed precisely once, eliminating the risk of duplicates or data loss.
- Transaction Support: Unlike some competing solutions, Solid Queue seamlessly integrates with database transactions, ensuring consistent data when both your application and background jobs interact.
- Concurrency Control: Solid Queue offers granular control over concurrency, enabling you to specify the number of workers for each job type and optimize resource utilization.
- Active Job Compatibility: If your Rails application relies on Active Job, you'll appreciate Solid Queue's effortless compatibility, simplifying migration and adoption.
How Solid Queue Sets Itself Apart
- Data Integrity: Solid Queue is committed to data integrity, which is critical for applications handling sensitive information or intricate business logic. It guarantees that jobs are processed reliably, eliminating the possibility of duplicates.
- Database Transactions: Integrating Solid Queue with database transactions is a standout feature. In contrast, GoodJob and Sidekiq may necessitate additional configuration through something like Redis and management to maintain transactional consistency.
- Concurrency Control: While all three solutions offer concurrency control, Solid Queue's flexibility in specifying worker counts for different job types empowers you to fine-tune resource allocation for optimal performance.
Examples
To show how to use SolidQueue in a Ruby on Rails application, you'll need to take the following steps on a new Rails application.
Install the gem. Add the following to your Gemfile
and run bundle install
.
gem "solid_queue"
Run the installer
rails generate solid_queue:install
This adds a few migrations, replaces a few configurations in the test,
development,
and production
environment, and adds a solid_queue.yml
config file to the app. All the values are commented out by default.
gsub config/environments/development.rb
gsub config/environments/test.rb
gsub config/environments/production.rb
create config/solid_queue.yml
rails railties:install:migrations FROM=solid_queue
Copied migration 20240102191754_create_solid_queue_tables.solid_queue.rb from solid_queue
The configuration wasn’t added to my development.rb
environment file but made it to the production.rb
file. You may need to append the following to your preferred environment configurations if you aren't adding it to config/application.rb
config.active_job.queue_adapter = :solid_queue
Inside the config/solid_queue.yml
configuration file, be sure to uncomment the defaults.
default: &default
dispatchers:
- polling_interval: 1
batch_size: 500
workers:
- queues: "*"
threads: 5
processes: 1
polling_interval: 0.1
development:
<<: *default
test:
<<: *default
production:
<<: *default
Be sure to configure Solid Queue as your Active Job queue adapter
# config/application.rb
config.active_job.queue_adapter = :solid_queue
Run the migrations
rails db:migrate
This creates several tables related to different states of a given job:
== 20240102191754 CreateSolidQueueTables: migrating ===========================
-- create_table(:solid_queue_jobs)
-> 0.0024s
-- create_table(:solid_queue_scheduled_executions)
-> 0.0009s
-- create_table(:solid_queue_ready_executions)
-> 0.0012s
-- create_table(:solid_queue_claimed_executions)
-> 0.0009s
-- create_table(:solid_queue_blocked_executions)
-> 0.0009s
-- create_table(:solid_queue_failed_executions)
-> 0.0007s
-- create_table(:solid_queue_pauses)
-> 0.0007s
-- create_table(:solid_queue_processes)
-> 0.0009s
-- create_table(:solid_queue_semaphores)
-> 0.0012s
-- add_foreign_key(:solid_queue_blocked_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
-> 0.0062s
-- add_foreign_key(:solid_queue_claimed_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
-> 0.0054s
-- add_foreign_key(:solid_queue_failed_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
-> 0.0050s
-- add_foreign_key(:solid_queue_ready_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
-> 0.0061s
-- add_foreign_key(:solid_queue_scheduled_executions, :solid_queue_jobs, {:column=>:job_id, :on_delete=>:cascade})
-> 0.0053s
== 20240102191754 CreateSolidQueueTables: migrated (0.0381s) ==================
Start Solid Queue’s supervisor to be able to run jobs. This should be a new terminal instance.
bundle exec rake solid_queue:start
Define and enqueue jobs as you would with Active Job:
# app/jobs/sample_job.rb
class SampleJob < ApplicationJob
def perform
# Your job logic here
end
end
# Enqueue the job
SampleJob.perform_later
rails console
> SampleJob.perform_later
Pretty simple, right? I love that it lives within the framework and doesn’t make me learn a complicated new standard. I come from Sidekiq, and the workflow is highly similar, minus a few bits.
Handling Failed jobs and retries
Solid Queue does not have an automatic retry feature; instead, it depends on Active Job for this functionality. Any failed jobs will be stored in the system, and a record of the failed execution will be created in the solid_queue_failed_executions
table. These failed jobs will remain in the system until manually discarded or re-enqueued.
The team plans to introduce a UI component called Mission Control that allows you to examine and diagnose any specific job or group of jobs efficiently.
For now, you can manually intervene with these methods:
failed_execution = SolidQueue::FailedExecution.find(...) # Find the failed execution related to your job
failed_execution.error # inspect the error
failed_execution.retry # This will re-enqueue the job as if it was enqueued for the first time
failed_execution.discard # This will delete the job from the system
This seems nice for the ability to manually retry if necessary in your app’s logic rather than having it be arbitrary. Some might argue the latter.
Adding some UI in the meantime
I noticed on Twitter a library called Panoptic. The simple UI was built with Bootstrap CSS to provide a GUI for Solid Queue.
You can add it much like Sidekiq if you’re familiar with that configuration style. It’s an engine that bolts on the app.
# add to your Gemfile
gem "panoptic"
Then in your config/routes.rb
file, add the following one-liner
# config/routes.rb
Rails.application.routes.draw do
mount Panoptic::Engine => "/panoptic"
# more code
end
You can visit localhost:3000:/panoptic
to see it in action.
Again, the team has stated that something called Mission Control is on the near horizon, so I wouldn't depend heavily on this library. It is something, though!
In pursuit of increased developer happiness with Puma
While running bundle exec rake solid_queue:start
each time you need background job processing, it works fine; a handy Puma plugin integration in Solid Queue allows for some nice automations. When Puma is running (rails server), the processing of your job will kick in alongside it.
I need to add a one-liner to my puma.rb
file.
# config/puma.rb
plugin :solid_queue
Conclusion
Solid Queue is a formidable choice for background job processing in Ruby on Rails applications. It prioritizes reliability, data integrity, and seamless database transaction integration.
While GoodJob and Sidekiq remain excellent alternatives, Solid Queue's unique attributes make it a compelling candidate for your next project without additional dependencies or resources. I think you will need to consider the extra database space it might reserve, but hopefully, it shouldn't be enough to cause too much concern.
Top comments (0)