Article orginally published here : https://www.bootrails.com/blog/rails-sidekiq-tutorial/
Sidekiq allows Rails to launch any task in the background. Let's see how, from zero to production.
Motivation
At BootrAils, we are trying to keep the stack as simple as possible.
However, the need for the existence of background jobs arises very quickly when you create a new Rails application for business purposes. Think about the "forgot password ?" feature. An email needs to be sent. deliver_later is very recommended, and this method needs "background jobs" to work properly.
And you can't do that easily, neither in Ruby nor Ruby-on-Rails.
Rails Q: handling a request, is there a good utility or idiom to run a block of code asynchronously so it doesn't block the response but also doesn't incur the overhead of a job/queue system?
— Justin Searls (@searls) January 18, 2022
do_later { fire_and_forget() }
Just have a one-off call & don't want the user to wait.
In theory, ActiveJob (in charge of background jobs, as the name implies) is already included in every default Rails app.
But... the set up, dependencies, as well as production, test and development settings are entirely left to the developer.
There are two giants in this space : delayed_jobs (sometimes known as "DJ"), and Sidekiq. Sidekiq is more well-known, maintained, and documented, I would advise here to choose Sidekiq for any new Rails application - YMMV.
This tutorial is here to remove fear about installing background jobs into an application whose primary purpose is meant to be "just MCV".
Prerequisites
Here are the tools we will use in this tutorial :
$> ruby -v
ruby 3.1.0p0 // you need at least version 3 here
$> bundle -v
Bundler version 2.2.11
$> npm -v
8.3.0 // you need at least version 7.1 here
$> yarn -v
1.22.10
$> psql --version
psql (PostgreSQL) 13.1 // let's use a production-ready database locally
$> redis-cli ping // redis is a dependency of Sidekiq
PONG
$> foreman -v
0.87.2
Create a fresh new Rails app - without Sidekiq first
mkdir sidekiqrails && cd sidekiqrails
echo "source 'https://rubygems.org'" > Gemfile
echo "gem 'rails', '7.0.1'" >> Gemfile
bundle install
bundle exec rails new . --force -d=postgresql --minimal
# Create a default controller
echo "class WelcomeController < ApplicationController" > app/controllers/welcome_controller.rb
echo "end" >> app/controllers/welcome_controller.rb
# Create a default route
echo "Rails.application.routes.draw do" > config/routes.rb
echo ' get "welcome/index"' >> config/routes.rb
echo ' root to: "welcome#index"' >> config/routes.rb
echo 'end' >> config/routes.rb
# Create a default view
mkdir app/views/welcome
echo '<h1>This is h1 title</h1>' > app/views/welcome/index.html.erb
# Create database and schema.rb
bin/rails db:create
bin/rails db:migrate
Then open application.rb and uncomment line 6 as follow :
# inside config/application.rb
require_relative "boot"
require "rails"
# Pick the frameworks you want:
require "active_model/railtie"
require "active_job/railtie" # <== Uncomment
# ... everything else remains the same
Then create the parent Class of all jobs :
# inside app/jobs/application_job.rb
class ApplicationJob < ActiveJob::Base
end
Side note We created the Rails app with the --minimal flag, so that you can discover clearly what is needed to run a job. With the default install, active_job/railtie is already uncommented, and the parent Class of all jobs already exists.
Add redis and sidekiq gems to your Rails app
Open your Gemfile, and add at the very bottom
gem 'redis'
gem 'sidekiq'
And in your terminal, run
bundle install
Then add the following line inside application.rb
# inside config/application.rb
# ...
class Application < Rails::Application
config.active_job.queue_adapter = :sidekiq
# ...
Create a hello world job
Create a new file under app/jobs/hello_world_job.rb
# inside app/jobs/hello_world_job.rb
class HelloWorldJob < ApplicationJob
queue_as :default
def perform(*args)
# Simulates a long, time-consuming task
sleep 5
# Will display current time, milliseconds included
p "hello from HelloWorldJob #{Time.now().strftime('%F - %H:%M:%S.%L')}"
end
end
Nothing fancy. The purpose of this job is to be run asynchronously when called. Whereas the classic HTTP requests/responses will be handled inside the only available Thread. However any to HelloWorldJob can be initially triggered inside an HTTP request. Let's see how.
Call the hello world job from the view
Modify routes.rb as follow :
# inside config/routes.rb
Rails.application.routes.draw do
get "welcome/index"
# route where any visitor require the helloWorldJob to be triggered
post "welcome/trigger_job"
# where visitor are redirected once job has been called
get "other/job_done"
root to: "welcome#index"
end
Create app/controllers/other_controller.rb
# inside app/controllers/other_controller.rb
class OtherController < ApplicationController
def job_done
end
end
Create app/views/other/job_done.html.erb
<!-- inside app/views/other/job_done.html.erb -->
<h1>Job was called</h1>
Now our job is ready to be called from the initial view :
<%# inside app/views/welcome/index.html.erb %>
<h1>This is h1 title</h1>
<%= form_with url: welcome_trigger_job_path do |f| %>
<%= f.submit 'Launch job' %>
<% end %>
And the call will happen inside the controller, like this :
# inside app/controllers/welcome_controller.rb
class WelcomeController < ApplicationController
def trigger_job
HelloWorldJob.perform_later
redirect_to other_job_done_path
end
end
Add a Procfile.dev
You probably want to try everything right now, but wait a minute : our jobs must be sent from another thread. This is done locally by adding a Procfile.dev file at the root of our project :
web: bin/rails s
worker: bundle exec sidekiq -C config/sidekiq.yml
Add config/sidekiq.yml
# inside config/sidekiq.yml
development:
:concurrency: 5
production:
:concurrency: 10
:max_retries: 1
:queues:
- default
And finally add the sidekiq initializer here : config/initializers/sidekiq.rb
# inside config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
end
Sidekiq.configure_client do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
end
redis://localhost:6379/1 is the default URL of a locally installed redis database. If it's not available here, REDIS_URL is the environment variable that will come to the rescue.
Launch local app
Now it's time to see if everything is going well.
Launch your local server like this :
foreman start -f Procfile.dev
Let's see what is displayed locally :
Click on the "Launch job" button, and immediately display logs in your terminal :
18:43:31 web.1 | Started POST "/welcome/trigger_job" for ::1 at 2022-01-22 18:43:31 +0100
18:43:31 web.1 | Processing by WelcomeController#trigger_job as HTML
18:43:31 web.1 | [ActiveJob] Enqueued HelloWorldJob (Job ID: 9b9aa325-d118-45ff-a74b-99cad73ecda8) to Sidekiq(default)
18:43:31 web.1 | Redirected to http://localhost:5000/other/job_done
18:43:31 web.1 |
18:43:31 web.1 | Started GET "/other/job_done" for ::1 at 2022-01-22 18:43:31 +0100
18:43:31 web.1 | Completed 200 OK in 4ms (Views: 3.3ms | ActiveRecord: 0.0ms | Allocations: 2644)
18:43:31 web.1 |
18:43:31 worker.1 | 2022-01-22T17:43:31.776Z pid=3002 tid=3ze class=HelloWorldJob jid=166d945e2b7cd15e37addc7c INFO: Performing HelloWorldJob (Job ID: 9b9aa325-d118-45ff-a74b-99cad73ecda8) from Sidekiq(default) enqueued at 2022-01-22T17:43:31Z
18:43:36 worker.1 | "hello from HelloWorldJob 2022-01-22 - 18:43:36.784"
18:43:36 worker.1 | 2022-01-22T17:43:36.785Z pid=3002 tid=3ze class=HelloWorldJob jid=166d945e2b7cd15e37addc7c INFO: Performed HelloWorldJob (Job ID: 9b9aa325-d118-45ff-a74b-99cad73ecda8) from Sidekiq(default) in 5008.15ms
18:43:36 worker.1 | 2022-01-22T17:43:36.786Z pid=3002 tid=3ze class=HelloWorldJob jid=166d945e2b7cd15e37addc7c elapsed=5.47 INFO: done
We copy/pasted here a simplified version of the logs, so that you can see what is actually happening. The GET "/other/job_done" is triggered immediately, at 18:43:31; whereas the job is actually called 5 seconds later, at 18:43:36. This is what is called "background jobs".
Push Rails and Sidekiq to production
Follow step below :
# heroku is connected
heroku login
heroku create
# This will modify local files
echo "web: bundle exec puma -C config/puma.rb" > Procfile
echo "worker: bundle exec sidekiq -e production -C config/sidekiq.yml" >> Procfile
bundle lock --add-platform x86_64-linux
# This will modify you heroku app
heroku addons:create heroku-postgresql:hobby-dev
heroku addons:create heroku-redis:hobby-dev
heroku buildpacks:add heroku/ruby
git add . && git commit -m 'ready for prod'
git push heroku main
# app works, but worker (for background jobs) is missing
heroku ps:scale worker=1
So far everything should work properly.
Wait 1 minute (maximum 2), so that every Heroku dependency is properly set up.
Open the application in your browser (the URL should contain heroku.com, so we're not trying localhost this time).
Click on the "launch" button again, and check your logs by typing
heroku logs
Can you see the worker that correctly prints "Hello world" ? Sidekiq works !
A last word (of caution)
Background jobs tend to swallow exceptions without any complaint. Failed to connect to the Redis database ? It won't tell you. Bug appeared ? May not be shown. So our advice would be to make sure that everything is working properly by keeping the HelloWorldJob available (to quickly manually check if Sidekiq is working), and to install the Sidekiq web UI right from the start (not covered in this tutorial).
Top comments (0)