Sidekiq Tutorial & Overview: Is Sidekiq For Ruby Still Worth Working With After 10 Years?

Sherlock Holmes needs Dr. Watson. Ruby programming needs Sidekiq. Or Delayed_job, or Resque?

Sometimes our clients are interested in what technologies we are using to build their web application projects. We are happy to share, although it’s not easy to lecture people on technologies and not to be a total bore. So, prepare yourselves, that time of year has come again.

All because Sidekiq, probably the most common background job system for Ruby, just turned 10 years old! This overview will be useful for those who are starting out with Ruby on Rails programming, and those who are picky about their technology stack. We provide tutorials on how to start Sidekiq, schedule and test Sidekiq workers, send emails, and more.

What is Sidekiq?

Bruce Lee against Chuck Norris via Giphy

Sidekiq is a free and open-source Ruby job scheduler that was created by Mike Perham. Ruby programming language is one of our most used and well-mastered tools we respect and love. If you’re interested in why we recommend it for clients’ projects, check out our separate article on Ruby. If you don’t have that spare 10 minutes of your life to get to know our inner development processes (we totally understand), then let’s round it up in 4 sentences and introduce you to the major topic of this article:

1. Ruby is a programming language.

2. When developing a Ruby on Rails application, you might find yourself overwhelmed with myriads of tasks that must be executed at the same time.

3. For example, sending emails, charging credit cards, interacting with external APIs, and other data processing.

4. Developers use tools like Sidekiq that allow them to run background jobs, i.e., automated processes outside of the request-response cycle so users don’t have to wait in a queue.

So, Sidekiq is a default tool for a Ruby application that improves its performance and response wait time.

Enterprise version

While Sidekiq is free by default, it doesn’t schedule jobs and only executes them. The paid version of the tool — Enterprise version — comes with scheduling. Mark Perham, its creator, didn’t aim to divide the tool into paid and free versions. Rather, this idea came to him accidentally.

Redis

Sidekiq works closely with Redis 4.0+ to organize job queues an d store statistics. Redis (stands for Remote Dictionary Server) is a fast open source in-memory key-value data store. It provides sub-millisecond response times and allows real-time applications to run millions of requests per second.

Workers and queues

The scheduler is run as the copy of the application in a separate process and can be configured to use several worker threads. Each worker is assigned to one or many queues. Queues represent the list of tasks in the order they are added and can be named by purpose (default, mail, reports) or by priority (low, normal, high) — it’s completely up to the developer.

Sidekick 6.3, the latest version of the tool, came out in November 2021. Introducing the new version, Mark Perham elaborates on how to use the term “worker” correctly. What exactly is a worker?

“Are you talking about a process? A thread? A type of job? I encourage developers to stop using the term worker and use include Sidekiq::Job in your job classes” — M.P.

class SomeJob
  include Sidekiq::Job
  sidekiq_options ...

  def perform(*args)
  end
end

Plugins

Sidekiq has a myriad of prominent features: retries with configurable periods, flexible error handling, a web UI and a ton of plugins extending its functionality further.

One of such plugins, for example, introduces repeated tasks on schedules, which were historically solved by tools based on cron. Cron is harder to configure and takes time to start the whole app for execution of a single task.

Another plugin monitors various metrics and checks for Sidekiq to avoid errors. You can find many more on GitHub!

How did Sidekiq get so popular?

Single, married, multi-threading? Image by @dwfries on Medium

In 2012, Mike Perham was the first to introduce a multi-threading background job system. Resque is multi-process and single-thread. Delayed_job can work ‌for single thread processing only. Sidekiq was the first multi-threading tool in the Ruby community. With time, it became the main driver of avoiding thread safety bugs in the Rails ecosystem. Because Sidekiq is omnipresent, it has stabilized the thread safety of Rails.

Who uses Sidekiq?

Sidekiq has become the de facto standard for web applications these days. According to Stackshare, Sidekiq is used by such well-known companies as:

• Adidas,
• GitLab,
• Product Hunt,
• Send Grid,
• New Relic,
• 500px,
• and many more. A pick into companies using Sidekiq via Stackshare

How does Sidekiq compare to other similar tools?

There are alternative projects with more or less the same set of features. In this article, we compare Sidekiq to its most popular competitors: Resque and Delayed_job.

Resque

Resque is a Ruby library for creating background jobs. You can place them on multiple queues and process them later. Background jobs can be any Ruby class or module. Existing classes can be converted to background jobs. Resque also offers an option to create new classes.

Resque has a slightly different concurrency model from Sidekiq, which may or may not be what you are looking for. While Sidekiq uses threads for its workers (and can’t scale onto many CPU cores), Resque uses processes, which makes it slightly more heavy-weight on one side, but lets it use cores of CPUs. This is important for computationally heavy tasks when the single-CPU threading model doesn’t apply well.

Sidekiq vs Resque statistic on stackshare

Unlike Sidekiq, Resque doesn’t require thread safety and works with any gem (Ruby programs and libraries). According to developers though, Sidekiq runs faster and uses much less memory.

Delayed_job

Delayed_job, a.k.a. DJ is a priority queue, database-oriented tool that allows to execute background tasks asynchronously. It was originally extracted from Shopify and has become very popular through the years.

According to Doug Breaker, who researched DJ in comparison to Sidekiq, there are several features the latter doesn’t have:

• Easy integration with Rails that doesn’t depend on a relational database/non-relational database.
• Custom data store. You can use the main data store to process tasks offline and customize a data store to fit your needs.
• No need to run multiple dependencies at once, e.g., instead of requiring multiple services and dependencies to run a program, you can limit the dependencies that you need. Delayed:job, sadly, doesn’t support multi-threading, unlike Sidekiq. The latter also runs faster, is scalable, and has the ability to run Redis automatically. Sidekiq vs Delayed_job statistic on stackshare

Using Sidekiq in plain Ruby

Now that you know of the alternatives, let’s learn how to use Sidekiq in practice.

Sidekiq is framework-agnostic and can be used in any Ruby application. The official Getting Started guide provides detailed steps on how to get up and running.

The gist is that you:

1. Add Sidekiq to your Gemfile:

# Gemfile

gem 'sidekiq'

1. Define worker classes:

# app/workers/generate_report_worker.rb

class GenerateReportWorker
  include Sidekiq::Worker

  def perform(user_id)
    # do your reporting here
  end
end

1. Start Sidekiq as a separate process:

$ bundle exec sidekiq

1. Start scheduling your workers:

# Task is to be executed as soon as possible
GenerateReportWorker.perform_async(user.id)

# Task is to be executed in 5 minutes
GenerateReportWorker.perform_in(5 * 60, user.id)

# Task is to be executed at a certain moment (3 hours from now)
GenerateReportWorker.perform_at(Time.now + 10800, user.id

1. To execute a worker immediately:

MySimpleWorker.new.perform("I was performed!")

Using Sidekiq in Ruby on Rails

There is not much difference between using Sidekiq in plain old Ruby or in Ruby on Rails programming. You still get a nice integration with Rails Active Job framework and you can use Date/Time helpers when scheduling future tasks:

Date/Time helpers examples

# Generate report next Monday
GenerateReportWorker.perform_at(Time.current.next_week, user.id)

E.g., generate a report in 5 minutes:

GenerateReportWorker.perform_in(5.minutes, user.id)

Active Job integration

Active Job is a standard interface for interacting with job runners. According to the official guide, it is a framework with a variety of queuing back-ends. The jobs could be:

• Regularly scheduled clean-ups,
• Billing charges,
• Emails,
• Anything you can imagine running in parallel. If you want your workers to not be Sidekiq-specific, you need to take several steps:

1. Define your workers as ActiveJob jobs:

# app/jobs/generate_report_job.rb

class GenerateReportJob < ActiveJob::Base
  # The name of the queue to put this job into
  queue_as :default

  def perform(user_id)
    # do your reporting here
  end
end

1. Configure Rails application to use the correct adapter:

# config/application.rb

class Application < Rails::Application
  # ...
  config.active_job.queue_adapter = :sidekiq
end

1. Use ActiveJob syntax for scheduling:

# Generate report as soon as possible
GenerateReportJob.perform_later(user.id)

There are plenty of configuration options.

Sending emails with ActionMailer and Sidekiq

ActionMailer allows you to send emails (surprise!) from your Ruby app. Here is what you need to send emails asynchronously with Sidekiq:

1. For creating a mailer:

# app/mailers/users_mailer.rb

class UsersMailer < ActionMailer::Base
  def welcome_email(user_id)
    @user = User.find(user_id)

    mail(to: @user.email, subject: "Welcome") do |format|
      format.text
      format.html
    end
  end
end

1. See views on emails:

app/views/users_mailer/welcome_email.html.erb - HTML version
app/views/users_mailer/welcome_email.text.erb - TEXT version

1. Finally, sending emails:

user = User.find(1)

mail = UsersMailer.welcome_email(user.id)
# mail.deliver_now
mail.deliver_later

Testing Sidekiq jobs (with RSpec framework)

Sidekiq provides tools for testing the various aspects of your workers at any stage of their lifecycle. To test workers directly, use:

worker = MyWorker.new
worker.perform(:my_arg)

Or you can use an inline mode:

# implementation
class DeleteFromRemoteApiWorker
  include Sidekiq::Worker

  def perform(item_ids)
    ApiWrapper.delete_items(item_ids) # dependency
  end
end


# test
describe DeleteFromRemoteApiWorker do
  let(:items) do
    # ...
  end

  it "delegates the work to the API wrapper as expected" do
    allow(ApiWrapper).to receive(:delete_items)
    item_ids = items.map(&:id)

    described_class.perform_async(item_ids)


    expect(ApiWrapper).to(
      have_received(:delete_items).with(item_ids)
    )
  end
end

Why do we (still) use Sidekiq in outsourced projects?

Mastering Sidekiq is not something you’ll need a triple digits IQ for. Nevertheless, ‌stable and reliable software is a must for processing thousands of workers. This tool uses exclusively Redis as its database, which might not suit all. Memory leaks are also a thing to be aware of…

Nevertheless, Sidekiq architecture is perfect when working with complex Ruby applications. For example, we created a virtual learning environment complete with live chats, video streaming, billing charges, and more, with Sidekiq in our tech stack.

Sidekiq is ideal for those who require fast speed, executing 7100 jobs per second. Multi-threading abilities mean that multiple jobs can be queued in the background without affecting the synchronous workflow of the application, improving overall performance.

So, is Sidekiq still our best buddy? Yes! Happy birthday, Sidekiq! 🎂

Image via Giphy