Leveraging Sidekiq for Higher Throughput

When your app starts to deal with more traffic, it can be useful to look for hot paths in your code to optimise. A great way for some quick wins is to figure out actions that take a (reasonably) long time, but don’t need to happen in-process. These can be things like sending email or pulling data from a third party API.

In the last year or so, I’ve been working on moving such time consuming tasks out of process, to a background worker queue. Note that the time to complete the task doesn’t improve, it just happens in the background, allowing your app to serve the request faster.

Sidekiq is a great option to look at when solving this problem. It allows us to package up potentially long-running logic into a job that can be placed on a queue to be worked off later. I say later, but usually the job is worked off right away, unless your dealing with thousands of jobs at a time.

For example, imagine we have a block of code in our request/response cycle that takes 5 seconds to execute. Normally, that would happen in-process and our response would be delayed. By moving the time consuming code to Sidekiq, we can replace it with a simple call to Sidekiq to queue it, meaning we can get on with sending our response in a much more timely fashion, with the code that takes 5 seconds being executed out of process in the Sidekiq queue.

Sidekiq also gives us great benefits like logging and automatic retries. When a job fails, it is added to the retries queue. When in this queue it will be retried 25 times over the next 21 days. This means if you have a bug, you can deploy a fix and the job will be rerun on the new code. You can keep an eye on all your jobs using the dashboard that comes with Sidekiq.

Let’s take a look at some code. Say we have an application that manages healthcare for a user. In the UK, we have local NHS health boards. For admin purposes, we need to look that up based on a users postcode. Postcodes.io lets us access that information, so we’ll use that API.

class User

  attr_accessor :postcode, :local_health_board

  def populate_healthcare_info
    url = "http://api.postcodes.io/postcodes/#{postcode}"
    begin
      response = HTTParty.get(url)
      local_health_board = response.body[:result][:nhs_ha]
    rescue HTTParty::Error
      puts "Unable to reach Postcode API"
    end
  end
end

Once we have a User, at some point after creation, we call populate_healthcare_info. This seems innocent enough, but what happens if this API is slow or completely unreachable? It’s not critical that we get data we need right away, so we can move this to a background worker. Let’s look at how to write a worker class for Sidekiq.

module Workers
  class FetchHealthBoardInfo
    include Sidekiq::Worker

    def perform(user_id)
      user = User.find(user_id)
      user.populate_healthcare_info
    end
  end
end

Worker classes must include the Sidekiq::Worker class and implement a perform method to work properly. Sidekiq works best when you pass primitive arguments to the worker, rather than rich objects. This is both to prevent any issues when Sidekiq serialises arguments and also to reduce memory footprint as Sidekiq stores arguments on the queue until the job is worked off. In the perform method we fetch the user and call populate_healthcare_info to pull in that information from the API.

Just to point out another way of doing this, Sidekiq provides extensions to class methods in Ruby, allowing us to simply delay them. If populate_healthcare_info was a class method, we could simply call User.delay.populate_healthcare_info(1) and the job would be queued for us. Note in this example, we’d have to change the implementation of populate_healthcare_info to handle this.

We can call this with Workers::FetchHealthBoardInfo(1), with 1 being the ID of the User we want to update. Sidekiq will queue and work off the job in the background, leaving your app free to respond to the user without delay.

Sidekiq isn’t limited to small tasks, I’ve used it successfully in the past to queue hundreds of thousands of jobs at a time and I’m always surprised how quickly it powers through them. To see more of it’s features in depth, the creator of Sidekiq has created this handy YouTube playlist, that covers the basics right up to more advanced features.

First published at https://markprovan.com/leveraging-sidekiq-for-higher-throughput/