Moving to solid_queue from sidekiq

A little bit about me, I am an engineer turned indiehacker for the first time and working on my startup devtree. I started working on it in July this year and growing it slowly.

Its a simple Ruby on Rails application running on Flyio backed by Postgres and background jobs running with Sidekiq and redis.

The problem

I use Upstash serverless Redis in my app that is provided by Flyio with their tight integration with Upstash.

Upstash pricing is per request based, which was different for apps running on flyio as they were providing a free tier based on memory restrictions which worked for me.

But last month they decided to unify their pricing everywhere after new year on 1st Jan. Based on my setup with sidekiq, it sends gazillion request per hour to check for new jobs which eats up their free quota of 10k requests per day.

As much I like sidekiq, I can't bear the redis costs when I am not making any money.

Solution

I was looking for alternative solutions specifically non-redis based. I had a few suggestions from my network which included solid_queue which surprisingly I forgot about.

Its a non-redis solution, works with postgres and I wouldn't spend weeks implementing it. So, I decided to give it a try.

Here's the process I followed:

1 - Create a new branch
2 - Add the gem

gem 'solid_queue'

3 - Change the active job adapter

config.active_job.queue_adapter = :solid_queue

4 - Install solid queue

rails generate solid_queue:install

5 - Run migrations for job tables

rails solid_queue:install:migrations
rails db:migrate

6 - Run the worker

bundle exec rake solid_queue:start
# or add it in Procfile.dev
# solid: bundle exec rake solid_queue:start

Now you can create your first job

rails g job my_job

# app/jobs/my_job.rb

class MyJob < ApplicationJob
  self.queue_adapter = :solid_queue
  # ...
end

Concepts

Solid queue will create a few tables for you to store your jobs but focus on 2 for now.
solid_queue_scheduled_executions and solid_queue_ready_executions

You can guess from their names, one is used to store jobs when they are to be run in the future.

The other is to store jobs which need immediate execution.

Dispatchers

Dispatchers are in charge of selecting jobs scheduled to run in the future that are due and dispatching them, which is simply moving them from the solid_queue_scheduled_executions table over to the solid_queue_ready_executions table so that workers can pick them up. They also do some maintenance work related to concurrency controls.

Workers

Workers are in charge of picking jobs ready to run from queues and processing them. They work off the solid_queue_ready_executions table

Configuration

You can also configure the polling interval, number of workers based on your requirements by using a yaml configuration file.

add a configuration file in config/solid_queue.yml:

development:
  dispatchers:
    - polling_interval: 5 #in seconds
      batch_size: 100
  workers:
    - queues: '*'
      threads: 5
      processes: 1
      polling_interval: 5 #in seconds

With this configuration, it will spin 5 worker threads to process the jobs every 5 seconds.

There are tons of other job level and queue level configurations that you can add for better control over concurrency, uniqueness and queue management that you can find here.

Caveat

I have been monitoring my app for a few days and unfortunately it has crashed a few times due out of memory which never happened before, seems like 512mb ram is not enough 🤷.

I haven't debugged the issue for now but will look into it for the root cause.

This is it form side, let me know if you would like to know more about the technical side of my startup.
I share about my startup publicly on twitter and linkedin.