DEV Community

Cover image for Moving to solid_queue from sidekiq
Zain Butt
Zain Butt

Posted on

Moving to solid_queue from sidekiq

A little bit about me, I am an engineer turned indiehacker for the first time and working on my startup devtree. I started working on it in July this year and growing it slowly.

Its a simple Ruby on Rails application running on Flyio backed by Postgres and background jobs running with Sidekiq and redis.

The problem

I use Upstash serverless Redis in my app that is provided by Flyio with their tight integration with Upstash.

Upstash pricing is per request based, which was different for apps running on flyio as they were providing a free tier based on memory restrictions which worked for me.

But last month they decided to unify their pricing everywhere after new year on 1st Jan. Based on my setup with sidekiq, it sends gazillion request per hour to check for new jobs which eats up their free quota of 10k requests per day.

upstash redis

As much I like sidekiq, I can't bear the redis costs when I am not making any money.


I was looking for alternative solutions specifically non-redis based. I had a few suggestions from my network which included solid_queue which surprisingly I forgot about.

Its a non-redis solution, works with postgres and I wouldn't spend weeks implementing it. So, I decided to give it a try.

Here's the process I followed:

1 - Create a new branch
2 - Add the gem

gem 'solid_queue'

Enter fullscreen mode Exit fullscreen mode

3 - Change the active job adapter

config.active_job.queue_adapter = :solid_queue

Enter fullscreen mode Exit fullscreen mode

4 - Install solid queue

rails generate solid_queue:install

Enter fullscreen mode Exit fullscreen mode

5 - Run migrations for job tables

rails solid_queue:install:migrations
rails db:migrate

Enter fullscreen mode Exit fullscreen mode

6 - Run the worker

bundle exec rake solid_queue:start
# or add it in
# solid: bundle exec rake solid_queue:start

Enter fullscreen mode Exit fullscreen mode

Now you can create your first job

rails g job my_job

# app/jobs/my_job.rb

class MyJob < ApplicationJob
  self.queue_adapter = :solid_queue
  # ...

Enter fullscreen mode Exit fullscreen mode


Solid queue will create a few tables for you to store your jobs but focus on 2 for now.
solid_queue_scheduled_executions and solid_queue_ready_executions

You can guess from their names, one is used to store jobs when they are to be run in the future.

The other is to store jobs which need immediate execution.


Dispatchers are in charge of selecting jobs scheduled to run in the future that are due and dispatching them, which is simply moving them from the solid_queue_scheduled_executions table over to the solid_queue_ready_executions table so that workers can pick them up. They also do some maintenance work related to concurrency controls.


Workers are in charge of picking jobs ready to run from queues and processing them. They work off the solid_queue_ready_executions table


You can also configure the polling interval, number of workers based on your requirements by using a yaml configuration file.

add a configuration file in config/solid_queue.yml:

    - polling_interval: 5 #in seconds
      batch_size: 100
    - queues: '*'
      threads: 5
      processes: 1
      polling_interval: 5 #in seconds
Enter fullscreen mode Exit fullscreen mode

With this configuration, it will spin 5 worker threads to process the jobs every 5 seconds.

There are tons of other job level and queue level configurations that you can add for better control over concurrency, uniqueness and queue management that you can find here.


I have been monitoring my app for a few days and unfortunately it has crashed a few times due out of memory which never happened before, seems like 512mb ram is not enough 🤷.

I haven't debugged the issue for now but will look into it for the root cause.

This is it form side, let me know if you would like to know more about the technical side of my startup.
I share about my startup publicly on twitter and linkedin.

Top comments (4)

katafrakt profile image
Paweł Świątkowski

Do you have the same amount of SolidQueue workers now as you had Sidekiq workers before? That would be an interesting find that it uses significantly more memory compared to Sidekiq.

zainbutt profile image
Zain Butt • Edited

With Sidekiq, I had 1 Process, which could execute hundreds of job threads sometime without any issues.
with solid queue, I have set it to have 5 workers. But One process.

katafrakt profile image
Paweł Świątkowski

Reading the docs now, I understand that even if you configure just one worker process you still end up with a supervisor process and dispatcher process next to it. So it's at least 3 processes for SolidQueue itself. While hopefully only worker process loads the whole Rails application, it is still understandable that extra two processes cause some memory overhead.

Thread Thread
zainbutt profile image
Zain Butt

I reduced the worker threads to 2. So, now in total it would have 4 processes. working fine for now.