Replacing Cron Jobs With Active Jobs in Rails

For years - decades, actually - I've been using the rails runner to handle recurring jobs via cron. This works, and the output of the job is emailed to me so I can keep track of what's going on.

But with Rails 8 out and Solid Queue being a contender, it's time to change things up a bit and move the cron jobs to Solid Queue as recurring jobs. But, I don't want to lose the ability to view the output easily. I can capture the output and email it to myself.

Hmm, even better, I can capture the output and keep a log. And with AI tools I can create an agent to watch those logs for me and alert me to something that looks like trouble.

This is going to rock.

This particular project has a rather robust crontab with 16 entries. most of them look something like this:

# new autocomplete stuff at 7:02AM each morning
2 7 * * *   /bin/bash -l -c 'rvm 3.2.3 --quiet && cd $APP_HOME && bundle exec rails runner batch/autocomplete_load.rb'

# Daily CWR ack processing
57 15 * * * /bin/bash -l -c 'rvm 3.2.3 --quiet && cd $APP_HOME && bundle exec rails runner CwrDestination.download_and_process_all_ack_files'

# Weekly batch scripts
15 0 * * 1  /bin/bash -l -c 'rvm 3.2.3 --quiet && cd $APP_HOME && bundle exec rails runner Track.make_renewals'

As you can see, these aren't implemented as ActiveJobs. Instead, most are class methods on my models which implement effectively procedural job code, while others with no obvious home are simply scripts. I can move them to jobs, and likely will at some point. But, for now, I can make these work in Solid Queue as recurring jobs, anyway. The script will be moved to a Job, but the others will simply run as "commands".

Getting Started With Solid Queue

We have to start by adding the Solid Queue gem to our bundle, installing it, then running the installer to set up the initial config files for our application:

bundle add solid_queue
bin/rails solid_queue:install

There are two config files generated - config/queue.yml and config/recurring.yml. Additionally, db/queue_schema.rb is generated, along with the bin/jobs script.

(See the install_generator.rb file for more information)

It also changes your config/environments/production.rb file in two ways. First, it sets config.active_job.queue_adapter to ":solid_queue", and it adds this line as well:

  config.solid_queue.connects_to = { database: { writing: :queue } }

That line assumes that Solid Queue will use a separate database referenced in your config/databases.yml as "queue". The Solid Queue docs show how to modify your file to accommodate this.

I'm using Postgres and it's all on one server, so setting up a separate database doesn't make sense. I simply commented that line out, causing Solid Queue to use the primary database.

An Aside - The Issue When Using SQL Schema Format

As an aside, there's an issue that will arise here if you're going to have a separate database for Solid Queue and you're using the sql schema format. Solid Queue creates the "queue_schema.rb", which is the reference for the database in the config/database.yml file ("queue") plus "_schema.rb".

As the example:

production:
  primary:
    <<: *default
    database: storage/production.sqlite3
  queue:
    <<: *default
    database: storage/production_queue.sqlite3
    migrations_paths: db/queue_migrate

I'm taking this aside to warn you that if you are using the sql schema format the queue_schema.rb file will be ignored when you try to set up the queue database. In that case, you need to turn it into a migration and let the db:setup task create the SQL schema file for you.

Back To Setup

Since I'm using the same database, I won't have a separate schema file and migration directory for Solid Queue. So, I created a migration called add_solid_queue, and for its change method I added all the code from the db/queue_schema.rb file, which I then removed.

At this point, I just have to run bin/rails db:migrate to create the Solid Queue tables.

The queue.yml Config File

The runner allows for a very robust configuration, but I have a fairly simple application. I need three workers:

1. A general worker for sending email and doing quick jobs in the "default" queue
2. A worker for handling audio encoding and importing - I just want to handle one of these at a time
3. A worker to handle the old crontab stuff - again, one at a time

My polling interval is set to 1 second for most of these as there's no reason to beat on the database for stuff that's going to take more time to run.

Because the config/queue.yml and config/recurring.yml files are sectioned by env, I add them to the git repo. Not necessary for what I'm doing, but makes life easier.

Here's my queue.yml file:

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "*"
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: [ "encoding", "import" ]
      threads: 1
      processes: 1
      polling_interval: 1
    - queues: [ "cron" ]
      threads: 1
      processes: 1
      polling_interval: 1

development:
  <<: *default

test:
  <<: *default

production:
  <<: *default

This is probably not useful for you, but I wanted you to see what I've done.

The recurring.yml Config File

Here's where the real fun is: the recurring.yml file. Taking the three items shown from the crontab, here's what I have in recurring.yml:

production:
  autocomplete_load:
    class: AutocompleteLoadJob
    queue: cron
    schedule: 'every day at 7:02am'
  cwr_process_ack_files:
    command: "CwrDestination.download_and_process_all_ack_files"
    queue: cron
    schedule: 'every day at 7:10am'
  make_renewals:
    command: "Track.make_renewals"
    queue: cron
    schedule: 'every monday at 12:15am'

For some items, the "schedule" is straight from the crontab. For instance, I have one that runs at 5:15AM on the first day of each quarter:

15 5 1 1,4,7,10 *   /bin/bash ...

I just put that crontab day/time string directly in the "schedule" key:

  quarterly_term_emails:
    command: "Track.quarterly_term_emails"
    queue: cron
    schedule: '15 5 1 1,4,7,10 *'

Logging

All of my crontab entries just log to stdout, but that means the output will end up in a log file somewhere. That's fine, but ultimately I want something like I had before where each one was emailed to me.

Better than email is to simply log to a database table so I can look back over these later, plus keep track of errors easily.

To do this, I first create a new database table and associated model called JobLog:

class CreateJobLogs < ActiveRecord::Migration[8.0]
  def change
    create_table :job_logs do |t|
      t.string :job_class, null: false, index: true
      t.uuid :job_id, null: false, index: true
      t.text :output
      t.datetime :started_at, null: false
      t.datetime :finished_at, null: false
      t.boolean :success, null: false, default: false, index: true
      t.boolean :needs_attention, null: false, default: false, index: true
      t.text :error_details

      t.timestamps
    end

    add_index :job_logs, :created_at
  end
end

And, the model:

class JobLog < ApplicationRecord
  scope :recent, -> { order(created_at: :desc) }
  scope :for_job, ->(job_class) { where(job_class: job_class) }
  scope :needs_attention, -> { where(needs_attention: true) }
  scope :problematic, -> { where("needs_attention = ? OR success = ?", true, false) }

  def failed?
    !success
  end

  def duration
    (finished_at - started_at).round(2)
  end

  validates :job_class, :job_id, presence: true
end

There are two flags here of interest: success and needs_attention. If the script fails, we can set success to "false" and log the error information. But it's also possible that the script won't "fail" but will find a problem. In that case, we can flag this in code so that it'll be called out to the administrator later. It might make sense to also send an email in these cases, and doing so would be an easy addition.

Of course, I also created a simple view for my admin dashboard where I can see these as they complete.

To do all of this, I created a concern which I can include in my jobs. It lives in app/jobs/concerns/job_logging.rb:

module JobLogging
  extend ActiveSupport::Concern

  included do
    around_perform :capture_job_output
    attr_accessor :current_job_log
    attr_accessor :job_class
  end

  def flag_for_attention(message = nil)
    if message
      puts "\n[NEEDS ATTENTION] #{message}"
    end
    @needs_attention = true
  end

  def alt_job_name(job_name)
    @job_class = job_name
  end

  private

  def capture_job_output
    original_stdout = $stdout
    captured_output = StringIO.new
    start_time = Time.current
    success = true
    error_details = nil
    @needs_attention = false

    custom_writer = Class.new do
      def initialize(string_io)
        @string_io = string_io
      end

      def write(message)
        @string_io.write(message)
        Rails.logger.info(message)
      end

      def close
        @string_io.close
      end
    end

    $stdout = custom_writer.new(captured_output)
    puts "Job started at: #{start_time}"

    begin
      yield
    rescue StandardError => e
      success = false
      error_details = generate_error_details(e)
      puts error_details
      raise e
    ensure
      end_time = Time.current
      duration = (end_time - start_time).round(2)
      puts "\nJob finished at: #{end_time}"
      puts "Total duration: #{duration} seconds"

      $stdout = original_stdout
      output = captured_output.string

      # Save to database
      self.current_job_log = JobLog.create!(
        job_class: @job_class || self.class.name,
        job_id: job_id,
        output: output,
        started_at: start_time,
        finished_at: end_time,
        success: success,
        error_details: error_details,
        needs_attention: @needs_attention
      )
    end
  end

  def generate_error_details(error)
    <<-ERROR_DETAILS

ERROR DETAILS
============
Error class: #{error.class}
Error message: #{error.message}
Backtrace:
#{error.backtrace.take(50).join("\n")}
    ERROR_DETAILS
  end
end

I use the CustomWriter class in here because I want the output to go to the database log as well as the Rails logger. With this, I can create a superclass called "LoggedJob":

class LoggedJob < ApplicationJob
  include JobLogging
end

And then simply use it as the superclass for any jobs to be logged:

class AutocompleteLoadJob < LoggedJob
  queue_as :cron
  ...
end

Handling Commands

We still need to handle the recurring jobs that are "commands". The issue is that they won't be logged to our job_logs table automatically, but it's pretty easy to hook them up.

First, we need to create a new superclass just for those, and it'll in turn be based on the class already used for recurring commands:

class RecurringLoggedJob < SolidQueue::RecurringJob
  include JobLogging

  def perform(*args)
    alt_job_name args.first
    super
  end
end

Note that I'm using alt_job_name - that was added specifically for these jobs. Since the job logger uses the job name by default, all of these end up showing up as "RecurringJob" unless we do something different. In this case, we just take the command that's passed in and log it.

In Production

You'll also have to make this work in production. I'm using Capistrano to deploy, and this simple tutorial from Rob Zolkos shows how to set it up:

https://www.zolkos.com/2024/02/21/how-i-deploy-solid-queue-with-capistrano

If you're using kamal or another deployment method it will be different.

Summary

Moving from old style cron to recurring jobs with Solid Queue is a great way to keep your entire application in one place. I used to have to worry about the crontab when moving servers and such, and I got too many emails every day. This takes care of them. I have also created a simple job to check every day and email me any jobs where "success" is false or "needs_attention" is true.

After I get a few weeks of these I'll set up a simple agent using an LLM that'll view the logs and alert me when something seems off.

Solid Queue also helped me remove the redis dependency, which is important moving forward.

Let me know below if you have questions, or reach out on X.