Background job processing with Marten and Mosquito

Marten is a web framework written in Crystal that makes building web applications easy and enjoyable. Mosquito is a background task runner for Crystal that uses Redis and that makes it easy to schedule and run tasks asynchronously.

Combining Marten with Mosquito can be a powerful way to build web applications with background tasks that run asynchronously. In this blog post, we'll show you how to get started with using Marten and Mosquito together.

Installation

Assuming that you already have a working Marten project that was initialized with the built-in authentication at hand, you can start by adding Mosquito to your shard.yml file:

dependencies:
  mosquito:
    github: mosquito-cr/mosquito

Then run shards install to install the new dependencies.

We can now ensure that Mosquito is properly required by our project. To do so, let's update the content of the src/project.cr file as follows:

# Third party requirements.
require "marten"
require "mosquito"
# other requirements...

# Project requirements.
require "./jobs/**"

# other requirements...

You'll notice that we require both mosquito and the files of a jobs folder, where we'll define the actual jobs.

Now we can create a new config/initializers/mosquito.cr initializer file where the actual Mosquito configuration will live. We won't cover all the Mosquito configuration options in this article, but this initializer could simply set the right Redis URL to use for now and could look something like this:

Mosquito.configure do |settings|
  settings.redis_url = (ENV["REDIS_URL"]? || "redis://localhost:6379")
end

Finally, we need to create a new file where we will (i) setup Marten itself and (ii) start the Mosquito runner. In this light, let's create a new src/worker.cr file with the following content:

require "./project"

Marten.setup

Mosquito::Runner.start

Defining jobs

For the purpose of this article, we will be defining jobs under a src/jobs folder. If your Marten project is split into multiple applications, you can also decide to create a jobs folder inside each of these applications.

So let's create a src/jobs folder for now and let's define a send_welcome_email_job.cr file in it with the following content:

class SendWelcomeEmailJob < Mosquito::QueuedJob
  params(user_id : Int64)

  def perform
    user = Auth::User.get!(id: user_id)
    user.send_welcome_email
    user.update!(welcome_email_sent: true)
  end
end

This example job simply retrieves a User model record using a user_id job parameter, sends an email by calling a hypothetical #send_welcome_email method, and updates the considered record by setting a hypothetical welcome_email_sent field to true.

Enqueuing jobs

Enqueuing a job is as simple as initializing an instance of a job class with the specified parameters and calling the #enqueue method. The job above was all about sending a welcome email to newly created users. As such, this job could be enqueued from a callback that we could define in the Auth::User model. For example:

module Auth
  class User < MartenAuth::User
    after_commit :trigger_welcome_email_sending, on: :create

    private def trigger_welcome_email_sending
      SendWelcomeEmailJob.new(user_id: id!.to_i64).enqueue
    end
  end
end

In this example, we are triggering the job from a model callback but it should be noted that you can enqueue jobs from pretty much anywhere in your Marten project codebase (eg. handlers, schemas, etc).

Running the Mosquito worker

As you might have guessed, the Mosquito worker runs in a dedicated process. This means that we have to explicitly start the runner we defined earlier by compiling and executing the src/worker.cr file. While in development, we can simply leverage the crystal run command to do so:

crystal run src/worker.cr

A few tips regarding deployments

At deployment time, it's likely that you already compile two binaries: your project's server (src/server.cr) and your project's management CLI (manage.cr). In addition to these, you will now need to ensure that your worker is compiled as well. This can be accomplished by relying on the crystal build command:

crystal build src/worker.cr -o bin/worker --release

You could also add a target to your shard.yml file and even trigger the execution of your compiled worker from a Procfile file depending on your deployment strategy. You can learn more about deploying Marten in the dedicated documentation.

Conclusion

In this blog post, we've gone through the steps of setting up Mosquito with Marten, including installation, configuration, and usage. Using Mosquito with the Marten web framework can help you build fast and scalable web applications that leverage background job processing capabilities.