Background Jobs with Lucky

As with a lot of web applications and APIs, there comes a point where you need to process some tasks in the background. To do this, you'll usually use a background job processor. This helps to make some things asynchronous and speed along the users of your app.

An example of this might be logging when your users login to your site. Maybe you store the IP address they used, when they logged in, how many attempts it took to sign in, and the device they used. If you did all this in-line, then the process would be to track all that info, and the user has to wait while you create this new record in your database before they can actually login. Or, you push that info to a background job, and log the user in. Then you can update that record at your own pace since there's nothing relying on the data.

If you're new to Crystal Lang and/or you're new to Lucky, but you're coming from another language like Rails, you may be familiar with Sidekiq. Now, Sidekiq does exist in the Crystal world which is amazing. It even exists by the same guy that created the Ruby version! But at the time of this writing, I spent nearly an hour trying to figure out how to get it integrated with Lucky, and I was running in to quite a few issues. Thankfully, I found an alternative that I had installed and running locally in maybe 2 minutes.

Mosquito

Take a look at Mosquito. The API for it really isn't too far off from what Sidekiq uses, so for me, the concept was quick and easy. This also uses Redis, and since I was going to use Redis with Sidekiq, there's no additional things I needed to install.

Assuming you have your Lucky app all setup and ready to go, you can start by adding Mosquito in:

dependencies:
  mosquito:
    github: robacarp/mosquito

Run your shards install, and you'll be good to go.

Now we just need to setup our mosquito runner. Create a new file ./src/mosquito.cr.

require "./dependencies"
require "./workers/*"

Mosquito::Runner.start

Setting up workers

Next we need to setup the workers. Create a new folder ./src/workers/, and let's make a login_worker.cr file in there.

class LoginWorker < Mosquito::QueuedJob
  params user_id : Int64
  params ip : String

  def perform
    user = UserQuery.find(user_id)
    if user
      # create new Login record from the user
    end
  end
end

Using the worker

At some point we have to kick off the LoginWorker we made, so in one of the actions, we can add this:

post "/login" do
  form = SessionForm.new(params)
  form.submit do |f, user|
    if user
      LoginWorker.new(user_id: user.id, ip: "127.0.0.1").enqueue
      # redirect logged in
    else
      # show your errors
    end
  end
end

Booting the whole thing

Now that you have mosquito installed, and setup, you'll just need to boot the whole app. Since Mosquito runs in it's own process, you have to run it in a separate process.

In your Procfile.dev add in the line worker: crystal src/mosquito.cr.

web: lucky watch --reload-browser
assets: yarn watch
worker: crystal src/mosquito.cr

Now when you run lucky dev, it'll boot mosquito in it's on separate process!

ProTip:

1. If your app is like mine, and just an api, your Procfile.dev might look more like

web: crystal src/server.cr
worker: crystal src/mosquito.cr

1. Mosquito needs to know about your stack since it runs in a separate process. If you're trying to slim this down, just keep in mind that whatever you put in your worker file, you'll need to require that separately from your app stack.

2. By default, Mosquito looks at your REDIS_URL ENV, or pulls just a normal localhost value. Be sure to update that for production, as well as your Procfile for your production.

3. If you would rather build the targets (maybe for docker), and run those separately, you can update your shard.yml with:
   

targets:
  server:
    main: src/server.cr
  mosquito:
    main: src/mosquito.cr

Then shards build --production --static --release --no-debug to get your server and mosquito binary files. Just run those however you need, and you should be good.