DEV Community

loading...

Robust queue job handling in Craft CMS

gaijinity profile image Andrew Welch Originally published at nystudio107.com on ・17 min read

Robust queue job handling in Craft CMS

Craft CMS uses queue jobs for long-run­ning tasks. Here’s how to avoid stalled or failed queue jobs by using an inde­pen­dent async queue runner

Andrew Welch / nystudio107

Craft cms yii2 async queue

Queu­ing in line is some­thing few peo­ple are fond of, but it allows each per­son to be ser­viced in order, in an effi­cient and fair manner.

Craft CMS also uses a queue, too. Any lengthy task is pack­aged off into a job , and jobs are put into a queue to be exe­cut­ed in order.

Queue enqueue dequeue craft cms

In com­put­er sci­ence terms, this is a First In, First Out (FIFO) data struc­ture that ser­vices things in what we’d con­sid­er to be a nat­ur­al order.

Here are some lengthy tasks that end up as jobs in the Craft CMS queue:

  • Updat­ing entry drafts and revisions
  • Delet­ing stale tem­plate caches
  • Gen­er­at­ing pend­ing image transforms
  • Re-sav­ing Elements
  • Updat­ing search indexes
  • Plu­g­in cre­at­ed queue jobs

The queue exists to allow these often lengthy jobs to run in the back­ground with­out block­ing access to the Craft CMS Con­trol Pan­el (CP) while they execute.

See How They Run

So now what we know what type of jobs will end up in the queue, let’s have a look at how these jobs are run. This ends up being a cru­cial­ly impor­tant bit of infor­ma­tion to under­stand about queue jobs.

Queue runners

Giv­en that Craft CMS is built on top of the Yii2 frame­work, it makes sense that it lever­ages the exist­ing Yii2-queue by lay­er­ing a QueueIn­ter­face on top of it.

It does this so that it can pro­vide a nice friend­ly sta­tus UI in the CP. The default Craft set­up uses a data­base (Db) dri­ver for queue jobs, stor­ing them in the queue table.

Although there are oth­er queue dri­vers avail­able, you’re unlike­ly to see any gains from using any­thing oth­er than the Craft CMS Db dri­ver. And you’d lose the afore­men­tioned sta­tus UI in the CP.

But how do the queue jobs run? Here’s how:

  • When some­one access the CP
  • Via web request

Here­in lies the rub with the default queue imple­men­ta­tion in Craft CMS. No queue jobs will run at all unless some­one logs into the CP, and even then, they are run via web request.

Here’s an excerpt from the Craft CMS CP.js JavaScript that kicks off the the web-based queue running:


runQueue: function() {
    if (!this.enableQueue) {
        return;
    }

    if (Craft.runQueueAutomatically) {
        Craft.queueActionRequest('queue/run', $.proxy(function(response, textStatus) {
            if (textStatus === 'success') {
                this.trackJobProgress(false, true);
            }
        }, this));
    }
    else {
        this.trackJobProgress(false, true);
    }
},

And here’s the queue/​run con­troller method that ends up get­ting exe­cut­ed via AJAX:


/**
 * Runs any waiting jobs.
 *
 * @return Response
 */
public function actionRun(): Response
{
    // Prep the response
    $response = Craft::$app->getResponse();
    $response->content = '1';
    // Make sure Craft is configured to run queues over the web
    if (!Craft::$app->getConfig()->getGeneral()->runQueueAutomatically) {
        return $response;
    }
    // Make sure the queue isn't already running, and there are waiting jobs
    $queue = Craft::$app->getQueue();
    if ($queue->getHasReservedJobs() || !$queue->getHasWaitingJobs()) {
        return $response;
    }
    // Attempt to close the connection if this is an Ajax request
    if (Craft::$app->getRequest()->getIsAjax()) {
        $response->sendAndClose();
    }
    // Run the queue
    App::maxPowerCaptain();
    $queue->run();
    return $response;
}

They are doing what they can to make sure the web-based queue jobs will run smooth­ly, but there’s only so much that can be done. Bran­don Kel­ly from Pix­el & Ton­ic even not­ed ​“Craft 2, and Craft3 by default, basi­cal­ly have a poor-man’s queue”.

Web requests are real­ly meant for rel­a­tive­ly short, atom­ic oper­a­tions like load­ing a web page. For this rea­son, both web servers and PHP process­es have time­outs to pre­vent them from tak­ing too long.

Stuck queue jobs

This can result in bad things happening:

  • Queue jobs stack up until some­one logs into the CP
  • Queue jobs can fail or get stuck if they take too long and time out
  • Queue jobs can fail or get stuck if they run out of memory
  • Lengthy queue jobs can impact CP or fron­tend per­for­mance as they run

For­tu­nate­ly, we can alle­vi­ate all of these prob­lems, and it’s not hard to do!

If you do have tasks that have failed due to errors, check out the Zen and the Art of Craft CMS Log File Read­ing arti­cle to learn how to debug it.

Queue Run­ning Solutions

For­tu­nate­ly Yii2, and by exten­sion Craft CMS, comes with a robust set of com­mand line tools that I detail in the Explor­ing the Craft CMS 3 Con­sole Com­mand Line Inter­face (CLI) article.

Shift into high gear

The queue com­mand gives us the abil­i­ty to shift our queue jobs into high gear by run­ning them direct­ly. PHP code that is run via the con­sole com­mand line inter­face (CLI) uses a sep­a­rate php.ini file than the web-ori­ent­ed php-fpm. Here’s an exam­ple from my server:


/etc/php/7.1/fpm/php.ini
/etc/php/7.1/cli/php.ini

The rea­son this is impor­tant is that while the web-based php-fpm might have rea­son­able set­tings for memory_​limit and max_​execution_​time, the CLI-based PHP will typ­i­cal­ly have no memory_limit and no max_execution_time.

This is because web process­es are exter­nal­ly trig­gered and there­fore untrust­ed, where­as things run­ning via the CLI is inter­nal and trusted.

So run­ning our queue jobs via PHP CLI gives us the fol­low­ing benefits:

  • Can be run any time, not just when some­one access­es the CP
  • Won’t be ter­mi­nat­ed due to lengthy exe­cu­tion time
  • Are unlike­ly to run out of memory

The queue con­sole com­mands we use are:

  • queue/run — Runs all jobs in the queue, then exits
  • queue/listen — Per­pet­u­al­ly runs a process that lis­tens for new jobs added to the queue and runs them

There are sev­er­al ways we can run these com­mands, dis­cussed in the Yii2 Work­er Start­ing Con­trol arti­cle, and we’ll go into detail on them here.

Solu­tion #1: Async Queue Plugin

If you want a no fuss, no muss solu­tion that just involves installing a plu­g­in, check out Oliv­er Stark’s Async Queue plu­g­in. From the plug­in’s description:

[Async Queue plu­g­in] replaces Craft’s default queue han­dler and moves queue exe­cu­tion to a non-block­ing back­ground process. The com­mand craft queue/run gets exe­cut­ed right after you push a Job to the queue.

If you want a sim­ple solu­tion that does­n’t require doing any­thing more than installing a plu­g­in, go this route. You’ll end up with a more robust solu­tion for run­ning your queue jobs.

That’s it, you’re done!

Solu­tion #2: Forge Daemons

If you use the won­der­ful Lar­avel Forge to pro­vi­sion your servers (dis­cussed in detail in the How Agen­cies & Free­lancers Should Do Web Host­ing arti­cle), Forge pro­vides a nice GUI inter­face to what it calls Dae­mons.

Go to your Serv­er, then click on Dae­mons, and you’ll see a GUI that looks like this:

Forge new daemon

Here’s the com­mand we’ve added for the dev​Mode​.fm website:


/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

Let’s break this down:

  • /usr/bin/nice -n 10 — this uses the Unix nice com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
  • /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the craft CLI exe­cutable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge and we want 2 of these process­es run­ning. Click Start Dae­mon and you’ll be greet­ed with:

Forge daemon added

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:


    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

The fol­low­ing is an entire­ly option­al peek under the hood at how this works with Lar­avel Forge. Feel free to skip it if you’re not inter­est­ed in the gory details.

What Forge calls Dae­mons are actu­al­ly com­mands run via the Super­vi­sor com­mand. When we cre­at­ed the above dae­mon in the Forge GUI, it cre­at­ed a file in /etc/supervisor/conf.d:


forge@nys-production /etc/supervisor/conf.d $ ls -al
-rw-r--r-- 1 root root 293 Aug 3 14:55 daemon-157557.conf

The con­tents of the file look like this:


[program:daemon-157557]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-157557.log

And indeed, this is exact­ly one of the meth­ods rec­om­mend­ed in the in the Yii2 Work­er Start­ing Con­trol arti­cle. Huzzah!

Solu­tion #3 Using Heroku

If you’re using Heroku for your host­ing needs, you can use Work­er Dynos to run your queue/listen command:


./craft queue/listen --verbose

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:


    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Solu­tion #4: Uti­liz­ing systemd

If you’re using a Lin­ux serv­er, but you’re not using Forge or Heroku, or per­haps you just want to con­fig­ure things your­self, we can accom­plish the same thing using sys­temd.

systemd is a way to start, stop, and oth­er­wise man­age dae­mon process­es that run on your serv­er. You’re already using it whether you know it or not, for things like your web serv­er, MySQL, and so on.

Here’s what I did for dev​Mode​.fm to use systemd to run our queue jobs. You’ll need sudo access to be able to do this, but for­tu­nate­ly most mod­ern VPS host­ing ser­vices give you this ability.

First I cre­at­ed a .service file in /etc/systemd/system/ named devmode-queue@.service via:


forge@nys-production ~ $ sudo nano /etc/systemd/system/devmode-queue@.service

Here’s what the con­tents of the file looks like:


[Unit]
Description=devMode.fm Queue Worker %I
After=network.target
After=mysql.service
Requires=mysql.service

[Service]
User=forge
Group=forge
ExecStart=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose
Restart=always

[Install]
WantedBy=multi-user.target

Let’s break this down:

  • /usr/bin/nice -n 10 — this uses the Unix nice com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
  • /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the craft CLI exe­cutable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge , we want it to restart the process on fail­ure and that it depends on both the network and mysql ser­vices being available.

The strange @ in the .service file name allows us to have mul­ti­ple instances of this ser­vice run­ning. We can then use the range syn­tax {1..2} to spec­i­fy how many of these process­es we want.

We’ve cre­at­ed the file, but we need to start the ser­vice via:


forge@nys-production ~ $ sudo systemctl start devmode-queue@{1..2}

And then just like our oth­er ser­vices such as nginx and mysqld we want it to start up auto­mat­i­cal­ly when we restart our server:


forge@nys-production ~ $ sudo systemctl enable devmode-queue@{1..2}

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:


    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Now you can start, stop, enable, dis­able, etc. your Craft CMS queue run­ner just like you would any oth­er sys­tem ser­vice. You can also mon­i­tor it via the jour­nalctl command:


forge@nys-production ~ $ sudo journalctl -f -u devmode-queue@*.service
-- Logs begin at Sat 2019-08-03 13:23:17 EDT. --
Aug 03 14:06:04 nys-production nice[1364]: Processing element 42/47 - Neutrino: How I Learned to Stop Worrying and Love Webpack
Aug 03 14:06:04 nys-production nice[1364]: Processing element 43/47 - Google AMP: The Battle for the Internet's Soul?
Aug 03 14:06:04 nys-production nice[1364]: Processing element 44/47 - The Web of Future Past with John Allsopp
Aug 03 14:06:04 nys-production nice[1364]: Processing element 45/47 - Web Hosting with ArcusTech's Nevin Lyne
Aug 03 14:06:04 nys-production nice[1364]: Processing element 46/47 - Tailwind CSS utility-first CSS with Adam Wathan
Aug 03 14:06:04 nys-production nice[1364]: Processing element 47/47 - Talking Craft CMS 3 RC1!
Aug 03 14:06:04 nys-production nice[1364]: 2019-08-03 14:06:04 [1427] Generating episodes sitemap (attempt: 1) - Done (1.377 s)
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Started
Aug 04 09:25:25 nys-production nice[1364]: Processing element 1/1 - Calendar of Upcoming Episodes
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Done (0.174 s)

The above com­mand will tail the systemd logs, only show­ing mes­sages from our devmode-queue service.

You can learn more about using journalctl in the How To Use Jour­nalctl to View and Manip­u­late Sys­temd Logs article.

Get­ting Dotenvious

This is an entire­ly option­al sec­tion you can skip if you’re not inter­est­ed in learn­ing how to not use .env files in pro­duc­tion, as rec­om­mend­ed by the php­dotenv pack­age authors:

php­dotenv is made for devel­op­ment envi­ron­ments, and gen­er­al­ly should not be used in pro­duc­tion. In pro­duc­tion, the actu­al envi­ron­ment vari­ables should be set so that there is no over­head of load­ing the .env file on each request. This can be achieved via an auto­mat­ed deploy­ment process with tools like Vagrant, chef, or Pup­pet, or can be set man­u­al­ly with cloud hosts like Pagod­abox and Heroku.

Addi­tion­al­ly some setups have envi­ron­men­tal vari­ables intro­duced by alter­nate means. I made a Doten­vy pack­age that makes it easy to gen­er­ate .env key/​value vari­able pairs as Apache, Nginx, and shell equivalents.

In this case, we’ll need to mod­i­fied the com­mand we run slight­ly, so that we’re defin­ing our envi­ron­men­tal vari­ables in the shell before run­ning Craft via the CLI.

All we need to do is change our com­mand to this:


/bin/bash -c ". /home/forge/devmode.fm/.env_cli.txt && /usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose"

All we’re doing is exe­cut­ing the .env_cli.txt file (cre­at­ed by Doten­vy) that con­tains our envi­ron­men­tal vari­ables we export to the shell environment:


forge@nys-production ~/devmode.fm (master) $ cat .env_cli.txt
# CLI (bash) .env variables
# Paste these inside your .bashrc file in your $HOME directory:
export ENVIRONMENT="live"
export SECURITY_KEY="XXXXX"
export DB_DRIVER="pgsql"
export DB_SERVER="localhost"
export DB_USER="devmode"
export DB_PASSWORD="XXXXX"
export DB_DATABASE="devmode"
export DB_SCHEMA="public"
export DB_TABLE_PREFIX=""
export DB_PORT="5432"
export SITE_URL="https://devmode.fm/"
export BASE_PATH="/home/forge/devmode.fm/web/"
export BASE_URL="https://devmode.fm/"
export REDIS_HOSTNAME="localhost"
export REDIS_PORT="6379"
export REDIS_DEFAULT_DB="0"
export REDIS_CRAFT_DB="3"
export PUBLIC_PATH="/dist/"
export DEVSERVER_PUBLIC="http://192.168.10.10:8080"
export DEVSERVER_HOST="0.0.0.0"
export DEVSERVER_POLL="1"
export DEVSERVER_PORT="8080"
export DEVSERVER_HTTPS="0"

The rest is the same as the com­mands we’ve dis­cussed above for run­ning queue/listen, and then we wrap it all in /bin/sh -c so it appears as a sin­gle com­mand that can be executed.

Deeply Queued

Here’s some queue job triv­ia that you may find interesting.

Craft cms queue job trivia

We men­tioned ear­li­er that the only way queue jobs are run nor­mal­ly is if you vis­it the CP. This is most­ly true.

There’s actu­al­ly a case where queue jobs can be run via fron­tend page load. If you have gen­er­ate­Trans­forms­Be­forePage­Load set to false (which is the default), and there are pend­ing trans­forms that need to be generated.

What hap­pens is the Assets::getAssetUrl() method push­es the image trans­form queue job, which calls Queue::push():


    /**
     * @inheritdoc
     */
    public function push($job)
    {
        // Capture the description so pushMessage() can access it
        if ($job instanceof JobInterface) {
            $this->_jobDescription = $job->getDescription();
        } else {
            $this->_jobDescription = null;
        }

        if (($id = parent::push($job)) === null) {
            return null;
        }

        // Have the response kick off a new queue runner if this is a site request
        if (Craft::$app->getConfig()->getGeneral()->runQueueAutomatically && !$this->_listeningForResponse) {
            $request = Craft::$app->getRequest();
            if ($request->getIsSiteRequest() && !$request->getIsAjax()) {
                Craft::$app->getResponse()->on(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);
                $this->_listeningForResponse = true;
            }
        }

        return $id;
    }

So if this is a site (fron­tend) request, and it’s not an AJAX request, Craft will add a call to Queue::handleResponse() that gets fired when the Response is about to be sent back to the client.

In this case, Craft will inject some JavaScript into the fron­tend request that pings the queue/run con­troller via AJAX:


    /**
     * Figure out how to initiate a new worker.
     */
    public function handleResponse()
    {
        // Prevent this from getting called twice
        $response = Craft::$app->getResponse();
        $response->off(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);

        // Ignore if any jobs are currently reserved
        if ($this->getHasReservedJobs()) {
            return;
        }

        // Ignore if this isn't an HTML/XHTML response
        if (!in_array($response->getContentType(), ['text/html', 'application/xhtml+xml'], true)) {
            return;
        }

        // Include JS that tells the browser to fire an Ajax request to kick off a new queue runner
        // (Ajax request code adapted from http://www.quirksmode.org/js/xmlhttp.html - thanks ppk!)
        $url = Json::encode(UrlHelper::actionUrl('queue/run'));
        $js = <<<EOD
<script type="text/javascript">
/*<![CDATA[*/
(function(){
    var XMLHttpFactories = [
        function () {return new XMLHttpRequest()},
        function () {return new ActiveXObject("Msxml2.XMLHTTP")},
        function () {return new ActiveXObject("Msxml3.XMLHTTP")},
        function () {return new ActiveXObject("Microsoft.XMLHTTP")}
    ];
    var req = false;
    for (var i = 0; i < XMLHttpFactories.length; i++) {
        try {
            req = XMLHttpFactories[i]();
        }
        catch (e) {
            continue;
        }
        break;
    }
    if (!req) return;
    req.open('GET', $url, true);
    if (req.readyState == 4) return;
    req.send();
})();
/*]]>*/
</script>
EOD;

        if ($response->content === null) {
            $response->content = $js;
        } else {
            $response->content .= $js;
        }
    }

While tech­ni­cal­ly this could hap­pen for any queue job that is Queue::pushed()​‘d in the queue dur­ing a fron­tend request, asset trans­forms are the only case I’m aware of that this hap­pens in practice.

For this rea­son, if you’re post­ing any XHR requests to Craft con­trollers or API end­points, make sure you add the header:


'X-Requested-With': 'XMLHttpRequest'

Craft looks for this head­er to deter­mine whether the request is AJAX or not, and will not attempt to attach its queue run­ner JavaScript to the request if the request is AJAX.

This head­er is set auto­mat­i­cal­ly by jQuery for $.ajax() requests, but it is not set auto­mat­i­cal­ly by pop­u­lar libraries such as Axios.

Wrap­ping Up

So which method do I use? If I’m spin­ning up a VPS with Forge (which I nor­mal­ly am), then I’ll use Solu­tion #2: Forge Dae­mons. If it’s not a Forge box, but I do have sudo access, then I’ll use Solu­tion #3: Uti­liz­ing sys­temd.

I use Solu­tion #1: Async Queue Plu­g­in on client sites where nei­ther of the above are the case. Oliv­er’s plu­g­in is great, and much bet­ter than the default way that queue jobs are run, but I like lever­ag­ing exist­ing sys­tems if pos­si­ble, and hav­ing nice con­trol over the process pri­or­i­ty is, well, nice.

There is always hope

Replac­ing the default web-based queue job sys­tem in Craft CMS with a more robust CLI-based sys­tem will pay dividends.

Your queue jobs will run more smooth­ly, fail less often, and have less of an impact on the CP/​frontend.

It’s worth it. Do it.

Hap­py queuing!

Further Reading

If you want to be notified about new articles, follow nystudio107 on Twitter.

Copyright ©2020 nystudio107. Designed by nystudio107

Discussion (0)

pic
Editor guide