In Simply.TV we have a number of cronjobs running to surveil some of our internal processes.
These jobs are scheduled to run every single minute using the whenever gem and its
runner job type.
We have a job scheduled to run every single minute:
every 1.minute do runner 'Parse.detect_crashed_jobs' end
It's using the
rails runner command to execute the method
Parse.detect_crashed_jobs and is doing that every minute.
Unfortunately some times our load rises on the server for various reasons and there's a chance that one of these jobs ends up running for more than a minute. This can result in the same job being scheduled to run simultaneously as the crontab will just keep processing the job every minute.
flock is shipped with most Linux distributions, at least this is tested on Linux Ubuntu 16.4 and upwards.
I've introduced a new job type and template to the
job_type :runner, "cd :path && bin/rails runner -e :environment \":task\" :output" set :job_template, "bash -l -c 'flock -n '\\''/tmp/:task-runner.lockfile'\\'' -c '\\'':job'\\'''"
job_template is using the
bash command exactly like
whenever defaults, but I've wrapped the actual job execution in a
flock writes a temp file (using the task as filename) and uses this temp file to track if the job is running already. If it's already running it's skipping execution and exits.
At the same time I'm also changing the
job_type for runner slightly by wrapping the task in double quotes instead of single quotes which is
whenevers default value. This is to allow the
job_template to wrap the flock command in single quotes. It just makes everything easier.
Now whenever a job is scheduled using
whenever via the
runner method, the job will be executed through
flock who'll ensure the same job isn't running twice.