DigDag - Schedule a Job(workflow) multiple times on same day

#digdag #scheduler #productivity #devops

Hey folks! So, I had a task of scheduling a Dashboard execution daily at 12.00 pm and 4.00 pm and was supposed to use DigDag Job Scheduler. But, after spending quite some time on DigDag’s documentation, I couldn’t find anything relevant as per my requirement. Hence, I decided to try a workaround.

"Instead of scheduling the Dashboard using a single workflow (the *.dig file), I can import/include the workflow into another workflow(s) and schedule it as per my need."

For example:

Note: The below example is demonstrated on Ubuntu 18.04 (Linux). Please refer DigDag’s documentation for other platform-specific commands.

Lets generate a sample digdag project schedule-demo using command:

$ digdag init schedule-demo

The above command will create a workflow schedule-demo.dig in schedule-demo directory/project with contents:

timezone: UTC


+setup:

 echo>: start ${session_time}


+disp_current_date:

 echo>: ${moment(session_time).utc().format(‘YYYY-MM-DD HH:mm:ss Z’)}


+repeat:

 for_each>:

 order: [first, second, third]

 animal: [dog, cat]

 _do:

 echo>: ${order} ${animal}

 _parallel: true


+teardown:

 echo>: finish ${session_time}

Let's schedule the above workflow for every one and two minutes on same day using file one_minute.dig and two_minutes.dig workflow.

Create workflows one_minute.dig and two_minutes.dig in the project schedule-demo with below-mentioned contents:

Workflow one_minute.dig:

!include : ‘schedule-demo.dig’

schedule:

 minutes_interval>: 1

Workflow two_minute.dig:

!include : ‘schedule-demo.dig’

schedule:

 minutes_interval>: 2

Now change your current working directory to schedule-demo using the command:

$ cd {current_path}/schedule-demo/

then push the project schedule-demo to digdag server using the command:

$ digdag push schedule-demo

(Note: On local Ubuntu setup, to start digdag server execute command ‘digdag server’ on the terminal)

If you are outside the project directory and want to push project use command:

$ digdag push schedule-demo --project /{path}/schedule-demo  --revision schedule-demo-v1

{path} => path to the schedule-demo project

Then, launch a browser (any) and add URL ‘http://localhost:65432/’ and under Workflow tab you will see the result similar to below image (you will have to wait for at least 5–6 minutes to get similar output as of mine):

If you observe the image, the output with ID 2 and ID 5 has 2 minutes difference (refer column 'Last Attempt' ) and the value of Workflow is two-minute. You will find similar result for the one-minute workflow (i.e. the execution time difference between every present and next workflow attempt will be 1 minute).

P.S. There could be different ways to achieve the result demonstrated in the above example. This is just one of those :)

If you enjoyed this blog, share it with your friends!

Thank-You for reading!

Sources:

https://www.digdag.io/
https://docs.digdag.io/getting_started.html
https://docs.digdag.io/workflow_definition.html?highlight=include#include-another-file