Airflow vs Dagu comparison

I am developing an OSS workflow scheduler called Dagu with the goal of developing a simpler workflow scheduler to replace Airflow.

Airflow has been a popular tool for several years, but as a replacement for Cron, it is an overkill solution.

Dagu allows you to define DAGs (workflows) more simply than Airflow and has all the functionality you need as a scheduler.

In this article, I would like to explain in detail the advantages of Dagu compared to Airflow.

Advantages of Dagu over Airflow

Dagu has the following five major advantages over Airflow

• Install easily (single binary, no dependencies on other libraries)
• Setup & Run easily (No database is required)
• Define DAGs (workflows) in a simple YAML format
• Use existing programs as-is without modifying
• Schedule with Cron expressions

1. Install easily

To install Dagu, you just need to install the binary.

brew install yohamta/tap/dagu

If brew is not available, you can install Dagu in the current directory by simply running the following Bash command.

curl -L https://raw.githubusercontent.com/yohamta/dagu/main/scripts/downloader.sh | bash

2. Setup & Run easily (No database is required)

To get Dagu running, simply run the dagu server command.

dagu server

Then go to http://127.0.0.1:8080 and you will see the dagu's web UI.

No database installation is required!

3. Define DAGs (workflows) in a simple YAML format

A workflow that outputs "hello world" and "done!" to the log can be defined simply as follows.

steps:
  - name: "step 1"
    command: "echo Hello World"
  - name: "step 2" command: "echo done!
    command: "echo done!
    depends:
      - "step 1"

Airflow defines the workflow in Python code. The same workflow as above can be defined as follows.

Airflow version of the same DAG

from airflow import DAG
from airflow.operators.bash import BashOperator
dag = DAG(dag_id="hello_world_dag")
task1 = BashOperator(
    task_id="hello_world",
    bash_command='echo Hello World',
    dag=dag)
task2 = BashOperator(
    task_id="done",
    bash_command='echo done',
    dag=dag)
task1 >> task2

The weakness of Airflow is that it is difficult to understand what the workflow does at first glance.

4. Use existing programs as-is without modifying

In the Dagu workflow, you can simply write any command as is. There is nothing special about it.

If you have a program that has been running in Cron, you can call it as a workflow from Dagu by simply copying and pasting the commands written in Crontab into a YAML file.

There is no need to write Python code to call the existing code or go to the trouble of rewriting it into Python code like you would have to do with Airflow.

5. Schedule with Cron expressions

Dagu workflow schedules can be easily set up with Cron expressions. Just write schedule in YAML file as follows.

schedule: "0 10 * * * *"
steps:
  - name: step 1
    command: echo Hello World

The Dagu scheduler can be invoked with dagu scheduler command. Just run it and your workflow will automatically follow the schedule.

dagu scheduler --dags=<directory>

Summary

The Dagu scheduler is easier to install than Airflow and allows workflows (or jobs) to be configured with simple, short YAML.

If you are running jobs in Cron, there are many things to consider when installing Airflow or any other workflow scheduler. For example, if you are deploying a database for the scheduler, you will need to make that database redundant so that it is not a single point of failure.

The big advantage of Dagu is that it is easy to setup because it does not depend on any database since its architecture is all file based.

Dagu's Source Code and Repository

Dagu is an open source software developed in Golang and React and is under continuous development. If you are interested, please contribute to the software in anyway you want. You're very welcomed :)

https://github.com/yohamta/dagu