Summary
I'm developing a No-code workflow engine with built-in web UI. I am writing this article because I am looking for new contributors. If you are interested, please take a look at the GitHub repository :)
Dagu is an ideal workflow engine to use for existing code base, personal projects, or smaller use cases with fewer groups of people.
What is a workflow engine?
A workflow engine is a tool like Airflow, which was developed by Airbnb and is very popular among data scientists and DevOps engineers.
A workflow is a series of business processes or tasks executed according to a schedule or event. A workflow engine represents the dependencies of tasks as a DAG, which stands for Directed Acyclic Graph.
By representing a series of tasks as DAG, it is much easier to execute multiple related tasks in the correct order according to their dependencies.
Not only can you interpret and execute your workflow as a DAG, but you can also visualize the dependencies between tasks and represent complex workflows in a way that is easy to understand. This is the beauty of a workflow engine like Airflow.
Yes, ...... Honestly, it's not that great looking. Will improve soon. The important point is not the look, but the function.
What's the difference compared to Airflow or Prefect?
So why am I building a new workflow engine now? Why not just use an already popular workflow engine?
The reason is that Airflow and Prefect require you to write Python code to define DAGs. The system I am working with has already been in operation for over ten years and has hundreds of thousands of lines of code written in other languages. It is practically impossible to rewrite these codes in Python. And while it is possible to call existing code from the Python code, this would complicate the situation even more.
I searched all over the Internet for a lightweight workflow engine that is easy to use while still leveraging existing code, but no desirable tool existed for such a use case.
So I decided to develop a new workflow engine, Dagu, which allows you to define DAGs in a simple YAML format. So, in that sense, Dagu is no-code in terms of the way to manage workflows, compared to Airflow, Prefect, and Luigi. Since it's just a YAML file that define commands to execute, existing programs can be easily migrated without modification. In addition, because it is self-contained, no-dependency, and it does not require DBMS. These features make Dagu an ideal workflow engine to use for existing code base, personal projects, or smaller use cases with fewer groups of people.
Dagu is an alternative to a code-based workflow engine
It is not fancy, but I have implemented it in my own projects and have seen great results. Complex task dependencies have been clarified, and inefficient operations have been greatly improved.
However, a great deal of effort is still required to develop Dagu. If you are interested in Dagu, please feel free to contribute in any way you want!
What needs to be done now is summarized in the following TODO issues. Please take a look at it if you like. Any kind of contribution will be welcomed.
https://github.com/yohamta/dagu/issues/102
Thanks!
Top comments (4)
what do you think about Temporal.io ?
I didn't know about Temporal.io Looks interesting. Running workflows on a cloud using k8s seems like a right direction in the future for large data pipelines. Are you using Temporal.io in your project or something?
yeah, I used it in many "small" projects, now evaluating it to move all things to temporal workflows in a large project. Specially for go it works soooo fast, for NetCore the sdk is in progress (I have a couple of services in netcore) but related to golang all things works like a charm!
Looks like it is using reflection to run arbitrary tasks in Go. Interesting. I think the approach can be applied to any workflow engine, including Dagu also. What is the advantages of Temporal that you've found it better for your projects, compared to other workflow orchestrator tools?