Probably you already know it but I know a tool that might cover the first three points of your list of things you're interested in: Luigi, a Python tool that you can use to build complex graphs of jobs. For example, AdRoll uses it for all sorts of things, like building containerized jobs and pipelines with AWS spot instances.
I don't yet have an answer to your question though :D I'll think about it
Probably you already know it but I know a tool that might cover the first three points of your list of things you're interested in: Luigi, a Python tool that you can use to build complex graphs of jobs. For example, AdRoll uses it for all sorts of things, like building containerized jobs and pipelines with AWS spot instances.
I don't yet have an answer to your question though :D I'll think about it
Thanks rhymes! I'll give it a read.
I've been working with our Data Engineers on Airflow but Luigi looks really interesting.
Airflow seems interesting too.
I look forward to your future comparison article ;-)
Oh man... the list of future posts is growing by the day. I had no idea I had so much to say until I started posting here.
Ahaha keep going