Outline
- TLDR
- Prologue
- Adventure, trials, and tribulations
- Death, rebirth, and transformation
- Battle, freedom, and victory
- Epilogue
TLDR
Mage pivoted from an AI platform to an open-source data pipeline tool and is making a huge impact on the lives of data engineers around the world.
Prologue
Once upon a time, there lived a promising young mage that left the magic academy early, journeying off on her own to make a difference in the world.
She believed that helping townsfolk harness the power of AI would create an economic boom across villages around the world.
Adventure, trials, and tribulations
The young mage created an AI tool to help developers at small companies build, train, and deploy AI models. Initially, the villagers were excited and had lots of interest in using the tool. Many of them lined up for days outside the village just to schedule demos and paid trials.
However, when the day approached to implement the tool, villagers kept giving the young mage reasons for why they weren’t ready. Some of these reasons included:
“We need a data warehouse first.”
“We need to make our first data hire.”
“We need a data pipeline management tool first.”
“We need to organize and clean our data first.”
After many sleepless nights, the young mage sensed that an evil dark presence had secretly infiltrated the village. This dark force came to be known as the Harbinger of Unnecessary Tools.
In that moment, the young mage realized that the AI tool was not a necessity; the villagers had more urgent problems that needed a remedy immediately. This realization was devastating because the AI tool had been worked on for over a year.
The young mage was defeated by the Harbinger of Unnecessary Tools. In pain and despair, she was driven out of the town and went into hiding; uncertain of her future.
Here are a few lessons inscribed in her tome:
Pay attention to why users don’t give a resounding “yes”. Avoid “maybe” like the black plague. They can be a secret poison because it gives hope that it’ll eventually be a “yes” when in fact it’s a “no”.
Fail fast. Find ways to prove the product wrong as fast as possible. The quicker it fails, the more chances there are to try something different and succeed.
Death, rebirth, and transformation
The young mage began doubting herself, questioning whether she left the magic academy too early. Injured and depleted of mana (energy that powers magic), she began wandering aimlessly through the abyss of the multiverse. Along the way, she spoke with nearly a thousand data professionals and asked them this question: what was the most boring part of your work?
Enlightened by the responses, the young mage meditated on the reasons why the villagers weren’t ready to implement the AI tool. After meditating and deciphering the arcane knowledge of responses gathered throughout the multiverse, the young mage had a revelation: companies need urgent help moving their data and preparing it for usage.
The young mage began rebuilding and leveling up her powers. She trained day and night for what seemed like an eternity. The young mage took some of the technology she used in the previous AI tool, infused it with power-ups, and open-sourced it. Legend has it that her reborn powers are known as the Data Pipeline Tool.
However, this was no ordinary tool; it has 3 major differentiators:
- It’s designed to have the easiest developer experience by providing a user interface; enabling developers to build data pipelines visually, quickly, and intuitively.
- It combines 3 use cases that have strong synergy and affinity for one another: batch processing pipelines, data integration pipelines, and streaming pipelines.
- Engineering best practices are built-in. The tool enables modular design of data pipelines; making each step in your pipeline easily reusable and simple to test with data validations.
At full power, the mage was ready to return and defend the village from the Harbinger of Unnecessary Tools!
Battle, freedom, and victory
The mage walked across astral planes and arrived at the village that was being oppressed by the Harbinger of Unnecessary Tools. She summoned all her powers and open-sourced the data pipeline tool.
After releasing the tool, spells of fire, water, wind, and lightning were cast at the dark force. With every passing moment, the open-source tool grew more powerful. As the battle raged on, bugs were eliminated, scalability issues were banished, powerful new features were added, and chromatic color began returning to the village.
After a fortnight of intense dueling, the Harbinger of Unnecessary Tools was finally defeated! The darkness of an unnecessary product, that had previously haunted the people, was lifted once and for all.
Everyone was liberated from the dark force’s grip and joy overflowed in the streets. The entire village praised the mage for saving them from the pain of using data tools with a dreadful developer experience. Countless villagers expressed their gratitude and gave many thanks to the mage.
Here are some of the testimonies from developers in that village:
“I was awestruck when I used Mage for the 1st time. It’s super clean and user friendly.” — Ajith Shetty, Data Engineer
“Recently tried Mage 🧙 and I must say I’m amazed by its developer centric usability.” — Salman Ahmed, Data Engineer
“Mage is such a refreshing orchestrator compared to Airflow.” — Anil Kulkarni, Senior Data Engineer
“All throughout this Slack space, you guys are quick, resourceful, and have an open mind. It really separates you from other orchestrators.” — Greg Lenane, Senior Analytics Engineer
“It took minimal work to start understanding and building pipelines as opposed to Airflow.” — Pedro Dellazzari, Data Scientist
“I would like to express my love for using Mage. My experience with it has been fantastic so far.” — Fabián Sepúlveda, Data Engineer
“I truly appreciate being part of this amazing community and am honored to have had the opportunity to contribute to its success.” — Dhia Gharsallaoui, Data Architect
“Just wanted to say that I’m incredibly impressed with what Mage is capable of. It’s incredibly powerful and user friendly.” — Matt Pegler, SVP of Innovation
“Hey! Transferring all our stuff to Mage from Airflow. We have around 80 pipelines running (and will be growing), managed by a team of 4.” — Nazari Goudin, Head of Data
“Pushed existing dbt repo into Mage repo as sub module. Oh Man, Mage is 🔥” — Vijayasarathy Muthu, Data Engineer
“To be honest, I am really loving Mage.” — Alexander Bolaño, Senior Data Engineer
“Congrats on creating one helluva DX. Night and day for all other tools we’ve been testing.” — Tomas Roaldsnes
“I have never seen such a friendly place to ask questions, I love the openness of it!” — Davis Vance, Data Engineer
“I finished the tutorial and my reaction was… DAMMMMM! It is a really nice platform! OMG” — Paulo Mota, Data Engineer
“I love the tool and see potential for not only being a standard tool but also a standard user experience. This is an amazing product y’all.” — Farman Pirzada, Senior Software Engineer
“Just deleted Cloud Composer yesterday and fully moved to Mage.” — Le Minh Nguyen, Data Scientist
“I’ve been using Mage for 2 months now and I must say that I’m really impressed with the work that has been done so far.” — Dion Salomon, Data Engineer
“It massively lowers the barrier for entry on data engineering, which has had to turn into its own specialized profession (Airflow isn’t exactly fun to debug).” — Nicolas Essipova, CTO
“I think this is one of the most beautiful pieces of software I’ve ever used. There is powerful sorcery at work here.” — Patrick Clark, Data Engineer
Since the release of the open-source data pipeline tool in June 2022, the project has received over 4.6 thousand stars on GitHub, over 2.1 thousand community members on Slack, and over a hundred teams using the tool in production.
In return for defeating the Harbinger of Unnecessary Tools, the Magic Council of Venture Capital decided to award the mage with an additional investment of $5 million. This investment was led again by Gradient Ventures, included previous investors (Essence VC, Designer Fund, Mana Ventures), and added an amazing group of strategic angel investors:
Guillermo Rauch (CEO @ Vercel)
Scott Breitenother (CEO @ Brooklyn Data Co)
Ananth Packkildurai (Data Engineering Weekly)
Ryan Boyd (Co-founder @ MotherDuck)
Jordan Tigani (CEO @ MotherDuck)
Benn Stancil (CTO @ Mode)
and several other data industry thought leaders.
In addition to acquiring more gold, the mage gathered a powerful and wise group of advisors:
- Zach Wilson (EcZachly)
- Benjamin Rogojan (Seattle Data Guy)
- Alexey Grigorev (DataTalks.Club)
- Joe Reis (Fundamentals of Data Engineering)
Epilogue
Mage is on a mission to make data engineering more accessible so that developers can harness the power of data to create magical experiences.
That’s why Mage is recruiting other fellow mages who want to go on an adventure and achieve this mission together. Here are the roles in the party that are currently open:
- Full Stack Engineer (Senior)
- Front-end Engineer (Senior)
- Backend Engineer (Senior)
- Developer Success Engineer (aka Developer Relations)
- Head of Developer Success
In the coming years, Mage will create a cooperative experience so that developers can build data pipelines with their team and level up together. After that journey, Mage will go on an epic quest to create the 1st open world community experience in the data universe.
Thank you so much for your relentless support of the mission and continued belief in the century long vision!
Top comments (0)