Hey everyone, I am planning to work on a project that includes fetching/merging data from various sources (CSV, JSON, XML) and then save them into databases/files of my choice.
So far I tried Azure Data Factory and Logic App Service for that purpose however they were not what I was looking for especially for translating CSV data into Table Storage tables. I wish I had the ability to specify which rows to be deleted based on some condition.
The end game is to fetch data from 20+ different sources, merge them into several data flows and then translate them into NoSQL databases or JSON files.
Additionally, I have only recently checked the capabilities of R and it seemed very promising in terms of data preparation. However I am not experienced on how to get all of these run on a cloud-based platform.
Thank you for the suggestions.
Top comments (3)
Hi Izzet,
When talking about 'low-code' or ‘no-code’ platforms, the term is interpreted differently and thus you have a few 'low-code' flavours that are not alike in functionality or appearance.
In my opinion, ‘low-code’ platforms fall into 3 silos: BPM/Workflow based, ETL tools and general application development platforms. These silos all target a different type of user as well.
BPM focused platforms are often more intuitive, guided and are quickly learnt compared to the rest. These often involve pre-built task-oriented apps or integrations focusing on workflows which work straight out of the box and focus on common business processes. These platforms have pre-built connectors to other major platforms and are normally on the ‘no code’ side of things. They appeal to non-technical business users in their simplicity.
However, these offer limited functionality in terms of extending the scope as well as adding extended logic to systems. Typical usage of these platforms involves approval processes, moving information and documents, triggering emails and flows.
Examples:
ETL tools are focused on the extraction , transformation and management of data. As the focus is on data, the scope of the pipelines or connectors that you can use are fairly large and powerful but specific to data sources and are generally used for bulk processing. The downside of these is that because of the focus, they aren’t capable of creating entire systems of use as well as being fairly technical to use.
Common usage involves, migrating legacy systems, data warehousing, data analytics
Examples:
Application Development platforms are more general-purpose low-code platforms that provide an application development environment. This involves visual functions and flexible connectors of commonly used functionality, resulting in more dynamic creation and customization of tasks and processes.
These types of platforms allow developers to rapidly build custom applications and integrations to every need, things such as Web Applications, migrations of legacy systems and custom CRMs are a few.
These often involve a higher level of technical expertise but massively reduce the time to production as well as entry barriers to users compared to traditional development.
For the purpose of this comparison, general purpose platforms allow you to also develop both BPM and ETL applications along with added custom business logic in the middle. While technically you are limited to the functionality that the platform provides, the functionality far outweighs more rigid platforms.
Examples:
Which to choose?
In terms of pulling and merging data from 20+ sources, you're most likely going to run into issues with strict templated BPM type platforms as you will need a variety of data manipulation and custom functionality specific to your needs.
The advantage of these systems is that you can customize your imports and data merges to your specification with limited development knowledge, and additional advantage is that you can extend the functionality of the system for more sources, different files and databases without being tied down to development choices of the more 'templated' systems.
Depending on the modifications and enhancements needed to be done on the data, your best bet would be an ETL tool which is specifically focused on your task at hand. However, systems rarely function in isolation, so the need may arise to extend the scope of your system beyond mere data retrieval and storage, in that case a more general application development platform like Linx would be better suited. However, your budget will also affect your decision of platform.
Full disclosure, I work at Linx , I'n my years of experience I've used low-code development platforms to do build various integrations and enterprise systems. The one issue I run into is that if you are buying off the shelf software, there will always come a time where this software cannot address your unique problem and thus you have to keep on purchasing more and more out the box systems. Using a general development platform allows you to build in only the required functionality of each system with you having complete control.
Best of luck.
Thank you very much Ronan for this very detailed response. You have listed many tools that I'm not familiar, I'll start giving them a try one by one now 💫
Awesome