DEV Community

Cover image for Tech Exceptions Show: Accelerating Data Engineering with Azure
Microsoft Azure

Tech Exceptions Show: Accelerating Data Engineering with Azure

Adi Polak
1 out of 25 influential women in Software Development according to Apiumhub. I am a software developer who would like to learn more!
・2 min read

Data Engineers is the new hotness, many developers have been already working with distributed data technologies, such as Apache Kafka, Apache Spark, Apache Cassandra as backend developers, building infrastructure for analytics and enabling a healthy flow of data in the organization.

The reason we see more media coverage for that topic is the maturity of Data Science and Machine Learning. ML used to be most hyped. Following the hype, companies realized that they need to enable smarter products, So what did they do? they started hiring Data Scientists, although, they didn't have the infra to support them.

Due to that, many companies are now focused on building in-house ML platforms to enable their Data Scientists to get more value out of the data.
Yes, you read correctly, more value out of the data.
Let's take a look at the Data Science needs pyramid:

Data science layers towards AI by Monica Rogati:
Alt Text

You can see the clear need for Data Infrastructure Engineers and Data Engineers, they are at the base of the pyramid. This means, without them, data scientists won't be able to do their job efficiently. Think about it as similar to Maslow's Human needs pyramid. At the base, there are the physical needs, without them, we won't exist.

Alt Text

you are probably curious, why do I share this with you, well I had a wonderful conversation with Sheel Choksi, Solution Architect at Ascend and we talked exactly about that. How we can help Data Engineers do more and accelerate development and ML in organizations. Watch Now 📺!

Discussion (0)