Explaining Data Engineering to a 5 Year-Old (data is mud)

#datascience #beginners #database

You are at the playground, and it starts raining. What happens to the soil? It gets muddy - and then you can try to play with it and make shapes out of it. Maybe get a bit creative and colors it or add engravement?

What does that have to do with data engineering? Well, data in this story is mud, and trust me data can get muddy.

A data engineer's job is to collect the mud, see if it can be used to create clay. Remove any leaves or rocks from it. Then figure out where to put it in a place so that it stays in a good condition. Finally, they do their best to ensure that the mud is as clean as possible, and is stored in a safe and comfortable environment.

In data terms, data engineer's responsibilities are but are not limited to:

- Data Collection (collecting mud)
They are responsible for and sometimes advise on the best tools used for data collection. Support in the data collection process. Ensuring that the collected data is as clean and error-free as possible.

- Data Storage (storing the mud for easy access)
They would be responsible for storing the data. It is not an easy task, they need to ensure that data is stored in a way that is most suitable for the data type. For example, videos and images are stored differently than text. Also, they need to ensure that data retrieval is as optimal as possible by data analysts and data visualist. They also need to ensure that data is stored securely and protected from any breach attempt.

- Data Clean up (removing leaves and rocks from the mud)
They are sometimes responsible for cleaning data entry mistakes, outliers, discrepancies in the data to some extent.

- Data Analysis (figuring out the mud type and what we can make of it)
Though this is the job of a data analyst. Data engineers do not always get the joy of digging deep into the data and understanding its hidden messages. But they do take on primal data analysis.

On a closing note, data engineers may not be at the frontline, but they pave the road for data visualization and analysis. They are the foundation that data science is built on. They are our data avengers.