Although Data Engineer vs Data Scientist is not a good comparison, because one is an engineering role while the other is an analytical/quantitative role. One's job starts only on the completion of another Job. Very different career paths and skills.
Data Engineer
Data Engineer works on the data pipeline. Here the word pipeline is very fascinating and the meaning of this word is the same as you are thinking right now.
What are Pipelines:
The pipeline is the path where the oil or water or any liquid can transfer to the other place. Similarly, here in the data engineering field, we can transfer data from the Upstream system to the downstream System. No doubt, here we can also apply some Business logic or validation to restrict the data to be in a specific form.
Hence, the data engineer can clean the data from the raw data, and fetch some useful data from the Upstream ( Actual Place of Data (Source)) to the Downstream ( Place to be placed (Destination)).
It is Quoted that:
Data is the new oil
Every Company is trying to get more and more oil to become rich in the tech world
My Opinion on Data Engineering :
I think it's better to compare data engineering with software engineering. The difference is that software engineers build the tools, while data engineers use the tools.
For example
A software engineers build Spark, while data engineers use Spark.
It is right here about data engineering that is not as intellectually challenging. Think about it, would you rather be the one that builds the distributed systems underlying Spark, or do you just want to drive into the companies that just need to pull a good product with good UI/UX? If you work at such a company, I think Data Engineering is a bit more intellectually challenging than frontend web development. If you want to build cool distributed systems, then you gotta go work for companies like Google, Alibaba, Amazon, or even IBM.
It might be better to compare DE to backend engineering. I think a backend engineer with solid data modeling skills and database knowledge would be able to transition to a DE role very easily. However, a DE who only works with pyspark and cloud technologies would find it a bit more difficult to transfer to a backend engineer role.
Data Scientist
Data Scientist role is very simple. It works on Downstream data or destination data provided by a Data Engineer. Data Scientist Use the data to make some insights of any product of anything. It's work more like prediction and Recommendation strategies.
For Example
You are viewing the airbuds on AliExpress, it has been noticed that AliExpress recommend many other similar product to us. Same in the case of YouTube Videos Recommendation.
Data-Scientist made automate model using the downstream data to fetch the useful insights. Obviously, for data science field use need to have a strong knowledge of mathematics apart technology.
Both the field are super fascinating to me and hopefully after getting the difference and their role you might get the same intensions.
Conclusion
If an "engineer" wants to stay relevant and a "beast", then they should build the tools, rather than use them.
By the Way, I am falling in love with Big Data 💖
Join my Ecosystem:
LinkedIn
ZapeeoSheikh
Top comments (0)