DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Cover image for Data Engineering 101: Introduction to Data Engineering
Gathuru_M
Gathuru_M

Posted on

Data Engineering 101: Introduction to Data Engineering

Data Engineering is the process of building data pipelines and making quality data available for efficient data-driven decision-making.
A person who performs these activities is called a Data Engineer.

But what are data pipelines exactly...
In data processing, there is the flow of data from say a point A to B to C i.e., from an application to a data warehouse or from a data source to the database. This series of processing steps is called a data pipeline.
In these series of steps, each step delivers an output that is the input to the next step. This continues until the pipeline is complete. However, in some cases, independent steps may be run in parallel.

Data Pipeline Patterns

What’s the difference between a data analyst and a data engineer?
Data scientists and data analysts analyze data sets to gain knowledge and insights. Data engineers on the other hand build systems for collecting, validating, and preparing that high-quality data which is then used by data scientists to promote better business decisions.

With that said, these are some of the Essential skills required to be a Data Engineer in 2022

  • Data Structures
  • SQL
  • NoSQL
  • Understanding of Data Lakes and Data Warehouse
  • Python
  • Big Data - Hadoop, Apache Spark(PySpark), Hive, and Apache Kafka
  • Cloud Services - AWS, Microsoft Azure, Google Cloud, Snowflake, etc.
  • Visualization - Tableau, PowerBI, Looker, Qlikview, etc.

I wish you all the best as you choose to pursue this journey.

Thanks for reading!
Any questions? Leave your comment below to start fantastic discussions!

Top comments (1)

Collapse
 
dendihandian profile image
Dendi Handian

Wish everyone who starting data engineering career a good luck

Why You Need to Study Javascript Fundamentals

The harsh reality for JS Developers: If you don't study the fundamentals, you'll be just another β€œCoder”. Top learnings on how to get to the mid/senior level faster as a JavaScript developer by Dragos Nedelcu.