DEV Community

Cover image for The Data Engineer Roadmap ๐Ÿ—บ
Bobby Iliev
Bobby Iliev

Posted on • Originally published at devdojo.com

The Data Engineer Roadmap ๐Ÿ—บ

Introduction

This has been inspired by the Full Stack Developer's Roadmap post written by @ender_minyard ๐Ÿ™Œ

With the ever growing data volumes and demands, the data engineering career has been one of the fastest growing jobs for the past few years.

According to the 2021 Stack Overflow survey, data engineers are one of the top 5 highest paid professionals right after SREs and DevOps engineers:

If you are looking to become a data engineer, here are some resources for data engineering that you can save for later.

Table Of Contents

  • ๐Ÿ’ป Fundamentals
  • ๐Ÿ‘ฉโ€๐Ÿ’ป Programming basics
  • ๐Ÿงช Testing
  • ๐Ÿ“Š Database Fundamentals
  • ๐Ÿ  Data warehouses
  • ๐Ÿ“ฆ Object storage
  • โšก Data processing
  • ๐Ÿ“ฉ Messaging
  • ๐Ÿ’ฝ Cluster computing
  • โฒ Workflow Scheduling
  • ๐Ÿ“บ Monitoring data pipelines
  • ๐Ÿ‘จโ€๐Ÿ’ป Infrastructure as Code
  • ๐Ÿ›ซ CI/CD

๐Ÿ’ป Fundamentals

Having a solid understanding of the Linux operating system could be a must in many IT related roles. You are going to benefit a lot if you know the basics of the following:

๐Ÿ‘ฉโ€๐Ÿ’ป Programming basics

As with any IT related role it is essential to have fundamental knowledge of programming in general. The programming language itself does not matter that much, but you need to have good understanding of programming paradigms and best practices.

๐Ÿงช Testing

๐Ÿ“Š Database Fundamentals

Having a solid understanding of SQL, data normalization and ACID transactions is a must for all data engineers.

Relational Databases

Non-relational databases

๐Ÿ  Data warehouses

๐Ÿ“ฆ Object storage

โšก Data processing

Batch

Hybrid

Streaming

๐Ÿ“ฉ Messaging

๐Ÿ’ฝ Cluster computing

โฒ Workflow Scheduling

๐Ÿ“บ Monitoring data pipelines

๐Ÿ‘จโ€๐Ÿ’ป Infrastructure as Code

๐Ÿ›ซ CI/CD

Conclusion

This was inspired by the Data Engineer Roadmap open source repository here:

https://github.com/datastacktv/data-engineer-roadmap

I wanted to build upon the roadmap and provide a list of resources for each topic.

Let me know if I've missed anything! Hope you find this useful and make sure to keep learning ๐Ÿ™Œ

You can follow me on Twitter at: @bobbyiliev_

Discussion (4)

Collapse
ruanbekker profile image
Ruan Bekker

Bobby, you never fail to impress! Amazing post! ๐ŸŽ‰

Collapse
bobbyiliev profile image
Bobby Iliev Author

Thank you Ruan! Really appreciate this ๐Ÿ™Œ

Collapse
mccurcio profile image
Matt Curcio

Really excellent content!
Thanks

Collapse
bobbyiliev profile image
Bobby Iliev Author

Thank you! Really appreciate this!