DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
The Proxy Economy: Residential, Datacenter, and ISP Rotation

The Proxy Economy: Residential, Datacenter, and ISP Rotation

Comments
5 min read
The Database Query That Could Cost a Company Millions(And Why Data Engineers Exist)

The Database Query That Could Cost a Company Millions(And Why Data Engineers Exist)

Comments
5 min read
Automating Serverless Data Ingestion: How to Connect External APIs to BigQuery using Python and Cloud Functions

Automating Serverless Data Ingestion: How to Connect External APIs to BigQuery using Python and Cloud Functions

Comments
12 min read
The Data Liberation: Amazon Athena and the Architecting of a Serverless Future

The Data Liberation: Amazon Athena and the Architecting of a Serverless Future

Comments
3 min read
Amazon S3 Tables Just Got Smarter: Intelligent-Tiering & Native Replication Explained

Amazon S3 Tables Just Got Smarter: Intelligent-Tiering & Native Replication Explained

Comments
4 min read
When code-gen suggests deprecated Pandas APIs — a subtle drift that broke a pipeline

When code-gen suggests deprecated Pandas APIs — a subtle drift that broke a pipeline

Comments
3 min read
Why Data SLAs Fail — and How to Enforce Them with a Unified Reliability Framework

Why Data SLAs Fail — and How to Enforce Them with a Unified Reliability Framework

Comments
2 min read
Unveiling the Power of Databases in the Realm of Big Data

Unveiling the Power of Databases in the Realm of Big Data

Comments
2 min read
When an AI Suggests DataFrame.append: Missing Pandas Deprecations in Generated Code

When an AI Suggests DataFrame.append: Missing Pandas Deprecations in Generated Code

Comments 1
3 min read
Data Engineering Isn’t About Tools — It’s About Thinking Like This

Data Engineering Isn’t About Tools — It’s About Thinking Like This

Comments
2 min read
Analysing Drivers of Digital Transformation in Corporate Innovation Capacity Using Amazon SageMaker Studio and Kaggle API

Analysing Drivers of Digital Transformation in Corporate Innovation Capacity Using Amazon SageMaker Studio and Kaggle API

Comments
2 min read
Amazon Kinesis vs Amazon MSK: The Complete Guide for Stream Processing on AWS

Amazon Kinesis vs Amazon MSK: The Complete Guide for Stream Processing on AWS

Comments
29 min read
Exploring Dynamic Return Types in PySpark pandas_udf

Exploring Dynamic Return Types in PySpark pandas_udf

Comments
2 min read
Day 30: From Zero to Production-Ready Spark Data Engineer

Day 30: From Zero to Production-Ready Spark Data Engineer

Comments
2 min read
Mastering Serverless Data Pipelines: AWS Step Functions Best Practices for 2026

Mastering Serverless Data Pipelines: AWS Step Functions Best Practices for 2026

Comments
5 min read
💀 RIP Copy-Paste: Google NotebookLM Just Killed Manual Data Entry

💀 RIP Copy-Paste: Google NotebookLM Just Killed Manual Data Entry

Comments
3 min read
Unified Data Fabric: Serverless Spark on ROSA Integrating with AWS Glue Catalog

Unified Data Fabric: Serverless Spark on ROSA Integrating with AWS Glue Catalog

Comments
18 min read
Day 27: Building Exactly-Once Streaming Pipelines with Spark & Delta Lake

Day 27: Building Exactly-Once Streaming Pipelines with Spark & Delta Lake

Comments
1 min read
Day 28: Spark Streaming Performance Tuning

Day 28: Spark Streaming Performance Tuning

Comments
1 min read
Day 29: Building a Production-Grade Real-Time ETL Pipeline with Spark & Delta

Day 29: Building a Production-Grade Real-Time ETL Pipeline with Spark & Delta

Comments
1 min read
Production AI: Monitoring, Cost Optimization, and Operations

Production AI: Monitoring, Cost Optimization, and Operations

Comments
9 min read
Building a Realistic Banking Dummy Data Generator with Bad-Data Simulation

Building a Realistic Banking Dummy Data Generator with Bad-Data Simulation

Comments
1 min read
Building an AI-Powered Customer Churn Prediction Pipeline on AWS (Step-by-Step)

Building an AI-Powered Customer Churn Prediction Pipeline on AWS (Step-by-Step)

2
Comments
5 min read
The Day Our Pipeline Went From 10 Minutes to 6 Seconds (Part 2 of 3)

The Day Our Pipeline Went From 10 Minutes to 6 Seconds (Part 2 of 3)

Comments
6 min read
Day 26: Spark Streaming Joins

Day 26: Spark Streaming Joins

Comments
1 min read
loading...