DEV Community

Discussion on: Consider SQL when writing your next processing pipeline

Collapse
 
thorstenhirsch profile image
Thorsten Hirsch

much more work being done inside data warehouses

Let's not hope so. We've been there before (pre-cloud era) and saw that mixing the data persistency layer with the data transformation layer was a bad idea. Databases and data warehouses might be sufficient for a few nightly batch jobs, but it's a very restrictive runtime environment for data processing, hard to scale, hard to debug, hard to monitor. Modern data processing engines also need to support data streams (e.g. Apache Kafka) that need to be filtered in realtime.