DEV Community

loading...

Discussion on: Consider SQL when writing your next processing pipeline

Collapse
benbirt profile image
BenBirt Author

Honestly, anything that makes it easier/simpler to write pipelines is good in my book! (And Spark's SQL APIs definitely count there.)

That said, at least for data that can easily be brought into a single data warehouse, my preference is still to get rid of that extra layer - reducing complexity - and let the highly scalable warehouse do the grunt work.