DEV Community

loading...

Discussion on: Consider SQL when writing your next processing pipeline

Collapse
benbirt profile image
BenBirt Author

Thanks for reading my post!

To my mind, a processing pipeline is anything that reads data from a number of source(s), joins/transforms/filters those data, and outputs the results to some number of destination(s). (Note that it is rare, but occasionally the output destination is the same as the input source.) So I would say both of your examples would qualify.

I wasn't familiar with Blaze, but having had a quick look, it does look like I am suggesting a similar approach, but indeed just going straight to SQL instead.

Collapse
simkimsia profile image
simkimsia

Actually when you define processing pipeline as "anything that reads data from a number of source(s), joins/transforms/filters those data, and outputs the results to some number of destination(s)."

You're talking essentially about ETL right?

Thread Thread
benbirt profile image
BenBirt Author

More or less, yes!