DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’» is a community of 963,864 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in

Discussion on: Consider SQL when writing your next processing pipeline

Collapse
 
benbirt profile image
BenBirt Author

Thanks for reading my post!

To my mind, a processing pipeline is anything that reads data from a number of source(s), joins/transforms/filters those data, and outputs the results to some number of destination(s). (Note that it is rare, but occasionally the output destination is the same as the input source.) So I would say both of your examples would qualify.

I wasn't familiar with Blaze, but having had a quick look, it does look like I am suggesting a similar approach, but indeed just going straight to SQL instead.

Collapse
 
simkimsia profile image
simkimsia

Actually when you define processing pipeline as "anything that reads data from a number of source(s), joins/transforms/filters those data, and outputs the results to some number of destination(s)."

You're talking essentially about ETL right?

Thread Thread
 
benbirt profile image
BenBirt Author

More or less, yes!