DEV Community

Discussion on: Processing Streaming Twitter Data using Kafka and Spark - Part 2: Creating Kafka Twitter producer

Collapse
 
ozeta profile image
Marco Carrozzo

why spark and not stream dsl/processor api?

Collapse
 
dbsheta profile image
Dhoomil B Sheta

Spark has mllib and I have worked with it so it comes as a natural choice to me when I want to analyze large data. Have you tried any machine learning use cases using streams api?

Collapse
 
ozeta profile image
Marco Carrozzo • Edited

Ha ok, I understood. Yeah I was on a big project that involved
kafka streams for data collection and manipulation + dl4j for ml in java. However the Company collapsed before I could get my hands on the machine learning task :D

However, you should look for stream dsl / processor apis. they are very powerful, native and works in a similar manner to spark topologies