DEV Community

loading...

Using DJL.AI For Deep Learning Based Sentiment Analysis in NiFi DataFlow

tspannhw profile image Timothy Spann Originally published at datainmotion.dev on ・2 min read

Using DJL.AI For Deep Learning Based Sentiment Analysis in NiFi DataFlow

Introduction:

I will be talking about this processor at Apache Con @ Home 2020 in my "Apache Deep Learning 301" talk with Dr. Ian Brooks.

Sometimes you want your Deep Learning Easy and in Java, so let's do that with DJL in a custom Apache NiFi processor running in CDP Data Hubs.

Grab the Source:

https://github.com/tspannhw/nifi-djlsentimentanalysis-processor

Grab the Recent Release NAR to install to your NiFi lib directories:

https://github.com/tspannhw/nifi-djlsentimentanalysis-processor/releases/tag/1.2

Example Run

probnegative

0.99

No value set

probnegativeperc

99.44

No value set

probpositive

0.01

No value set

probpositiveperc

0.56

No value set

rawclassification

[class: "Negative", probability: 0.99440, class: "Positive", probability: 0.00559]

Demo Data Source

https://newsapi.org/v2/everything?q=cloudera&apiKey=REGISTERFORAKEY

Reference:

Deep Learning Note:

The pretrained model is DistilBERT model trained by HuggingFace using PyTorch.

Tip

Make sure you have 1-2 GB of RAM extra for your NiFi instance for running each DJL processor. If you have a lot of text, run more nodes and/or RAM. Make sure you have at least 8 cores per Deep Learning process. I prefer JDK 11 for this.

See Also: https://www.datainmotion.dev/2019/12/easy-deep-learning-in-apache-nifi-with.html

Discussion (0)

pic
Editor guide