Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink
Article 6 - This.
*Article 5 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html
*Article 4 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html
*Article 3 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html
*Article 2 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html
Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html
*Source Code: * https://github.com/tspannhw/flume-to-nifi
This is a simple use case of being a gateway between REST API and Kafka. We can do a lot more than that in NiFi. We can be a Kafka Consumer and Producer as well as POST REST calls and receive any REST calls on configurable ports. All with No Code.
NiFi can act as a listener for HTTP Requests and provide HTTP Responses in a scriptable full Web Server mechanism with JETTY. Or it can listen for HTTP REST calls on a port and route those files anywhere. https://community.cloudera.com/t5/Community-Articles/Parsing-Web-Pages-for-Images-with-Apache-NiFi/ta-p/248415 . We can also do websockets https://community.cloudera.com/t5/Community-Articles/An-Example-WebSocket-Application-in-Apache-NiFi-1-1/ta-p/248598. https://community.cloudera.com/t5/Community-Articles/Accessing-Feeds-from-EtherDelta-on-Trades-Funds-Buys-and/ta-p/248316
It is extremely easy to do this in NiFi.
Kafka Consumer to REST POST
HTTP REST to Kafka Producer
Full Monitoring on Apache NiFi
A Very Common Use Case: Ingesting Stock Feeds From REST to Kafka
References
- https://www.datainmotion.dev/2019/08/creating-apache-kafka-topics.html
- https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221
- https://community.cloudera.com/t5/Community-Articles/IoT-Edge-Use-Cases-with-Apache-Kafka-and-Apache-NiFi-MiniFi/ta-p/249232
- https://dataworkssummit.com/nosql-day-2019/session/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi/
- https://community.cloudera.com/t5/Community-Articles/NiFi-REST-API-FlowFile-Count-Monitoring/ta-p/244208
- https://dzone.com/articles/simple-apache-nifi-operations-dashboard-part-2-spr
- https://dzone.com/articles/building-a-custom-apache-nifi-operations-dashboard
- https://www.progress.com/tutorials/jdbc/connect-and-import-data-from-any-rest-api-in-apache-nifi
Top comments (0)