Sizing Your Apache NiFi Cluster For Production Workloads
Cloudera Flow Management provides an enterprise edition of support Apache NiFi managed by Cloudera Manager. The official documentation provides a great guide for sizing your cluster.
https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-nifi-sizing.html
If the use case fits, NiFi Stateless Engine may fit and perform better utilizing no disk.
Check out that heap usage and utilization, you may need to increase. 24-32 Gigabytes of RAM is a nice sweet spot for most instances.
Check out how your nodes, threads and queues are doing. If queue is not processing fast or thread count is high, you may need more cores, RAM or nodes.
When you are managing your cluster in Cloudera Manager, make sure you increase the default JVM memory for Apache NiFi. 512MB is not going to cut it for anything but single user development.
Do this correctly and process a billion events!!! https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/. - Notice the hardware and performance sections of that article
General tips:
Make sure you use SSD for Provenance and other repositories. Faster disk, happier user. https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-disk-configuration.html
Monitor your flows to see how much resources you need: https://www.datainmotion.dev/2020/07/report-on-this-apache-nifi-1114-monitor.html.
Three or more nodes plus external Zookeepers. https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-cluster-layout.html
Use Records, if it's semistructured GrokReader can help. https://www.nifi.rocks/record-path-cheat-sheet/ If it's CSV, JSON, XML, Parquet, Logs then use Readers and writers. They are much faster, easier and cleaner.
Good clean code is better than spaghetti. https://docs.cloudera.com/cfm/2.0.1/nifi-tuning/topics/cfm-tuning-your-data-flow.html. Saw no to bad coding: https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html.
Recommendations: https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-recommendations.html
Minimize use of CPU or Memory intensive processors (or make a not of them during sizing): https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-resource-intensive-processors.html
There are a few decisions to make on repositories, talk to your Cloudera friends. https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-configuration-best-practices/content/configuration-best-practices.html
Top comments (0)