Apache Storm - Slow Topology Upload

Note: This is an old post from notes. This may not be applicable anymore but sharing in case it helps someone.

Overview

Apache Storm after HDP 2.2 seems to have a hard time with large topology jars and takes a while to upload them. There have been a few reports of Storm topology jars uploading slowly. I ran into this a few years ago. The fix is to increase the nimbus.thrift.max_buffer_size setting.

Fix

Increase nimbus.thrift.max_buffer_size from the default of 1048576 to 20485760.

References

• https://mail-archives.apache.org/mod_mbox/storm-user/201403.mbox/%3CFC98EE12-4AED-4D06-9917-C449B96EB08A@gmail.com%3E
• http://stackoverflow.com/questions/27092653/storm-supervisor-connectivity-error-downloading-the-jar-from-nimbus
• https://qnalist.com/questions/4768442/nimbus-fails-after-uploading-topology-reading-too-large-of-frame-size