We are integrating kafka into our Django server. The messages never got send out, although the "send" method of the kafka producer is called.
Finally found the root cause, which is due to the gunicorn. The kafka producer instance is initiated with some threads in it during the server start, in the main process. Later the gunicorn fork the instance into workers, but not the threads. So there will be empty threads for the kafka producer in workers.
How to solve it? We don't want to initiate that kafka producer for each send request, which is a waste of time. So we made a lazy instantiation to get the kafka producer: if it does exist, instantiate one, and set it for our kafka producer instance; if it exist(must be instantiated by last send message call), return the kafka producer.
Top comments (0)