If you're using Kafka in a Node.js app, it's likely that you'll need
node-rdkafka. This is a library that wraps the librdkafka library and makes it available in Node.js. According to the project's README, "All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library."
Installation steps may vary depending on the machine you're using, and whether you need SASL/SSL (Simple Authentication and Security Layer) support. You might need this, for example, if you're connecting to Confluent Cloud, as it is a required protocol to protect resources (see docs).
Here's a short, step by step guide on installing the
node-rdkafka client specifically on M1 (the first chip released by Apple specifically for Mac, article here).
Currently, as of September 30 '22, you have to do it manually. You can find the reason why in the
confluent-kafka Python client README.md:
NOTE: The pre-built Linux wheels do NOT contain SASL Kerberos/GSSAPI support. If you need SASL Kerberos/GSSAPI support you must install librdkafka and its dependencies using the repositories below and then build confluent-kafka using the instructions in the "Install from source" section below.
Here are the steps you'll need to take for SSL/SASL support (run in your home directory):
git clone https://github.com/edenhill/librdkafka.git cd ./librdkafka ./configure --install-deps --source-deps-only make sudo make install
Use homebrew to install openssl.
Navigate out of
./librdkafkaand into your Node.js project directory.
Export these variables with these commands, to make sure that the path to openssl is correct:
export CPPFLAGS=-I/opt/homebrew/opt/openssl@3/include export LDFLAGS=-L/opt/homebrew/opt/openssl@3/lib
npm install openssl
node-rdkafka-- now you've got SSL support for your Node.js/Kafka project!
Hopefully this helps! If you're still running into issues, you can post a request for help in Confluent Community.
Top comments (0)