Installing node-rdkafka on M1 for use with SASL

If you're using Kafka in a Node.js app, it's likely that you'll need node-rdkafka. This is a library that wraps the librdkafka library and makes it available in Node.js. According to the project's README, "All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library."

Installation steps may vary depending on the machine you're using, and whether you need SASL/SSL (Simple Authentication and Security Layer) support. You might need this, for example, if you're connecting to Confluent Cloud, as it is a required protocol to protect resources (see docs).

Here's a short, step by step guide on installing the node-rdkafka client specifically on M1 (the first chip released by Apple specifically for Mac, article here).

Currently, as of September 30 '22, you have to do it manually. You can find the reason why in the confluent-kafka Python client README.md:

NOTE: The pre-built Linux wheels do NOT contain SASL Kerberos/GSSAPI support. If you need SASL Kerberos/GSSAPI support you must install librdkafka and its dependencies using the repositories below and then build confluent-kafka using the instructions in the "Install from source" section below.

Here are the steps you'll need to take for SSL/SASL support (run in your home directory):

git clone https://github.com/edenhill/librdkafka.git
cd ./librdkafka
./configure --install-deps --source-deps-only
make
sudo make install

Then,

1. Use homebrew to install openssl.

2. Navigate out of ./librdkafka and into your Node.js project directory.

3. Export these variables with these commands, to make sure that the path to openssl is correct:
   

   export CPPFLAGS=-I/opt/homebrew/opt/openssl@3/include
   export LDFLAGS=-L/opt/homebrew/opt/openssl@3/lib
   

4. install openssl with npm install openssl

5. Next, install node-rdkafka -- now you've got SSL support for your Node.js/Kafka project!

Hopefully this helps! If you're still running into issues, you can post a request for help in Confluent Community.