Understanding highWaterMark in Node.js Streams

If you're working with Node.js streams, you've probably come across the highWaterMark parameter. But what is it, and why is it important? In this post, we'll explore the basics of highWaterMark and how it affects your stream pipelines.

What is highWaterMark?

Simply put, highWaterMark is a parameter that sets a limit on the amount of data that can be stored in the internal stream buffer. When the buffer reaches this limit, Node.js stops asking for more data until the buffer is emptied. This helps prevent your program from consuming too much memory and crashing.

However, it's important to remember that highWaterMark is a threshold, not a strict limit. You can keep manipulating data even when the buffer size exceeds highWaterMark. But be aware that the data will still be buffered in memory, which could cause your memory to fill up and you to lose all the benefits of Node.js streams.

If you're using the writable.write() method, it returns a boolean value that indicates whether it's safe to continue writing data to the stream. Specifically, if the internal stream buffer size is bigger than highWaterMark, it'll return false. In this case, you need to wait for the 'drain' event to be emitted before you can resume writing data to the stream. If the buffer size is smaller than highWaterMark, then it'll return true, and you can continue writing data without waiting.

Important things to keep in mind:

• highWaterMark is a parameter that sets a limit on the amount of data that can be stored in the internal stream buffer.

• This limit is a threshold, not a strict limit, so you can keep manipulating data beyond it if you need to.

• However, exceeding the threshold could cause your memory to fill up, so it's important to be mindful of your memory usage.

• If you're using the writable.write() method, check the boolean value it returns before keep writing data to the stream.

By understanding highWaterMark and how it works, you can write more efficient and robust code that takes advantage of Node.js streams. Happy streaming! 🌊

If you want to stay up to date with more Node.js tips and tricks, follow me on Twitter, And don't forget to check out the Twitter thread version of this post for a quick summary.