DEV Community

loading...
Cover image for Demystifying  Streams in Nodejs

Demystifying Streams in Nodejs

CALVIN JOB PURAM
I thrive to design and develop ideas into project that makes life easy, solve problems and implement innovations.
・6 min read

Streams are yet another fundamental concept in Nodejs so let's now learn all about them. First of all, what are streams? with streams, we can process (read and write) data piece by piece (chunks), without completing the whole read or write operation, and without keeping all the data in memory. For example, when we read a file using streams, we read part of the data, do something with it, then free our memory and repeat this until the entire file has been processed. Think of Youtube and Netflix, which are both called streaming companies because they stream videos using the same principle so instead of waiting until the entire video file load, the processing is done piece by piece or in chunks so that you can start watching before the entire file has been downloaded. So the principle here is not just about Nodejs but universal to computer science in general.

  • Streams are the perfect candidates for handling large volumes of data like videos or data that we are receiving piece by piece from an external source.
  • There is more efficient data processing in terms of memory (no need to keep all the data in memory) and time (we don't have to wait until all the data is available before we can do something).

How To Implement Streams In Nodejs

In Nodejs, there are four fundamental types of streams:

  • readable streams
  • writable streams
  • duplex streams
  • transform streams

The readable and writable streams are the most important ones and in this article, I will focus on these two.

Readable Streams

These are the type of streams from which we can read (consume) data. Streams are everywhere in Nodejs core modules just like events that I talk about. For example, the data that comes in when an Http server gets a request is actually readable streams so all the data that is sent with the request comes in a piece by piece and not in one large piece. Also, another example from the file system is that we can read a file piece by piece by using a read streams from the fs module which can be useful for the large text file.

Streams are instances of the event emitter class (all streams can emit and listen to events). Readable streams can emit and we can listen to many different events but the most important ones are the data and the end event. Also, we have important functions that we can use on streams and in the case of readable streams, the most important ones are the pipe() and the read() function which you will see in practice later. The pipe function allows us to pluck streams together passing data from one stream to another without having to worry about events at all.

Writable Streams

these are the type of streams to which we can write data. Basically the opposite of readable streams. For example, the Http response that we can send back to the client which is actually writable streams that we can write data into. The idea is when we want to send data we have to write it somewhere and that somewhere is the writable streams. For example, if you want to send a large video file to a client, we will stream that result just like Netflix or Youtube. Also about events, the two most important ones are the drain and the finish events and also the most important functions are the write() and the end() functions which you will see in practice later.

Duplex Streams

These are the type of streams that are both readable and writable at the same time a good example is the web socket from the net module. A web socket is basically a communication channel between client and server that works in both directions and stays open while the connection is established.

Transform Streams

Transform streams are duplex streams that transform data as it is written or read (it can modify or transform a data). a good example is the Zlip core module which compresses data (it uses transform streams).

Note: these events and functions mentioned above in both readable and writable streams are for consuming streams that are already implemented. Nodejs implement the Http requests as streams and we can consume (use) them using the events and functions that are available for each type of streams. we can, of course, implement our own streams and then consume them using the same events and functions but the important thing is to know how to consume streams than implement them.

Streams in Practice

Let's now work with streams and I will start by creating a folder called streams and a file called app.js

mkdir stream
touch app.js
Enter fullscreen mode Exit fullscreen mode

Next, open the file in your favorite editor and we will start by reading a large text file and send it to the client so how do we do that? well, there are multiple ways and we are going to explore a few of them starting from the most basic ones and moving up.

In the app.js file we first require the fs module and the http module then we create a server and listen to an incoming request and specify a callback. So the first solution we will use is the easiest one we first read a file into a variable and then once that is done we send it to the client. Also, we will create another file that will contain some text which we will read from and send to the client.

  // app.js file
  const fs = require('fs');
  const server = require('http').createServer();

  server.on('request', (req, res) => {
  // first solution
   fs.readFile('text-file.txt', (err, data) => {
     if (err) console.log(err)
     res.end(data)
   });

  });

Enter fullscreen mode Exit fullscreen mode

This is the simplest solution but before we can test this it, we need to start a server

  // app.js file
  const fs = require('fs');
  const server = require('http').createServer();

  server.on('request', (req, res) => {
  // first solution
   fs.readFile('text-file.txt', (err, data) => {
     if (err) console.log(err)
     res.end(data)
   });

  });

  server.listen(8000, () => {
    console.log('listening...');
  })

Enter fullscreen mode Exit fullscreen mode

Now this works very fine if we run node app.js it will output all the data received from text-file.txt in your browser but the problem with this implementation is that nodejs loads all the entire text file into memory only when that is ready before it can then send the data. But when the file is too large and there are tons of requests hitting the server, you will quickly lose resources and everything will crash. the above solution only works when we are creating something local and not a production-ready app.

In the second solution, we will use streams. So the idea is we don't need to read the text file into a variable so instead of reading a data and storing it into a variable, we will just create readable streams then as we receive each chunk of data we will then send the response as writable streams.


  // app.js file
  const fs = require("fs");
  const server = require("http").createServer();

  server.on("request", (req, res) => {
  // second solution
  const readable = fs.createReadStream("text-file.txt");
  readable.on("data", chunk => {
    res.write(chunk);
  });

  readable.on("end", () => {
    res.end();
  });
});

server.listen(8000, () => {
  console.log("listening...");
});
Enter fullscreen mode Exit fullscreen mode

This is how we create readable streams that emit data piece by piece, listen to them and write this data we are receiving from the readable streams to the writable streams. Remember the response is a writable streams so we can use the write() function on it and in this case, we are streaming the data to the client bit by bit so when that is completed we call the end event.

We can also use the error event on readable streams

  // app.js file
const fs = require("fs");
const server = require("http").createServer();

server.on("request", (req, res) => {
  // second solution
  const readable = fs.createReadStream("textt-file.txt");
  readable.on("data", chunk => {
    res.write(chunk);
  });

  readable.on("end", () => {
    res.end();
  });

  readable.on("error", err => {
    console.log(err);
    res.statusCode = 500;
    res.end("file not found");
  });
});

server.listen(8000, () => {
  console.log("listening...");
});

Enter fullscreen mode Exit fullscreen mode

Now, this approach works perfectly but there is a problem, and the problem is that our readable stream is much faster in reading the data from the text file than sending the result with the response writable streams over the network and this will overwhelm the response streams and it can't handle the incoming data and this problem is call backpressure and is a real problem that can happen in real applications. so backpressure happens when the response cannot send data nearly as fast as it is receiving it. we have to fix this issue so let's do just that.

  // app.js file
const fs = require("fs");
const server = require("http").createServer();

server.on("request", (req, res) => {
  // third solution
  const readable = fs.createReadStream("text-file.txt");
  readable.pipe(res);
});

server.listen(8000, () => {
  console.log("listening...");
});

Enter fullscreen mode Exit fullscreen mode

So the solution here is to use the pipe() function which is available on all readable streams and it allows us to pipe the output of readable streams right into the input of writable streams and that will the fix the problem of back pressure because it will automatically handle the speed of the data coming in and the data going out. The aim of this article is just to introduce you to streams be sure to check the Nodejs documentation for more on streams.

Discussion (0)

Forem Open with the Forem app