A Node.js stream can help process large files, larger than the free memory of your computer, since it processes the data in small chunks.
Streams are a built-in feature in Node.js and represent asynchronous flow of data. This article explains how to stream a large file to an HTTP response in Node.js.
Streams in Node.js
- What is a Stream in Node.js?
- Connect streams with the pipe method
- Handle stream errors
- Connect streams with the pipeline method
- How to use streams to ETL data
- Stream to an HTTP response (this article)
Send a stream back to the client as a response to an HTTP request
This article is using Node v16.14.0 and ExpressJS 4.17.2.
In previous articles I have covered a lot of basic stream handling and theoretical background. Now, let's look at some implementation. In this tutorial we are going to use streams to efficiently send a file as an HTTP response and download it.
The big advantage of streams is that you can process large files (bigger than the available memory). In general, reading a large file into memory is an inefficient use of resources.
We are going to use ExpressJS with some CSV sample data from a previous article, How to use streams to ETL data.
Overview
- Initialize project and install dependencies
- Create a readable stream from a file
- Error handling and set headers for downloading file
1. Initialize project and install dependencies
We start with creating a folder for the project
mkdir streams-http
cd streams-http
We are going to use npm packages, hence, we have to initialize the project to get a package.jsonInitialize empty project to install dependencies, add -y flag to agree to everything.
npm init -y
Install ExpressJS.
npm i express
Create a folder for sample data and add the CSV data.
mkdir data
cd data
touch sample-data.csv
Copy all sample data into sample-data.csv and save it. Use copy+paste or fs.writeFile in the REPL or with the -p flag in the terminal.
id,firstName,lastName,email,email2,randomized
100,Jobi,Taam,Jobi.Taam@yopmail.com,Jobi.Taam@gmail.com,Z lsmDLjL
101,Dacia,Elephus,Dacia.Elephus@yopmail.com,Dacia.Elephus@gmail.com,Za jfPaJof
102,Arlina,Bibi,Arlina.Bibi@yopmail.com,Arlina.Bibi@gmail.com,zmzlfER
103,Lindie,Torray,Lindie.Torray@yopmail.com,Lindie.Torray@gmail.com,ibVggFEh
104,Modestia,Leonard,Modestia.Leonard@yopmail.com,Modestia.Leonard@gmail.com," Tit KCrdh"
105,Karlee,Cornelia,Karlee.Cornelia@yopmail.com,Karlee.Cornelia@gmail.com,PkQCUXzq
106,Netty,Travax,Netty.Travax@yopmail.com,Netty.Travax@gmail.com,psJKWDBrXm
107,Dede,Romelda,Dede.Romelda@yopmail.com,Dede.Romelda@gmail.com,heUrfT
108,Sissy,Crudden,Sissy.Crudden@yopmail.com,Sissy.Crudden@gmail.com,cDJxC
109,Sherrie,Sekofski,Sherrie.Sekofski@yopmail.com,Sherrie.Sekofski@gmail.com,dvYHUJ
110,Sarette,Maryanne,Sarette.Maryanne@yopmail.com,Sarette.Maryanne@gmail.com,rskGIJNF
111,Selia,Waite,Selia.Waite@yopmail.com,Selia.Waite@gmail.com,DOPBe
112,Karly,Tjon,Karly.Tjon@yopmail.com,Karly.Tjon@gmail.com,zzef nCMVL
113,Sherrie,Berriman,Sherrie.Berriman@yopmail.com,Sherrie.Berriman@gmail.com,rQqmjw
114,Nadine,Greenwald,Nadine.Greenwald@yopmail.com,Nadine.Greenwald@gmail.com,JZsmKafeIf
115,Antonietta,Gino,Antonietta.Gino@yopmail.com,Antonietta.Gino@gmail.com,IyuCBqwlj
116,June,Dorothy,June.Dorothy@yopmail.com,June.Dorothy@gmail.com,vyCTyOjt
117,Belva,Merriott,Belva.Merriott@yopmail.com,Belva.Merriott@gmail.com,MwwiGEjDfR
118,Robinia,Hollingsworth,Robinia.Hollingsworth@yopmail.com,Robinia.Hollingsworth@gmail.com,wCaIu
119,Dorthy,Pozzy,Dorthy.Pozzy@yopmail.com,Dorthy.Pozzy@gmail.com,fmWOUCIM
120,Barbi,Buffum,Barbi.Buffum@yopmail.com,Barbi.Buffum@gmail.com,VOZEKSqrZa
121,Priscilla,Hourigan,Priscilla.Hourigan@yopmail.com,Priscilla.Hourigan@gmail.com,XouVGeWwJ
122,Tarra,Hunfredo,Tarra.Hunfredo@yopmail.com,Tarra.Hunfredo@gmail.com,NVzIduxd
123,Madalyn,Westphal,Madalyn.Westphal@yopmail.com,Madalyn.Westphal@gmail.com,XIDAOx
124,Ruthe,McAdams,Ruthe.McAdams@yopmail.com,Ruthe.McAdams@gmail.com,iwVelLKZH
125,Maryellen,Brotherson,Maryellen.Brotherson@yopmail.com,Maryellen.Brotherson@gmail.com,nfoiVBjjqw
126,Shirlee,Mike,Shirlee.Mike@yopmail.com,Shirlee.Mike@gmail.com,MnTkBSFDfo
127,Orsola,Giule,Orsola.Giule@yopmail.com,Orsola.Giule@gmail.com,VPrfEYJi
128,Linzy,Bennie,Linzy.Bennie@yopmail.com,Linzy.Bennie@gmail.com,ZHctp
129,Vanessa,Cohdwell,Vanessa.Cohdwell@yopmail.com,Vanessa.Cohdwell@gmail.com,RvUcbJihHf
130,Jaclyn,Salvidor,Jaclyn.Salvidor@yopmail.com,Jaclyn.Salvidor@gmail.com,gbbIxz
131,Mildrid,Pettiford,Mildrid.Pettiford@yopmail.com,Mildrid.Pettiford@gmail.com,snyeV
132,Carol-Jean,Eliathas,Carol-Jean.Eliathas@yopmail.com,Carol-Jean.Eliathas@gmail.com,EAAjYHiij
133,Susette,Ogren,Susette.Ogren@yopmail.com,Susette.Ogren@gmail.com," BhYgr"
134,Farrah,Suanne,Farrah.Suanne@yopmail.com,Farrah.Suanne@gmail.com,hYZbZIc
135,Cissiee,Idelia,Cissiee.Idelia@yopmail.com,Cissiee.Idelia@gmail.com,PNuxbvjx
136,Alleen,Clara,Alleen.Clara@yopmail.com,Alleen.Clara@gmail.com,YkonJWtV
137,Merry,Letsou,Merry.Letsou@yopmail.com,Merry.Letsou@gmail.com,sLfCumcwco
138,Fanny,Clywd,Fanny.Clywd@yopmail.com,Fanny.Clywd@gmail.com,Go kx
139,Trixi,Pascia,Trixi.Pascia@yopmail.com,Trixi.Pascia@gmail.com,lipLcqRAHr
140,Sandie,Quinn,Sandie.Quinn@yopmail.com,Sandie.Quinn@gmail.com,KrGazhI
141,Dania,Wenda,Dania.Wenda@yopmail.com,Dania.Wenda@gmail.com,CXzs kDv
142,Kellen,Vivle,Kellen.Vivle@yopmail.com,Kellen.Vivle@gmail.com,RrKPYqq
143,Jany,Whittaker,Jany.Whittaker@yopmail.com,Jany.Whittaker@gmail.com,XAIufn
144,Lusa,Fillbert,Lusa.Fillbert@yopmail.com,Lusa.Fillbert@gmail.com,FBFQnPm
145,Farrah,Edee,Farrah.Edee@yopmail.com,Farrah.Edee@gmail.com,TrCwKb
146,Felice,Peonir,Felice.Peonir@yopmail.com,Felice.Peonir@gmail.com,YtVZywf
147,Starla,Juan,Starla.Juan@yopmail.com,Starla.Juan@gmail.com,aUTvjVNyw
148,Briney,Elvyn,Briney.Elvyn@yopmail.com,Briney.Elvyn@gmail.com,tCEvgeUbwF
149,Marcelline,Ricarda,Marcelline.Ricarda@yopmail.com,Marcelline.Ricarda@gmail.com,sDwIlLckbd
150,Mureil,Rubie,Mureil.Rubie@yopmail.com,Mureil.Rubie@gmail.com,HbcfbKd
151,Nollie,Dudley,Nollie.Dudley@yopmail.com,Nollie.Dudley@gmail.com,EzjjrNwVUm
152,Yolane,Melony,Yolane.Melony@yopmail.com,Yolane.Melony@gmail.com,wfqSgpgL
153,Brena,Reidar,Brena.Reidar@yopmail.com,Brena.Reidar@gmail.com,iTlvaS
154,Glenda,Sabella,Glenda.Sabella@yopmail.com,Glenda.Sabella@gmail.com,zzaWxeI
155,Paola,Virgin,Paola.Virgin@yopmail.com,Paola.Virgin@gmail.com,gJO hXTWZl
156,Aryn,Erich,Aryn.Erich@yopmail.com,Aryn.Erich@gmail.com,qUoLwH
157,Tiffie,Borrell,Tiffie.Borrell@yopmail.com,Tiffie.Borrell@gmail.com,cIYuVMHwF
158,Anestassia,Daniele,Anestassia.Daniele@yopmail.com,Anestassia.Daniele@gmail.com,JsDbQbc
159,Ira,Glovsky,Ira.Glovsky@yopmail.com,Ira.Glovsky@gmail.com,zKITnYXyhC
160,Sara-Ann,Dannye,Sara-Ann.Dannye@yopmail.com,Sara-Ann.Dannye@gmail.com,wPClmU
161,Modestia,Zina,Modestia.Zina@yopmail.com,Modestia.Zina@gmail.com,YRwcMqPK
162,Kelly,Poll,Kelly.Poll@yopmail.com,Kelly.Poll@gmail.com,zgklmO
163,Ernesta,Swanhildas,Ernesta.Swanhildas@yopmail.com,Ernesta.Swanhildas@gmail.com,tWafP
164,Giustina,Erminia,Giustina.Erminia@yopmail.com,Giustina.Erminia@gmail.com,XgOKKAps
165,Jerry,Kravits,Jerry.Kravits@yopmail.com,Jerry.Kravits@gmail.com,olzBzS
166,Magdalena,Khorma,Magdalena.Khorma@yopmail.com,Magdalena.Khorma@gmail.com,BBKPB
167,Lory,Pacorro,Lory.Pacorro@yopmail.com,Lory.Pacorro@gmail.com,YmWQB
168,Carilyn,Ethban,Carilyn.Ethban@yopmail.com,Carilyn.Ethban@gmail.com,KUXenrJh
169,Tierney,Swigart,Tierney.Swigart@yopmail.com,Tierney.Swigart@gmail.com,iQCQJ
170,Beverley,Stacy,Beverley.Stacy@yopmail.com,Beverley.Stacy@gmail.com,NMrS Zpa f
171,Ida,Dex,Ida.Dex@yopmail.com,Ida.Dex@gmail.com,hiIgOCxNg
172,Sam,Hieronymus,Sam.Hieronymus@yopmail.com,Sam.Hieronymus@gmail.com,dLSkVe
173,Lonnie,Colyer,Lonnie.Colyer@yopmail.com,Lonnie.Colyer@gmail.com,ZeDosRy
174,Rori,Ethban,Rori.Ethban@yopmail.com,Rori.Ethban@gmail.com,SXFZQmX
175,Lelah,Niles,Lelah.Niles@yopmail.com,Lelah.Niles@gmail.com,NwxvCXeszl
176,Kathi,Hepsibah,Kathi.Hepsibah@yopmail.com,Kathi.Hepsibah@gmail.com,SOcAOSn
177,Dominga,Cyrie,Dominga.Cyrie@yopmail.com,Dominga.Cyrie@gmail.com,IkjDyuqK
178,Pearline,Bakerman,Pearline.Bakerman@yopmail.com,Pearline.Bakerman@gmail.com,vHVCkQ
179,Selma,Gillan,Selma.Gillan@yopmail.com,Selma.Gillan@gmail.com,hSZgpBNsw
180,Bernardine,Muriel,Bernardine.Muriel@yopmail.com,Bernardine.Muriel@gmail.com,AnSDTDa U
181,Ermengarde,Hollingsworth,Ermengarde.Hollingsworth@yopmail.com,Ermengarde.Hollingsworth@gmail.com,IYQZ Nmv
182,Marguerite,Newell,Marguerite.Newell@yopmail.com,Marguerite.Newell@gmail.com,kSaD uaHH
183,Albertina,Nisbet,Albertina.Nisbet@yopmail.com,Albertina.Nisbet@gmail.com,Y jHyluB
184,Chere,Torray,Chere.Torray@yopmail.com,Chere.Torray@gmail.com,loElYdo
185,Vevay,O'Neill,Vevay.O'Neill@yopmail.com,Vevay.O'Neill@gmail.com,uLZSdatVn
186,Ann-Marie,Gladstone,Ann-Marie.Gladstone@yopmail.com,Ann-Marie.Gladstone@gmail.com,fwKlEksI
187,Donnie,Lymann,Donnie.Lymann@yopmail.com,Donnie.Lymann@gmail.com,deBrqXyyjf
188,Myriam,Posner,Myriam.Posner@yopmail.com,Myriam.Posner@gmail.com,gEMZo
189,Dale,Pitt,Dale.Pitt@yopmail.com,Dale.Pitt@gmail.com,OeMdG
190,Cindelyn,Thornburg,Cindelyn.Thornburg@yopmail.com,Cindelyn.Thornburg@gmail.com,kvhFmKGoMZ
191,Maisey,Hertzfeld,Maisey.Hertzfeld@yopmail.com,Maisey.Hertzfeld@gmail.com,OajjJ
192,Corina,Heisel,Corina.Heisel@yopmail.com,Corina.Heisel@gmail.com,luoDJeHo
193,Susette,Marcellus,Susette.Marcellus@yopmail.com,Susette.Marcellus@gmail.com,AXHtR AyV
194,Lanae,Sekofski,Lanae.Sekofski@yopmail.com,Lanae.Sekofski@gmail.com,FgToedU
195,Linet,Beebe,Linet.Beebe@yopmail.com,Linet.Beebe@gmail.com,DYGfRP
196,Emilia,Screens,Emilia.Screens@yopmail.com,Emilia.Screens@gmail.com,LXUcleSs
197,Tierney,Avi,Tierney.Avi@yopmail.com,Tierney.Avi@gmail.com,VegzbHH
198,Pollyanna,Thar,Pollyanna.Thar@yopmail.com,Pollyanna.Thar@gmail.com,GjYeEGK
199,Darci,Elephus,Darci.Elephus@yopmail.com,Darci.Elephus@gmail.com,DaQNdN
Create an index.js file (in root folder), which will be the main file for our code.
cd .. # if you are in the data folder
touch index.js
2. Create a readable stream from a file
First, we are going to create a basic express server, which listens on port 3000. Open index.js in your IDE and add the following code.
const express = require('express');
const app = express();
const PORT = 3000;
app.listen(PORT, () =>
console.log(`Server listening on port ${PORT}`),
);
Start the node server with running node index.js in the project root folder. You should see Server listening on port 3000 in your terminal. Terminate the server with CTRL+C.
Let's make another route to download the csv file. Add a GET handler with the route /get-data to index.js.
app.get('/get-data', (req, res, next) => {
// TBD
});
Now we have a route on which we are going to download the file in the end. We can proceed to create a readable stream to read the file. For creating a stream we have to import the fs module.
const fs = require('fs');
Create a stream to read file sample-data.csv.
app.get('/dl', (req, res, next) => {
const fileStream = fs.createReadStream(
`${__dirname}/data/sample-data.csv`,
);
});
The constant fileStream represents the data stream from the file. This stream we directly pipe into the response.
app.get('/dl', (req, res, next) => {
const fileStream = fs.createReadStream(
`${__dirname}/data/sample-data.csv`,
);
fileStream.pipe(res);
});
Start the server again node index.js. And open a web browser with http://localhost:3000/get-data. You should see the csv file.
3. Error handling and set headers for downloading file
Now we are sending a file as a stream, but it should download. Let's make it happen. As always, we have to think of error handling first. What could go wrong? The file could not exist.
To handle this, we have to listen to the open event on the readStream to check, if the file exists and only pipe the stream if it does.
app.get('/get-data', (req, res, next) => {
const fileStream = fs.createReadStream(
`${__dirname}/data/sample-data.csv`,
);
fileStream.on('open', () => {
fileStream.pipe(res);
});
});
And if the file doesn't exist, we return the error as response.
app.get('/get-data', (req, res, next) => {
const fileStream = fs.createReadStream(
`${__dirname}/data/sample-data.csv`,
);
fileStream.on('open', () => {
fileStream.pipe(res);
});
fileStream.on('error', err => {
next(err);
});
});
In most cases the pipeline method should be used, but pipeline destroys streams when an error occurs, and we would not be able to send a response back. Hence, for this use case manually error handling is acceptable.
At the moment the browser is displaying the file inline, it's loading the file in the browser. To tell the browser to download the file, we have to set:
- a Content-Type header on the response to specify what file we are sending, and
- the Content-Disposition header to attachment with a file name.
Express has a method for this attachment("FILENAME"). It sets the HTTP response Content-Disposition header field to “attachment”, and if a filename is given, then it sets the Content-Type based on the extension name via res.type(), and sets the Content-Disposition “filename=” parameter.
When using res.attachment('streamed-sample-data'), the content-type header will be set to text/csv and the content-disposition to the streamed-sample-data.csv.
app.get('/get-data', (req, res, next) => {
const fileStream = fs.createReadStream(
`${__dirname}/data/sample-data.csv`,
);
fileStream.on('open', () => {
res.attachment('streamed-sample-data.csv');
fileStream.pipe(res);
});
fileStream.on('error', err => {
next(err);
});
});
Restart your node server and go to http://localhost:3000/get-data. The file streamed-sample-data.csv should be downloaded.
TL;DR
- Error handling has to be done always, especially when working with streams.
- Error handling should be done in most cases with
pipeline, only if the stream should not be destroyed, manually error handling is necessary. - The express method
attachment()is used to add Content-Type and Content-Disposition headers to a response. - For the future, the Express framework has a method for sending files via stream sendFile().
Thanks for reading and if you have any questions , use the comment function or send me a message @mariokandut.
If you want to know more about Node, have a look at these Node Tutorials.
References (and Big thanks):
ExpressJS,HeyNode,Node.js - Streams,MDN - Streams,MDN - HTTP
Top comments (1)
I've started recently dig into C++ . I am JavaScript (Node.js) developer ,can you tell me what are capstone cases I should touch in C(++) lang family , to get closer to Node.js nature, File System (Linux/Unix), Stream API, any other suggestions or roadmaps , Mario ? Thanks !
p.s. your tuts always smells like fresh loaf of bread, appreciated it !