Streams are Node.js's superpower for handling large datasets efficiently. Let's dive into streams and pipelines.
Why Streams?
- Memory efficiency
- Time efficiency
- Composability
Types of Streams
- Readable
- Writable
- Duplex
- Transform
Basic Stream Usage
const fs = require('fs');
const readStream = fs.createReadStream('big.file');
const writeStream = fs.createWriteStream('output.file');
readStream.on('data', (chunk) => {
writeStream.write(chunk);
});
readStream.on('end', () => {
writeStream.end();
});
Enter Pipelines
Pipelines simplify stream composition and error handling.
const { pipeline } = require('stream/promises');
const fs = require('fs');
const zlib = require('zlib');
async function compressFile(input, output) {
await pipeline(
fs.createReadStream(input),
zlib.createGzip(),
fs.createWriteStream(output)
);
console.log('Compression complete');
}
compressFile('big.file', 'big.file.gz').catch(console.error);
Custom Transform Streams
const { Transform } = require('stream');
const upperCaseTransform = new Transform({
transform(chunk, encoding, callback) {
this.push(chunk.toString().toUpperCase());
callback();
}
});
pipeline(
process.stdin,
upperCaseTransform,
process.stdout
).catch(console.error);
Performance Tips
- Use
highWaterMarkto control buffering - Implement
stream.Readable.from()for async iterables - Leverage
stream.finished()for cleanup
Common Pitfalls
- Ignoring backpressure
- Mishandling errors
- Neglecting to end writable streams
Streams shine with large datasets or real-time data processing. Master them for scalable Node.js applications.
cheers🥂
Top comments (0)