Demystifying Buffers in Node.js

Table of Contents

1. What is Buffer?
2. What is Character Encoding?
3. Buffer in Node.js

• 3a. Buffer.alloc()
• 3b. Buffer.from()
• 3c. Buffer.allocUnsafe()
• 3d. Buffer.compare()
• 3e. Buffer.toString()

In this blog series, we'll unravel the mystery surrounding buffers, exploring their crucial role in optimising everything from streaming to web browsing.

1. What is Buffer?

Buffers --> These are temporary storage area in memory used to hold data while it's being transferred from one place to another.
In Node.js, Buffer objects are used to represent a fixed-length sequnce of bytes.

Let us say, we want to copy a file of size 10GB, from one device to another device. The device speed may vary, which would lead to various problems like :

• Memory Overload --> Reading the entire file into memory at once could easily overwhelm the available memory resource.

• Performance Degradation --> As the system would spend much of it's time in I/O operation.

• Data corruption --> If error occurs during file transfer, there would be no way to maintain data integrity and some data might corrupted.

2. What is Character Encoding?

Character Encoding --> It is the process of representing characters from a written language, numbers, and symbols as binary data (bits and bytes) for storage or transmission in a digital system.

It is used to represent a sequence of bits into a character. For eg :-
'11001001' might represent character 'a' in some encoding while 'd' in another encoding.

Here are some basic definition :-

• Character Set --> A character set is a collection of characters, such as letters, digits, punctuation marks, and special symbols, used in written communication.

• Binary Representation --> Since computer can only understand 0's and 1's, each character in character set in give a unique binary code.

• Code Pages And Standards --> Different character encoding use different encoding schemes for encoding. For eg -> ASCII uses 7-bits to represent 128 characters and UTF-8 uses 8-bits to represent characters.

Some examples of encoding are :- UTF-8, UTF-16, ASCII etc. In Node.js default encoding is UTF-8.

3. Buffer in Node.js

Buffer objects are used to represent a fixed-length sequence of bytes. The Buffer class is a subclass of JavaScript's Uint8Array class and extends it with methods that cover additional use cases.

a. Buffer.alloc(size,fill,encoding) --> It creates a new buffer with given size.

• size : The desired length of the new Buffer.
• fill : A value to pre-fill the new Buffer with. Default: 0.
• encoding : Encoding to be used. Default 'UTF-8'.

const { Buffer } = require('node:buffer');
// Buffer is available globally but it is 
// recommended to import explicitly


const buff1 = Buffer.alloc(5);
console.log(buff1);
// Buffer allocated 5 size filled with zero
// output is represented as hexadecimal number
// OUTPUT : <Buffer 00 00 00 00 00>


const buff2 = Buffer.alloc(5,1);
console.log(buff2);
// buffer allocated 5 size filled with 1
// OUTPUT : <Buffer 01 01 01 01 01>

b. Buffer.from(array or string) --> It takes array or string as an argument and creates the buffer size and value accordingly.

const { Buffer } = require('node:buffer');

const buff3 = Buffer.from('hello','utf-8');
console.log(buff3);
// buffer size allocated for string 'hello'
// with character encoding 'utf-8'(default)
// OUTPUT : <Buffer 68 65 6c 6c 6f>

const buff4 = Buffer.from([0x7f,17,0]);
console.log(buff4);
// buffer size and value according to array
// array elements can be number(integer OR hexadecimal)
// OUTPUT : <Buffer 7f 11 00>

c. Buffer.allocUnsafe(size) --> It creates a buffer of given size but with random values.( NOT RECOMMENDED ). It is little faster than "Buffer.alloc()" method

const { Buffer } = require('node:buffer');

const buff5 = Buffer.allocUnsafe(5);
console.log(buff5);
// faster than '.alloc()' method
// OUTPUT : <Buffer 60 67 6c 6d 6e>

d. Buffer.compare(buff1,buff2) --> Mainly used when we want to sort array of Buffers. It's output could vary :-

• 0 is returned if buff2 is the same as buff1
• 1 is returned if buff2 should come before buff1 when sorted.
• -1 is returned if buff2 should come after buff1 when sorted.

const { Buffer } = require('buffer');

const buff6 = Buffer.from('0123');
const buff7 = Buffer.from('1234');

console.log(Buffer.compare(buff6,buff7));
// OUTPUT : -1

e. Buffer.toString(encoding) --> This method converts a given buffer to string with given encoding. DEFAULT ENCODING : UTF-8

const { Buffer } = require('buffer');

const buff8 = Buffer.from('hello');
console.log(buff8.toString());
// OUTPUT : hello