Using Binary Data In JavaScript

This article discusses working with raw binary data in JavaScript.

Table Of Contents

• The Basics
• Bitwise Operations
• Encoding And Decoding Data
• Buffers
• Typed Arrays
• DataViews
• Applications

The Basics

Binary data in JavaScript comes in a few forms:

• Numbers
  • A single number variable can be seen as binary data.
• Buffers
  • There are two types of buffers ArrayBuffer and SharedArrayBuffer. Both are essentially fixed arrays of binary data.
  • Node has an object called Buffer which is like ArrayBuffer but slightly different.
• Blobs
  • You can extract an ArrayBuffer from a Blob.

To work with this binary data we have a few options:

• Bitwise Operations
  • We can use binary math to extract and encode meaningful data.
• TypedArrays
  • We can use TypedArrays to read the data as fixed number arrays.
• DataViews
  • We can use DataViews to work with and extract specific bytes.

Bitwise Operations

Bitwise operations allow us to change bits of numbers and create new numbers from two inputs. They are useful for encoding and decoding data.

Limitations
When using bitwise operations on numbers all numbers get converted into 32 bit signed integers even though all numbers are stored as 64 bit floats. However, you can use the BigInt number type and work with all 64 bits of the number.

const bigInt = (2n ** 64n) >> 32n;
4294967296n
const normalNumber = (2 ** 64) >> 32;
0

AND

The & operation sets a bit to 1 if only the bit it is being compared to is also 1.

const v1 = 0b1111;
const v2 = 0b1011;
const result = v1 & v2;
0b1011

OR

The | operation sets a bit to 1 if one of the bits being compared is 1.

const v1 = 0b1111;
const v2 = 0b1011;
const result = v1 | v2;
0b1111

XOR

The ^ operation sets a bit to 1 if only one of the bits being compared is 1 otherwise it sets it to 0.

const v1 = 0b1111;
const v2 = 0b1011;
const result = v1 ^ v2;
0b0100

NOT

The ~ operation flips all bits.

const result = ~0b1011;
0b0100

Zero Fill Left Shift

The << operation shifts bits to the left by a given amount.

const result = 0b0011 << 2;
0b1100

Signed Right Shift

The >>> operation shifts bits to the right by a given amount and preserves the sign if the number is negative.

const result = -5 >> 2;
-2

This operation can also be used for flooring a number:

const result = 1.2 >> 0;
1

Zero Fill Right Shift

The >>> operation shifts bits to the right by a given amount.

const result1 = -5 >>> 2;
1073741822
const result2 = 0b1100 >>> 2;
0b0011

Encoding And Decoding Data

Here is a basic example showing how you could use a single number to store 8 numbers with a range of 0 to 15.

const mask = 0xf;
const bitSize = 4;
const getValue = (data: number, index: number) => {
    index *= bitSize;
    return ((mask << index) & data) >>> index;
}
const setValue = (data: number, index: number, value: number) => {
    index *= bitSize;
    return (data & ~(mask << index)) | ((value & mask) << index)
}

So, we have two functions here. The getValue function will take the encoded number and return the value at the given index. While the setValue function will take the encoded number and encode the provided value at the given index and then return the new encoded number.

Here is a basic example of using it:

let v = 0;
v = setValue(v,4,12);
//v is now 786432
getValue(v,4);
//value is 12

Now, I am going to break down what is exactly going on line by line.

First thing we do is define as mask and bit size:

//0xf = 0b1111 = 15
const mask = 0xf;
const bitSize = 4;

This mask here is in the hex number format. We are defining the mask as 4 bits that are all 1. This will define the range of the numbers stored as 0 though 15. We also declare bitSize so we can access the values stored in the number more like an array.

Then we define the getValue function. Here is the logic of the function expanded out:

const getValue = (data: number, index: number) => {
    //get the bit index
    index *= bitSize;
    //create a new mask by right shifting by the bit index
    const newMask = mask << index;
    //remove all other data expect the data at the given index
    const value = newMask & data;
    //right shift the number to get the actual value
    return value >>> index;
}

And finally we define the setValue function. Here is the logic of the function expanded out:

const setValue = (data: number, index: number, value: number) => {
    index *= bitSize;
    //create a new mask by right shifting by the bit index
    let newMask = (mask << index);
    //invert the new mask
    newMask = ~newMask;
    //remove the data at the given index
    data = data & newMask
    //remove all bits that do not fit within the mask
    value = value & mask;
    //left shift the value to the index it will be stored at
    value = value << index;
    //or the data and the value thus encoding the value at the index
    return data | value;
}

Buffers

JavaScript uses buffers or "byte arrays" to work with raw binary data. You can not directly access the data. You need to use another object such as a Typed Array to work with the bytes of the buffer.

Buffers are useful because they can be sent to and shared with other threads without copying the data. They can also be easily sent and received through web sockets, compressed, and saved to the file system.

ArrayBuffer

An ArrayBuffer object is a fixed length array of bytes. You can make one like this:

const buffer = new ArrayBuffer(4);

In this case we set the length of the buffer to be 4 bytes. So, if we created a TypedArray that used 1 byte per number we could store four numbers in that buffer.

SharedArrayBuffer

A SharedArrayBuffer is basically the same as the ArrayBuffer but it can be shared by multiple threads.

const buffer = new SharedArrayBuffer(4);

Please check out the MDN Docs about the SharedArrayBuffer to learn how to properly get it working.
Also, if you plan to use one you may run into race conditions. You can use Atomics to aid with fixing that.

When an ArrayBuffer is transferred to another thread the thread that sent it losses access to it. The SharedArrayBuffer then basically acts as a pointer to the same buffer.

Typed Arrays

Typed Arrays are fixed length arrays that act as a view over a buffer. They can either be created with or without a buffer.
Here are the typed arrays:

//1 byte per number | range: -128 - 127
const int8 = new Int8Array();
//1 byte per number | range: 0 - 255
const uInt8 = new Uint8Array();
//1 byte per number | range: 0 - 255
const clampedInt8 = new Uint8ClampedArray();
//2 bytes per number | range: -32768 - 32767
const int16 = new Int16Array();
//2 bytes per number | range: 0 - 65535
const uInt16 = new Uint16Array();
//4 bytes per number | range: -2147483648 - 2147483647
const int32 = new Int32Array();
//4 bytes per number | range: 0 - 4294967295
const uInt32 = new Uint32Array();
//4 bytes per number | range: -3.4E38 - 3.4E38
const float32 = new Float32Array();
//4 bytes per number | range: -1.8E308 - 1.8E308
const float64 = new Float64Array();
//4 bytes per number | range: -2^63 - 2^63 - 1
const int65 = new BigInt64Array();
//4 bytes per number | range: 0 - 2^64 - 1
const uInt65 = new BigUint64Array();

You can create a typed array in a few ways:

//Creates a typed array with the given length
const ex1 = new Uint16Array(4);
//Creates a typed array from an array
const ex2 = new Uint16Array([1,2,3,4]);
//Creates a typed array from an ArrayBuffer
const byteSize = 4 * 2;
const buffer = new ArrayBuffer(byteSize);
const ex3 = new Uint16Array(buffer);
/* Creates a typed array from an ArrayBuffer 
at a given index and byte length */
const buffer2 = new ArrayBuffer(byteSize * 2);
const ex4 = new Uint16Array(buffer,byteSize,byteSize);
//Creates a typed array from a SharedArrayBuffer 
const SAB = new SharedArrayBuffer(byteSize)
const ex5 = new Uint16Array(SAB);

You can use a Typed Array by indexing it like a normal array:

const data = new Uint16Array(4);
data[0] = 300;
data[1] = 1000;
data[2] = 10_000;
data[3] = 5_000;

DataViews

The DataView object lets you set and get sets of bytes from buffers.

Here are its functions:

const buffer = new ArrayBuffer(16);
const dv = new DataView(buffer);
//1 byte signed int
const int8 = dv.getInt8(0);
dv.setInt8(0,int8);
//1 byte un-signed int
const uInt8 = dv.getUint8(0);
dv.setUint8(0,uInt8);
//2 byte signed int
const int16 = dv.getInt16(0);
dv.setInt16(0,int16);
//2 byte un-signed int
const uInt16 = dv.getUint16(0);
dv.setUint16(0,uInt16);
//4 byte signed int
const int32 = dv.getInt32(0);
dv.setInt32(0,int32);
//4 byte un-signed int
const uInt32 = dv.getUint32(0);
dv.setUint32(0,uInt32);
//4 byte float
const float32 = dv.getFloat32(0);
dv.setFloat32(0,float32);
//8 byte float
const float64 = dv.getFloat64(0);
dv.setFloat64(0,float64);
//8 byte signed big int
const int64 = dv.getBigInt64(0);
dv.setBigInt64(0,int64);
//8 byte un-signed big int
const uInt64 = dv.getBigUint64(0);
dv.setBigUint64(0,uInt64);

You can create a DataView object in a few ways:

const buffer = new ArrayBuffer(10);
//Create a DataView from an ArrayBuffer
const dv = new DataView(buffer);
//Create a DataView from part an ArrayBuffer with a given offset and length
const dv2 = new DataView(buffer,5,5);
//Create a DataView from a SharedArrayBuffer
const SAB = new SharedArrayBuffer(10);
const dv3 = new DataView(SAB);

Here is a basic example of using a 3 byte ArrayBuffer and a DataView object to update different parts of the buffer.

const buffer = new ArrayBuffer(3);
const dv = new DataView(buffer);
dv.setUint8(0,10);
dv.setUint16(1,300);
//value is 10
dv.getUint8(0);
//value is 300
dv.getUint16(1);

Applications

My voxel engine which I previously wrote about makes extensive use of binary data encoding, the SharedArrayBuffer and the DataView object. You can see what I wrote about it here:
How I made a multi-threaded voxel engine in TypeScript

I also wrote another library called Divine Binary Object which lets you encode and decode data from buffers.

However, there are actually a lot of other applications for binary data in JavaScript:

• Web Audio API
  • The API uses an raw audio buffers to create audio source nodes. Check out: BaseAudioContext.decodeAudioData
• Web GL API
  • You can use Typed Arrays to create mesh data.
• Web Sockets
  • You can communicate with raw binary data using web sockets.
• Compression Streams API
  • You compress and decompress raw binary data.
  • You can also do this in Node.
• Transferable objects
  • You can transfer ArrayBuffers and TypedArrays to other threads without copying them.