Let's start by explaining how data is sent over the web. It is not sent as a single continuous stream; instead, it is divided into smaller chunks. On the receiving end, the consumer or application is responsible for reassembling these chunks in the correct order and format once all the data has been received. This process happens automatically for images, videos, and other relatively large data types.
So what Streams API offers is a way to instead of waiting for your full data to be available
- handle data in real-time as they are received as chunks during the consumer stage and that is very useful and borderline essential when dealing with large amounts of data like the examples I would show in this article.
- it also allows you to treat our data as a stream which is useful on the backend side when you want to send a specific type of chunks and in the frontend when using like workers to send large files through the network"
Revised Text: "What the Streams API offers is a way to handle data as it arrives, rather than waiting for the entire dataset to be available. Here are two key benefits:
- Real-Time Data Handling: It allows you to process data in real-time as it is received in chunks. This capability is crucial when dealing with large amounts of data, such as the examples I will discuss in this article. (this article focuses on the first part)
- Stream-Based Data Management: The Streams API enables you to treat data as a continuous stream. This is useful on the back end for sending data in specific chunks and on the front end for using web workers to upload large files efficiently.
Let's start by comparing the traditional method of receiving data using the Fetch API with the new Streams API approach.
Traditional Approach with Fetch API
fetch("url") .then((response) => {
// Note that there is a middle step before we receive the final data
// Let's see what we actually receive
console.log(response.body); return response.text(); }) .then((data) => { // Perform operations with the data
});
In this example, response.body is a ReadableStream object:
ReadableStream { locked: false, state: 'readable', supportsBYOB: true }
Here, we encounter the first component of the Streams API: ReadableStream. The ReadableStream constructor creates and returns a readable stream object, which allows us to handle streaming data more efficiently. We can use this constructor to manage data in chunks rather than waiting for the entire dataset to be available.
{ arrayBuffer(): Promise<ArrayBuffer>; blob(): Promise<Blob>; formData(): Promise<FormData>; json(): Promise<any>; text(): Promise<string>; }
We need to implement a function that handles the object to access data as it is sent in real time. This function should:
1 Receive the ReadableStream as a promise.
- Wait for all chunks of data to be received.
- Merge the chunks into the full dataset. Return the complete data as a promise.
Diving into ReadableStream
interface ReadableStream<R = any> {
readonly locked: boolean;
cancel(reason?: any): Promise<void>;
getReader(options: { mode: "byob" }): ReadableStreamBYOBReader;
getReader(): ReadableStreamDefaultReader<R>;
getReader(options?: ReadableStreamGetReaderOptions): ReadableStreamReader<R>;
pipeThrough<T>(
transform: ReadableWritablePair<T, R>,
options?: StreamPipeOptions
): ReadableStream<T>;
pipeTo(
destination: WritableStream<R>,
options?: StreamPipeOptions
): Promise<void>;
tee(): [ReadableStream<R>, ReadableStream<R>];
}
interface ReadableStreamDefaultReader<R = any>
extends ReadableStreamGenericReader {
read(): Promise<ReadableStreamReadResult<R>>;
releaseLock(): void;
}
To work with the stream, we use getReader() which returns a ReadableStreamDefaultReader.
Here’s an example where we make a request to Lichess.org’s API for games in PGN format (think of it as text) for a certain user. The final result should be in text.
fetch("https://lichess.org/api/games/user/gg").then((response) => {
console.log(response);
const readablestream = response.body;
console.log(readablestream);
const reader = readablestream.getReader();
console.log(reader);
});
Output:
ReadableStream { locked: false, state: 'readable', supportsBYOB: true } ReadableStreamDefaultReader { stream: ReadableStream { locked: true, state: 'readable', supportsBYOB: true }, readRequests: 0, close: Promise { <pending> } }
note you can't have multiple readers at the same time as the getReader() will throw an error if the ReadableStream.locked = true, so if u want to change the reader you have first to release the lock using ReadableStreamDefaultReader.releaseLock()
fetch("https://lichess.org/api/games/user/gg").then((response) => {
const readablestream = response.body;
console.log(readablestream);
const reader = readablestream.getReader();
console.log(reader);
try {
reader.releaseLock();
const reader2 = readablestream.getReader();
// won't throw an error
const reader3 = readablestream.getReader();
// will throw an error
} catch (e) {
console.error(e.message);
// Invalid state: ReadableStream is locked
}
});
now we use the read function inside the reader which has two variables
- value: has the current chunk content in UintArray which we can convert to a string by converting each int to char and merge or simply using the TextDecoder().decode()
let string = result.push(
value.reduce((p, c) => {
return p + c.fromCharCode();
}, "")
); // or
let string = new TextDecoder().decode(value);
// both achieve the same thing converting Uint8Array to string
Full Code Example
- Here’s a full example of handling the stream and merging chunks:
fetch("https://lichess.org/api/games/user/gg")
.then((response) => {
return new Promise((resolve, reject) => {
const readablestream = response.body;
const reader = readablestream.getReader();
let result = [];
reader.read().then(function handlechunks({ done, value }) {
if (done) {
resolve(result);
return;
}
const pgn = new TextDecoder().decode(value);
result.push(pgn);
reader.read().then(handlechunks);
});
});
})
.then((result) => {
console.log(result);
});
// console.log(value)
Uint8Array(551) [
91, 69, 118, 101, 110, 116, 32, 34, 82, 97, 116, 101,
100, 32, 98, 108, 105, 116, 122, 32, 103, 97, 109, 101,
34, 93, 10, 91, 83, 105, 116, 101, 32, 34, 104, 116,
116, 112, 115, 58, 47, 47, 108, 105, 99, 104, 101, 115,
115, 46, 111, 114, 103, 47, 90, 122, 78, 66, 90, 119,
100, 71, 34, 93, 10, 91, 68, 97, 116, 101, 32, 34,
50, 48, 50, 48, 46, 48, 49, 46, 49, 48, 34, 93,
10, 91, 87, 104, 105, 116, 101, 32, 34, 86, 101, 101,
118, 101, 101, 50,
... 451 more items
]
// console.log(new TextDecoder().decode(value))
[Event "Rated blitz game"]
[Site "https://lichess.org/ZzNBZwdG"]
[Date "2020.01.10"]
[White "Veevee222"]
[Black "gg"]
[Result "0-1"]
[UTCDate "2020.01.10"]
[UTCTime "20:21:02"]
[WhiteElo "1858"]
[BlackElo "1863"]
[WhiteRatingDiff "-6"]
[BlackRatingDiff "+35"]
[Variant "Standard"]
[TimeControl "180+0"]
[ECO "C00"]
[Termination "Normal"]
1. e4 e6 2. d4 d6 3. c4 Nf6 4. Nc3 c5 5. f4 cxd4 6. Qxd4 Nc6 7. Qd1 b6 8. g3 Bb7 9. Bg2 Rc8 10. Nf3 Be7 11. O-O O-O 12. b3 Nb4 13. Bb2 a5 14. Re1 Qc7 15. a3 Na6 16. Rc1 Nc5 17. Qd4 Nxb3 18. Qd1 Nxc1 19. e5 0-1
for example link
for example, full code go
Now, we can access the games' PGNs progressively as they are sent through the network. For example, if we are using the loaded games in a website UI, the user won't have to wait in front of a blank or loading screen until all the games are loaded. Instead, data can be displayed progressively, which is much better from a UX standpoint.
for example full code go here
Top comments (0)