Oh, hey there!
I'm glad you made it to this second post of the "Build the System: HTTP server" series.
This post is dedicated to decoding HTTP requests and encoding the response. I will also, offer a reliable way to test
our code for a more resilient project.
If you haven't read the first post of the series yet, I think you might want to. Just click here to read it.
I'll wait patiently for your return.
This article is a transcript of a Youtube video I made.
Alright, now that I know we're all on the same page, let's write some code.
For this project, I will use JavaScript and Deno, but the concepts don't change no matter what language or runtime you
are using.
Also one last disclaimer: this project first aim is to educate it will in no way be complete or the most performant!
I will discuss specifically the improvements we can bring to make it more performant and I will go through various
iteration with that in mind. At the end of the project, if there are part worth salvaging, I will replace the essential
parts.
All that to say, just enjoy the ride.
The first thing that I need to do is to announce listening on a port.
The incoming connection will be represented by a Readable/Writable resource.
First, I will need to read from the resource a specific amount of bytes. For this example, I will read around a KB.
The variable xs is a Uint8Array. I already wrote an article about this but long story short, a Typed Array is an array
that can only hold a specific amount of bit per item. In this case we need 8 bits (or one byte) array because you need 8 bits
to encode a single UTF-8 character.
🐙 You will find the code for this post here: https://github.com/i-y-land/HTTP/tree/episode/02
As a convenience, I will decode the bytes to a string and log the result to the console.
Finally, I will encode a response and write it to the resource.
// scratch.js
for await (const connection of Deno.listen({ port: 8080 })) {
const xs = new Uint8Array(1024);
await Deno.read(connection.rid, xs);
console.log(new TextDecoder().decode(xs));
await Deno.write(
connection.rid,
new TextEncoder().encode(
`HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`
)
);
}
Now, I will run the code:
deno run --allow-net="0.0.0.0:8080" scratch.js
On a different terminal session I can use curl to send an HTTP request.
curl localhost:8080
On the server's terminal, we can see the request, and on the client's terminal we can see the response's body:
"Hello, World"
Great!
To get this started on the right foot, I will refactor the code into a function named serve in a file called
server.js. This function will take a listener and a function that takes a Uint8Array and returns a Promise of a
Uint8Array!
// library/server.js
export const serve = async (listener, f) => {
for await (const connection of listener) {
const xs = new Uint8Array(1024);
const n = await Deno.read(connection.rid, xs);
const ys = await f(xs.subarray(0, n));
await Deno.write(connection.rid, ys);
}
};
Notice that the read function returns the number of byte that was read. So we can use the subarray method to pass
a lense on the appropriate sequence to the function.
// cli.js
import { serve } from "./server.js";
const $decoder = new TextDecoder();
const decode = $decoder.decode.bind($decoder);
const $encoder = new TextEncoder();
const encode = $encoder.encode.bind($encoder);
if (import.meta.main) {
const port = Number(Deno.args[0]) || 8080;
serve(
Deno.listen({ port }),
(xs) => {
const request = decode(xs);
const [requestLine, ...lines] = request.split("\r\n");
const [method, path] = requestLine.split(" ");
const separatorIndex = lines.findIndex((l) => l === "");
const headers = lines
.slice(0, separatorIndex)
.map((l) => l.split(": "))
.reduce(
(hs, [key, value]) =>
Object.defineProperty(
hs,
key.toLowerCase(),
{ enumerable: true, value, writable: false },
),
{},
);
if (method === "GET" && path === "/") {
if (
headers.accept.includes("*/*") ||
headers.accept.includes("plain/text")
) {
return encode(
`HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`,
);
} else {
return encode(
`HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n`,
);
}
}
return encode(
`HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n`,
);
},
)
.catch((e) => console.error(e));
}
Now that I have an way to parse the headers, I think it's a good opportunity to officialize all of this and write a new
utility function and the appropriate tests.
// library/utilities.js
export const parseRequest = (xs) => {
const request = decode(xs);
const [h, body] = request.split("\r\n\r\n");
const [requestLine, ...ls] = h.split("\r\n");
const [method, path] = requestLine.split(" ");
const headers = ls
.map((l) => l.split(": "))
.reduce(
(hs, [key, value]) =>
Object.defineProperty(
hs,
key.toLowerCase(),
{ enumerable: true, value, writable: false },
),
{},
);
return { method, path, headers, body };
};
// library/utilities_test.js
Deno.test(
"parseRequest",
() => {
const request = parseRequest(
encode(`GET / HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n`),
);
assertEquals(request.method, "GET");
assertEquals(request.path, "/");
assertEquals(request.headers.host, "localhost:8080");
assertEquals(request.headers.accept, "*/*");
},
);
Deno.test(
"parseRequest: with body",
() => {
const request = parseRequest(
encode(
`POST /users HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n{"fullName":"John Doe"}`,
),
);
assertEquals(request.method, "POST");
assertEquals(request.path, "/users");
assertEquals(request.headers.host, "localhost:8080");
assertEquals(request.headers.accept, "*/*");
assertEquals(request.body, `{"fullName":"John Doe"}`);
},
);
Now that I have a parseRequest function, logically I need a new function to stringify the response...
// library/utilities.js
import { statusCodes } from "./status-codes.js";
export const normalizeHeaderKey = (key) =>
key.replaceAll(/(?<=^|-)[a-z]/g, (x) => x.toUpperCase());
export const stringifyHeaders = (headers = {}) =>
Object.entries(headers)
.reduce(
(hs, [key, value]) => `${hs}\r\n${normalizeHeaderKey(key)}: ${value}`,
"",
);
export const stringifyResponse = (response) =>
`HTTP/1.1 ${statusCodes[response.statusCode]}${
stringifyHeaders(response.headers)
}\r\n\r\n${response.body || ""}`;
// library/utilities_test.js
Deno.test(
"normalizeHeaderKey",
() => {
assertEquals(normalizeHeaderKey("link"), "Link");
assertEquals(normalizeHeaderKey("Location"), "Location");
assertEquals(normalizeHeaderKey("content-type"), "Content-Type");
assertEquals(normalizeHeaderKey("cache-Control"), "Cache-Control");
},
);
Deno.test(
"stringifyResponse",
() => {
const body = JSON.stringify({ fullName: "John Doe" });
const response = {
body,
headers: {
["content-type"]: "application/json",
["content-length"]: body.length,
},
statusCode: 200,
};
const r = stringifyResponse(response);
assertEquals(
r,
`HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
);
},
);
So now, we have everything we need to refactor our handler function and make it more concise and declarative.
import { serve } from "./library/server.js";
import {
encode,
parseRequest,
stringifyResponse,
} from "./library/utilities.js";
if (import.meta.main) {
const port = Number(Deno.args[0]) || 8080;
serve(
Deno.listen({ port }),
(xs) => {
const request = parseRequest(xs);
if (request.method === "GET" && request.path === "/") {
if (
request.headers.accept.includes("*/*") ||
request.headers.accept.includes("plain/text")
) {
return Promise.resolve(
encode(
stringifyResponse({
body: "Hello, World",
headers: {
"content-length": 12,
"content-type": "text/plain",
},
statusCode: 200,
}),
),
);
} else {
return Promise.resolve(
encode(stringifyResponse({ statusCode: 204 })),
);
}
}
return Promise.resolve(
encode(
stringifyResponse({
headers: {
"content-length": 0,
},
statusCode: 404,
}),
),
);
},
)
.catch((e) => console.error(e));
}
So at this we can deal with any simple request effectively. To wrap this up and prepare the project for future iteration,
I will add a test for the serve function. Obviously, this function is impossible to keep pure and to test without
complex integration tests -- which I keep for later.
An actual connection is a bit figety so I thought I could mock it using a file as the resource since files are
readable/wriatable.
The first thing I did is to write a function to factorize an async iterator and purposely make it break after the first
iteration.
After that, I create a file with read/write permissions. With that, I can write the HTTP request, then move the cursor
back to the beginning of the file for the serve function to read back. Within the handler function, I make some
assertions on the request for sanity's sake, then flush the content and move the cursor back to the beginning before
writing a response.
Finally, I can move the cursor back to the beginning one last time, to read the response, make one last assertion then
cleanup to complete the test.
// library/server_test.js
import { assertEquals } from "https://deno.land/std@0.97.0/testing/asserts.ts";
import { decode, encode, parseRequest } from "./utilities.js";
import { serve } from "./server.js";
const factorizeConnectionMock = (p) => {
let i = 0;
return {
p,
rid: p.rid,
[Symbol.asyncIterator]() {
return {
next() {
if (i > 0) {
return Promise.resolve({ done: true });
}
i++;
return Promise.resolve({ value: p, done: false });
},
values: null,
};
},
};
};
Deno.test(
"serve",
async () => {
const r = await Deno.open(`${Deno.cwd()}/.buffer`, {
create: true,
read: true,
write: true,
});
const xs = encode(`GET /users/1 HTTP/1.1\r\nAccept: */*\r\n\r\n`);
await Deno.write(r.rid, xs);
await Deno.seek(r.rid, 0, Deno.SeekMode.Start);
const connectionMock = await factorizeConnectionMock(r);
await serve(
connectionMock,
async (ys) => {
const request = parseRequest(ys);
assertEquals(
request.method,
"GET",
`The request method was expected to be \`GET\`. Got \`${request.method}\``,
);
assertEquals(
request.path,
"/users/1",
`The request path was expected to be \`/users/1\`. Got \`${request.path}\``,
);
assertEquals(
request.headers.accept,
"*/*",
);
await Deno.ftruncate(r.rid, 0);
await Deno.seek(r.rid, 0, Deno.SeekMode.Start);
const body = JSON.stringify({ "fullName": "John Doe" });
return encode(
`HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: ${body.length}\r\n\r\n${body}`,
);
},
);
await Deno.seek(r.rid, 0, Deno.SeekMode.Start);
const zs = new Uint8Array(1024);
const n = await Deno.read(r.rid, zs);
assertEquals(
decode(zs.subarray(0, n)),
`HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
);
Deno.remove(`${Deno.cwd()}/.buffer`);
Deno.close(r.rid);
},
);
At this point we have a good base to work from. Unfortunately our server is a bit limitted, for example, if a request
is larger than a KB, we'd be missing part of the message, that means no upload or download of medium size files.
That's what I plan to cover on the next post. This will force us to be a little bit more familiar with
manipulation of binary bytes.
At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!
Ok bye now...
Top comments (0)