Web Caching - Last-Modified/If-Modified-Since

Less load on the server and less bandwidth usage for the same result? Where should I sign up? Nowhere, you just need to know the right headers.

Support code

Let's keep it simple - NodeJS, no dependencies. Build with me some endpoints, each using different headers, and find out how the browser behaves based on the headers received.

Go directly to the /no-headers endpoint or take a (very quick) look at the easiest server there is.

import { createServer } from "http";
import noHeaders from "./src/index.mjs";

createServer((req, res) => {
  switch (req.url) {
    case "/no-headers":
      return noHeaders(req, res);
  }
}).listen(8000, "127.0.0.1", () =>
  console.info("Exposed on http://127.0.0.1:8000")
);

import fs from "fs/promises";
import path from "path";

export function to(promise) {
  return promise.then((res) => [res, null]).catch((err) => [null, err]);
}

export async function getView(name) {
  const filepath = path.resolve(
    process.cwd(),
    "src",
    "views",
    name + ".html"
  );
  return await to(fs.readFile(filepath, "utf-8"));
}

export async function getViewStats(name) {
  const filepath = path.resolve(process.cwd(), "src", "views", name + ".html");
  return await to(fs.stat(filepath));
}

Add an HTML file at src/views/index.html. Its content is irrelevant.

No Headers - Endpoint

It simply reads the file and sends it to the requester. Apart from the Content-Type, no caching-related header is added.

// src/no-headers.mjs
import { getView } from "./utils.mjs";

export default async (req, res) => {
  res.setHeader("Content-Type", "text/html");

  const [html, err] = await getView("index");
  if (err) {
    res.writeHead(500).end("Internal Server Error");
    return;
  }

  res.writeHead(200).end(html);
};

Start the server (node index.mjs), open /no-headers, and check the developer tools > network tab. Enable preserver log and hit refresh a few times.

Open any of them, and check the Request Headers - there is nothing related to caching, and the browser obeys.

HTTP/1.1 200 OK
Content-Type: text/html
Date: <date>
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

Last-Modified - Endpoint

Spec

Create a new endpoint (to be registered at the url /last-modified). It reads the modification time of the file (mtime) and adds it formatted as UTC under the Last-Modified header.

// src/last-modified.mjs
import { getView, getViewStats } from "./utils.mjs";

export default async (req, res) => {
  res.setHeader("Content-Type", "text/html");

  const [stats, errStats] = await getViewStats("index");
  if (errStats) {
    res.writeHead(500).end("Internal Server Error");
    return;
  }

  const lastModified = new Date(stats.mtime);
  res.setHeader("Last-Modified", lastModified.toUTCString());

  const [html, errGet] = await getView("index");
  if (errGet) {
    res.writeHead(500).end("Internal Server Error");
    return;
  }

  res.writeHead(200).end(html);
};

In fact, among the response headers to /last-modified, you find:

HTTP/1.1 200 OK
Last-Modified: Thu, 15 Nov 2023 19:18:46 GMT

Anyway, if you refresh the page, the entire resource is still downloaded.

Yet something changed - the browser found Last-Modified, so it reuses the value for the If-Modified-Since Request Header. The serve receives that value and, if the condition is found to be not true (not modified since), returns the status 304 Not Modified.

import { getView, getViewStats } from "./utils.mjs";

export default async (req, res) => {
  res.setHeader("Content-Type", "text/html");

  const [stats, _] = await getViewStats("index");

  const lastModified = new Date(stats.mtime);
  lastModified.setMilliseconds(0); // IMPORTANT
  res.setHeader("Last-Modified", lastModified.toUTCString());

  const ifModifiedSince = new Headers(req.headers).get("If-Modified-Since");
  if (
    ifModifiedSince &&
    new Date(ifModifiedSince).getTime() >= lastModified.getTime()
  ) {
    res.writeHead(304).end();
    return;
  }

  // This is done ONLY IF it was not a 304!
  const [html, _] = await getView("index");

  res.writeHead(200, headers).end(html);
};

By spec Last-Modified

Note:

• The Response Header Last-Modified is always added, even in the case of 304 Not Modified.
• The Request Header if-modified-since may not be present - definitely happens on the first call from a new client.

Most importantly, HTTP dates are always expressed in GMT, never in local time.

While formatting a date using toUTCString, you may observe that the resulting string loses information about milliseconds. However mtime retains millisecond precision - it may have a few milliseconds more than the value received from the client, which, after formatting, loses those milliseconds.

To ensure a valid comparison between the two values, it becomes necessary to remove the milliseconds from the mtime before performing the comparison.

lastModified.setMilliseconds(0);

Finally, request the resource few times.

Now, just go and update the HTML file. Then ask the browser to refresh and expect to receive a 200 OK Response.

It's essential to recognize that the 304 response is consistently more lightweight than the 200 response. Beyond just the reduced data payload, it contributes to a decrease in server load. This optimization extends beyond mere HTML file reads and can apply to any intricate or resource-intensive operation.

Last-Modified is a weak caching header, as the browser applies a heuristic to determine whether to fetch the item from the cache or not. Heuristics vary between browsers.