Poking around with some web frameworks like Next and Astro I was posed the question of how hard is some of the stuff they are doing and could we do it custom? My initial reaction was no way, these frameworks do a lot and I would not want that burden. But it sat in the back of my head and what I really wanted was some data. Could I enumerate all the features that were usefully interesting and how hard the were to implement to give a more informed decision? Not that this changes anything, upkeep and edge-case catching is like 90% of the work but can we actually make minimal versions of some features? So to test this I want to start with react SSR. I've actually done a little bit on this before in the context of a tiny static-site generator. That one used a few tricks (read: better libraries) to make it easier but this time I wanted a more real React experience which requires taming the dragon, warts and all.
The first thing we need to do is add support for SSR so we have a baseline. React and React DOM have most of the functionality available to us. Since we're using Deno we don't even need to clutter our project with an package.json, we can just use npm identifiers.
import * as React from "npm:react";
import * as ReactDom from "npm:react-dom";
Since we need to disambiguate what different types of js/jsx/ts/tsx do we'll use an extra extension .react.js . Then we can add some path matchers to the index route:
filePaths.push(...[
".html",
".server.js",
".server.ts",
".server.jsx",
".server.tsx",
".react.js",
".react.jsx",
".react.tsx"
].map(ext => baseDir + inputPath + ext));
These files need to have a specific format so we can properly use them. I'm choosing to use the NextJS paradigm, so we have a default export which is the react component, and a getServerSideProps function which gets invoked to to generate the initial props.
//app.react.jsx
import * as React from "npm:react";
export async function getServerProps(){
return {
name: "John Doe",
userId: 123,
dob: "5/6/1990"
};
}
export default function App(props){
return <>
<h1>My React App</h1>
<p>Hello {props.name}!</p>
<p>Your userId is {props.userId}</p>
<p>Your date of birth is {props.dob}</p>
</>;
}
Then when generating responses we check the extension. We already have a handler for .server.{js/jsx/ts/tsx} so right below that we add one for react.
//server.js - below matcher for .server. files
if (/\.react\.(js|ts|jsx|tsx)/.test(fileMatch[1])) {
const mod = await import(fileMatch[1]);
const props = await mod.getServerProps();
const stream = await ReactDom.renderToReadableStream(React.createElement(mod.default, props));
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
})
}
At this point we can sort of render. It takes the default head/body tags the browser generates but we can see it did at least template in our values if we navigate to /app.
To improve we need to make sure the outer shell is also rendered. We'll need to create a folder outside of routes because we don't want it to be picked up, I'm going to call it layouts this will never be available to the client so everything in here it only for server rendering. We'll call the component that provides the outer structure document.jsx.
//components/document.jsx
import * as React from "npm:react";
export function Document(props){
return <html lang="en">
<head>
<title>{props.title ?? "dev server"}</title>
<link rel="stylesheet" href="css/app.css" />
</head>
<body>
<main id="react-root">{props.children}</main>
</body>
</html>
}
We can add an extra property for the title on the page
//app.react.jsx
export function getTitle(){
return "My React App";
}
Then we just pass our page component into this (you also need to import Document):
//server.js - modifying where we match .react. files
if (/\.react\.(js|ts|jsx|tsx)/.test(fileMatch[1])) {
const mod = await import(fileMatch[1]);
const props = await mod.getServerProps();
- const stream = await ReactDom.renderToReadableStream(React.createElement(mod.default, props))
+ const title = mod.getTitle?.();
+ const stream = await ReactDom.renderToReadableStream(
+ React.createElement(Document, { title },
+ React.createElement(mod.default, props)));
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
})
}
Now when we render we should get a complete page with title and the font should be arial letting us know that we're actually loading styles correctly.
Hydration
At this point we can render but what if we try something with dynamic behavior? Let's add a folder for components under routes called js/components.
//./routes/js/components/counter.jsx
import * as React from "npm:react";
export function Counter() {
const [value, setValue] = React.useState(0);
return <div>
<div className="counter">{value}</div>
<button onClick={() => setValue(value - 1)}>-</button>
<button onClick={() => setValue(value + 1)}>+</button>
</div>
}
//app.react.js
import * as React from "npm:react";
+import { Counter } from "./js/components/counter.jsx";
export async function getServerProps(){
return {
name: "John Doe",
userId: 123,
dob: "5/6/1990"
};
}
export function getTitle(){
return "My React App";
}
export default function App(props){
return <>
<h1>My React App</h1>
<p>Hello {props.name}!</p>
<p>Your userId is {props.userId}</p>
<p>Your date of birth is {props.dob}</p>
+ <Counter />
</>;
}
It renders but we can't push the buttons. There's no client-side scripts to hook up! Let's see what we can do. Let's start with our _document, we know we'll need to add a script tag to something so let's wire that up.
import * as React from "npm:react";
export function Document(props){
return <html lang="en">
<head>
<title>{props.title ?? "dev server"}</title>
<link rel="stylesheet" href="css/app.css" />
</head>
<body>
<main id="react-root">{props.children}</main>
+ <script type="module" dangerouslySetInnerHTML={{ __html: props.script ?? "" }}></script>
</body>
</html>
}
There were a few ways we could have done this. We could have generated a file but that is more complicated than just injecting some js (though perhaps better in the long term...). So script is a completely in-lined script. So now we can pass in a script prop. But what is our script? We know that document.jsx is not going to the client, but it seems very likely that at least part of route/app.react.jsxis. So how can we do that? Let's just try passing in the whole page script.
//server.js - modifying where we match .react. files
if (/\.react\.(js|ts|jsx|tsx)/.test(fileMatch[1])) {
const mod = await import(fileMatch[1]);
const props = await mod.getServerProps();
const title = mod.getTitle?.();
let script = await Deno.readTextFile(fileMatch[1]);
if(mod.default.name === "default"){
throw new Error("Page default exports must also have names");
}
script += `
\n
\n
ReactDom.hydrateRoot(document.querySelector("#react-root"), React.createElement("${mod.default.name}", ${JSON.stringify(props)}));`
console.log(script)
const stream = await ReactDom.renderToReadableStream(
React.createElement(Document, { title, script },
React.createElement(mod.default, props)));
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
})
}
So here's where we can try to brute-force it. What we do is get the whole file as text, and then append the hydration command to the bottom.
ReactDom.hydrateRoot(document.querySelector("react-root"), React.createElement("${mod.default.name}", JSON.
parse(${JSON.stringify(props)})));
We need the name of the component and the props. The props we already have, we just need to serialize them. NextJS also has the limitation that server props must be serializable and that's because it's doing something like this, serializing them to the DOM for hydration. The name is a hack (and we'll see this has a problem, see if you can spot it yet). Default exports without names have the name default as their export name. The problem with anonymous default exports is they cannot be referenced inside their own module. Since we're re-using the module code, we need to make sure we have referenceable name to pass to hydrateRoot. When we load this, we'll find that it doesn't quite work. While Deno can happily deal with JSX, the browser cannot, the module must be transpiled. Deno cannot do this for us and it's a very complex thing to implement so while we pride ourselves on not pulling in extra libraries, we need to move to the next simplest tool, esbuild.
esbuild is a bundler, and nothing but a bundler. It's very narrow in scope and extremely fast so as a library it's something that I can really recommend. Note that esbuild publishes both to npm and to deno.land/x so we can use either. I'm not sure if there's much of a difference but maybe the deno version works better with deno, so lets use that.
Cleanup
Okay so we've kinda settled on a path forward. But first let's take some time to refactor. We've been modifying a small piece of code in server.js and I think it could really stand to be split out. So for this I'm going to create a new folder called "responders". This will hold on to implementations of different file responses namely the controllers for .server.js/.server.ts and .react.jsx/.react.tsx.
//./responders/server-responder.js
import { resolve, toFileUrl } from "https://deno.land/std@0.205.0/path/mod.ts";
export async function serverResponder(path, req){
const moduleImportPath = toFileUrl(resolve(Deno.cwd(), path));
const mod = await import(moduleImportPath);
if (req.method === "GET") {
return mod.get?.(req) ?? mod.default?.(req)
?? new Response("Method not allowed", { status: 405 });
} else if (req.method === "DELETE") {
return mod.del?.(req)
?? new Response("Method not allowed", { status: 405 });
} else {
return mod[req.method.toLowerCase()]?.(req)
?? new Response("Method not allowed", { status: 405 });
}
}
//./responders/react-responder.js
import * as React from "npm:react";
import * as ReactDom from "npm:react-dom/server";
import { Document } from "../layouts/document.jsx";
import { transform } from "https://deno.land/x/esbuild/mod.js";
import { resolve, toFileUrl } from "https://deno.land/std@0.205.0/path/mod.ts";
export async function reactResponder(path){
const moduleImportPath = toFileUrl(resolve(Deno.cwd(), path));
const mod = await import(moduleImportPath);
const props = await mod.getServerProps();
const title = mod.getTitle?.();
const page = await Deno.readTextFile(path);
const transpiledPage = await transform(page, { loader: "tsx" });
const script = `
${transpiledPage.code}
ReactDom.hydrateRoot(document.querySelector("#react-root"), React.createElement("${mod.default.name}", ${ JSON.stringify(props) }));
`;
const stream = await ReactDom.renderToReadableStream(
React.createElement(Document, { title, script },
React.createElement(mod.default, props)));
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
});
}
Here I'm adding esbuild and transpiling the source with the .tsx loader so that the JSX is converted into React.createElement calls (typescript will also be converted for client use). Then we're doing what we were before. Note that we need to resolve the module url correctly because we're no longer at root so we can use the path relative to the CWD and convert to a file url so we can use it with import (it will not accept windows paths).
Now we can load these on demand. This is nice because it means that we do not need to pay the cost of having react if we aren't using it, we'll never load it. Of course because we do reference it in the responder and the components Deno will cache it so it would take a little more work to make sure Deno absolutely never tries anything smart.
// server.js - in the filetype switch
switch(ext){
case ".js":
case ".ts":
case ".tsx":
case ".jsx": {
if(/\.server\.(js|ts|jsx|tsx)/.test(fileMatch[1])){
const mod = await import("./responders/server-responder.js");
return mod.serverResponder(fileMatch[1], req);
}
if (/\.react\.(js|ts|jsx|tsx)/.test(fileMatch[1])) {
const mod = await import("./responders/react-responder.js");
return mod.reactResponder(fileMatch[1]);
}
}
// falls through
default: {
const file = await Deno.open(fileMatch[1]);
return new Response(file.readable, {
headers: {
"Content-Type": typeByExtension(ext)
}
});
}
}
So now we've fixed the syntax error but we another one. We can't find npm:react. To solve this we can use import maps.
//server-responder.js - right before converting to stream
const importMap = {
imports: {
"npm:react": "https://esm.sh/react@18.2.0"
}
};
const stream = await ReactDom.renderToReadableStream(
React.createElement(Document, { title, script, importMap },
React.createElement(mod.default, props)));
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
});
In document.jsx we can use it.
//./layouts/document.jsx
import * as React from "npm:react";
export function Document(props){
return <html lang="en">
<head>
<title>{props.title ?? "dev server"}</title>
<link rel="stylesheet" href="css/app.css" />
{
props.importMap
? <script type="importmap" dangerouslySetInnerHTML={{ __html: JSON.stringify(props.importMap) }} />
: null
}
</head>
<body>
<main id="react-root">{props.children}</main>
<script type="module" dangerouslySetInnerHTML={{ __html: props.script ?? "" }}></script>
</body>
</html>
}
This will clear up the react error. What this does is tell the browser to map npm:react to some other url, in this case it's the react hosted on esm.sh. Note that it's important that we use a CDN that provides actual ESM transpilation because React in the year 2023 does not have a native ESM export and is literally holding the entire ecosystem back. We could have hosted it ourselves too, it's just more work. Now since it just so happens that the route from the app.react.jsx is the same both server-side and client-side (a nice side-effect of not separating assets and routes by folders) so we don't actually need an import map for it client-side, it just works!
Still we have another problem. The component file is still JSX and we can't use that client-side. So We need to get a bit smarter and since we've already added esbuild nothing is stopping us from transpiling for the client.
Transpiling JSX and TSX
We'll make a slight change to the way we detect the extname. In Deno this will only get the final part of the path (eg foo.server.js will return js not server.js) but we really want the file name to be anything before the first . and everything after to be the true extension name as this makes matching easier.
//server.js - in the filetype switch
const filePath = fileMatch[1];
const ext = filePath.split(".").filter(x => x).slice(1).join(".");
switch(ext){
case "server.js":
case "server.ts":
case "server.jsx":
case "server.tsx":
{
const mod = await import("./responders/server-responder.js");
return mod.serverResponder(filePath, req);
}
case "react.jsx":
case "react.tsx":
{
const mod = await import("./responders/react-responder.js");
return mod.reactResponder(filePath);
}
// falls through
default: {
const file = await Deno.open(filePath);
return new Response(file.readable, {
headers: {
"Content-Type": typeByExtension(ext)
}
});
}
}
The new ext takes into account the compound extension (note the .filter(x => x) which will remove empty dots which solves for paths that start with dots like ./path/to/foo.server.js). This cleans up the matcher so it's easier to see what's happening. We'll just add new ones for jsx, ts and tsx.
// responders/transpile-responder.js
import { transform } from "https://deno.land/x/esbuild/mod.js";
export async function transpileResponder(path) {
const codeText = await Deno.readTextFile(path);
const transpiled = await transform(codeText, { loader: "tsx" });
return new Response(transpiled.code, {
headers: {
"Content-Type": "text/javascript"
}
});
}
And slot this into our matcher:
//server.js - in the filetype switch
case "jsx":
case "ts":
case "tsx":
{
const mod = await import("./responders/transpile-responder.js");
return mod.transpileResponder(filePath);
}
And we can transpile all non-javascript on the fly!
Now when we try to load the page we have valid javascript but a hydration error. React's hydration errors are awful and don't tell you anything but with some strategic rendering to string we can figure out what happened. It turns out we SSR'd with
<h1>My React App</h1>
<p>Hello <!-- -->John Doe<!-- -->!</p>
<p>Your userId is <!-- -->123</p>
<p>Your date of birth is <!-- -->5/6/1990</p>
<div>
<div class="counter">0</div>
<button>-</button>
<button>+</button>
</div>
But we hydrated with
<App name="John Doe" userId="123" dob="5/6/1990"></App>
That is, we accidentally add the string "App" instead of the reference to the App function. We just need to delete the quotes around mod.default.name -> React.createElement(${mod.default.name}, ${ JSON.stringify(props) }) . Finally, it works with no errors!
Un-shipping server code
So while functionally this seems to work we have another problem. Since we just took the page code and pushed it down we actually have the getServerProps and getTitle functions on the client too. This aren't supposed to be used there and this can actually cause problems or be dangerous if we expose server-side details we didn't mean to. So we need to find a way to remove them. What we know is that pretty much anything outside the default export is really for the server, if we used variables from outside the scope they could reference things we didn't want, we also just don't have anything besides the props on the client. The serialization forms a boundary we shouldn't reach outside. This places an artificial limit on the code we can write but it also just makes more sense and so we'll make a rule: the page component cannot reference outside variables, they must be passed in as props. With this rule in place we can look at this differently. We only need to ship the code in the default export.
We can actually do a cool trick. If we just want certain functions to be shipped we can use Function.toString() to get their source code.
// responder/react-responder.js
export async function reactResponder(path){
const moduleImportPath = toFileUrl(resolve(Deno.cwd(), path));
const mod = await import(moduleImportPath);
const props = await mod.getServerProps();
const title = mod.getTitle?.();
const script = `
${mod.default.toString() }
import * as React from "https://esm.sh/react@18.2.0";
import * as ReactDom from "https://esm.sh/react-dom@18.2.0";
ReactDom.hydrateRoot(document.querySelector("#react-root"), React.createElement(${mod.default.name}, ${ JSON.stringify(props) }));
`;
const stream = await ReactDom.renderToReadableStream(
React.createElement(Document, { title, script },
React.createElement(mod.default, props)));
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
});
}
This will simple serialize the JS text of mod.default which includes the function signature. Since Deno will have already transpiled the JSX for run-time we don't even need to run esbuild, we got it for free. Also, because we're not using the server-side imports we don't need the import map either, we can just manually set these (you might still want the import map though to make book-keeping easier). So our code is now simpler, smaller and faster. Very nice! But it comes with a new problem. On the client we don't know what Counter is because we lost that import. This is a tricky issue. We don't know which imports were for the server-side stuff and which were for the component. Even if we did we don't have access to the import names at run-time. It seems this is where you need smart bundler. You basically want to take the default export and then tree-shake everything else out of the module. Unfortunately esbuild cannot do this, it doesn't take AST plugins and for good reason, this allows it to be really optimized. So that seems to be end, we need to not only have bundler but we need to write a plugin for it.
Some other strategies I looked at:
- Regex magic - This is super brittle and ugly. Parsing correctly requires actual context-free grammar or you'll always have issues.
-
Using proxies to collect references to components after a render - I thought this would work but while we can get the names and references we can't determine the module from which they came eg (
npm:my-component). We could add new rules saying no external components (which I personally like but that's a deal breaking in a lot of use-cases). - Manually setting import info in a getClientDeps function - This would require manual updating which sucks but it's especially bad because you need to remember to update it any time you have an actual dependency change and there will be nothing to enforce this until the page doesn't load.
After some thought the best way around this without sprinkling magic complexity everywhere like NextJS is to just segment the files. All components sans the layouts are actual components in their own file with their own imports and will work on the client. There is no intermixing server and client code. The .react.js files will instead just tell us where to look up the component file for the root of the client react tree.
This is what I came up
// .responders/react-responder.js
import React from "npm:react";
import ReactDom from "npm:react-dom/server";
import { Document } from "../layouts/document.jsx";
import { resolve, toFileUrl } from "https://deno.land/std@0.205.0/path/mod.ts";
export async function reactResponder(path){
const moduleImportPath = toFileUrl(resolve(Deno.cwd(), path));
const mod = await import(moduleImportPath);
const props = await mod.getServerProps();
const title = await mod.getTitle?.();
const [componentPath, exportName] = await mod.getRootComponent();
const script = `
import * as React from "npm:react";
import * as ReactDom from "npm:react-dom";
import { ${exportName ?? "default"} } from "${componentPath}";
ReactDom.hydrateRoot(document.querySelector("#react-root"), React.createElement(${exportName}, ${ JSON.stringify(props) }));
`;
const componentMod = await import(new URL(componentPath, moduleImportPath));
const importMap = {
imports: {
"npm:react": "https://esm.sh/react@18.2.0",
"npm:react-dom": "https://esm.sh/react-dom@18.2.0"
}
};
const stream = await ReactDom.renderToReadableStream(
React.createElement(Document, { title, script, importMap },
React.createElement(componentMod[exportName], props)));
return new Response(stream, {
headers: {
"Content-Type": "text/html"
}
});
}
I normalized all react references to npm:react to just make things easier, we need the import map again. And the actual route file:
// ./routes/app.react.jsx
export function getServerProps(){
return {
name: "John Doe",
userId: 123,
dob: "5/6/1990"
};
}
export function getTitle(){
return "My React App";
}
export function getRootComponent(){
return ["./js/components/app.jsx", "App"];
}
It's pretty much just data, we could even theoretically convert it to json (I added path matching for js and ts since these make sense now). It feels heavy for a file which is probably why NextJS wanted to merge it but this is easy to implement and it's pretty clear (what the path is relative to is perhaps not obvious, we resolve it as relative to .react.js file). For completeness here's ./components/app.jsx
import * as React from "npm:react";
import { Counter } from "./counter.jsx";
export function App(props) {
return <>
<h1>My React App</h1>
<p>Hello {props.name}!</p>
<p>Your userId is {props.userId}</p>
<p>Your date of birth is {props.dob}</p>
<Counter />
</>;
}
Refactor responders to plugins
I had the idea to make dynamic fetches when using frameworks like react so we get those libraries on demand. It was a cool idea but it doesn't fully work because we still have to know about those file types ahead of time to match their extensions. I think it might be better to just use a simple plugin system, where we register them up-front. This means the matching logic can be moved to the responder module and this will clean up server.js a bit and hopefully avoid some duplicated code later. We'll use the function match(path) => boolean to match paths and we'll use defaultPaths(barePath) => string[] to generate possible default paths for paths that do not end in extensions.
//./responders/react-responder.js
const extensions = [
"react.js",
"react.jsx",
"react.ts",
"react.tsx",
];
export function match(path){
const ext = path.split(".").filter(x => x).slice(1).join(".");
return extensions.includes(ext);
}
export function defaultPaths(barePath){
return extensions.map(ext => barePath + "." + ext);
}
//...
//./responders/server-responder.js
export const extensions = [
"server.js",
"server.ts",
"server.jsx",
"server.tsx"
];
export function match(path){
const ext = path.split(".").filter(x => x).slice(1).join(".");
return extensions.includes(ext);
}
export function defaultPaths(barePath) {
return extensions.map(ext => barePath + "." + ext);
}
//...
//./responders/transpile-responder.js
const extensions = [
"jsx",
"ts",
"tsx"
];
export function match(path) {
const ext = path.split(".").filter(x => x).slice(1).join(".");
return extensions.includes(ext);
}
export function defaultPaths(barePath) {
return extensions.map(ext => barePath + "." + ext);
}
//...
It's a tad redundant to always get a file extension for match but it makes it easier to share the extension data with the defaultPaths. We'll also change the main function of each responder to be the default export so it's consistent. We also need to gather up all the responder modules.
//server.js
import { toFileUrl, resolve, join } from "https://deno.land/std@0.205.0/path/mod.ts";
const responders = await Promise.all(Array.from(Deno.readDirSync("./responders")).map(f => import(toFileUrl(resolve(join(Deno.cwd(), "./responders"), f.name)))));
A bit of path magic to make it work. The we'll need to fix up the default paths.
//./server.js in serve
//...
//normalize path
if (inputPath.endsWith("/")) {
inputPath += "index";
}
if(!inputPath.includes(".")){
potentialFilePaths.push(baseDir + inputPath + ".html");
potentialFilePaths.push(...responders.flatMap(responder => responder.defaultPaths(baseDir + inputPath)));
} else {
const path = baseDir + inputPath;
potentialFilePaths.push(path);
}
//find
const fileMatch = await probeStat(potentialFilePaths);
//...
Lastly I wanted to keep the default file responder as part of server.js. We'll always need at least one so if there's nothing in the responder folder then the whole thing becomes a basic file server.
Conclusion
So getting this to work without bundler magic is very hard. It's not surprising why NextJS is investing in a bundler. Though one thing that really sticks out is how much complexity we add for just miniscule dev ergonomics. Not using JSX and using something like htm would make all this easier (removing the bundler entirely), it's a lot of overhead to avoid a couple of quotes. React should really have a tagged-template mode. Also all of this is indirection is actually bad for dev ergonomics too! One of the reasons I did this is because I'm absolutely sick of magic caches and sorting through code that's been crushed by a bundler into something I don't recognize and can't easily debug. While we can't get rid of this completely (ts/jsx) this preserves the module import graph completely on the client-side making it easy to find things as you are working and preserving line numbers. This obviously is not useful for a production build and there's a lot of work that would need to go in to support both modes over the same code, but it's depressing no tools really work like this for local development.
Top comments (0)