A few sneak peeks into Hey.com technology (VI - Template page updates)

Update as of Dec 2020: now that the Hey support code has been extracted and officially released under the name Hotwire, we know that this article actually speaks about the Turbo Streams. Basecamp renamed and polished a few things but the principles stay the same as described here.

In parts III and IV we’ve seen how partial page updates can be done via “Turbolinks frames“ (the <turbolinks-frame> element), either automatically or upon request. That was nice and cool but that’s not the only way to do partial updates that Hey supports. In fact, I think that it’s even not the nicest and coolest way to do them! At least when compared to today’s topic - the <template> element…

This element (and associated JavaScript code) handles page updates in many places in Hey. Let’s now study a use case with just about the perfect level of complexity − non-trivial but not too complex: the “Mark seen” function. But first of all, we need to talk about how the main email listing is organized in the Imbox HTML code.

Email listing in Imbox

We all know that the Imbox page has two lists of emails: the new ones and the previously seen ones below. But how do they look in the HTML source? Let’s point our dev tool picker:

Oh, it’s just a single list! A single flat list of <article> elements. How do the new and previously seen emails differ then? As is apparent from the image, the already seen emails have a data-seen="true" attribute, whereas the new ones don’t.

OK, but how is the “PREVIOUSLY SEEN” header made then? Turns out it’s pure CSS and it’s a neat trick (see the image below): the CSS rule with the header in a :before pseudo-class targets the first element in the .postings (emails) list that has the data-seen attribute set. (Either the very first such element in the list when there are no new emails, or the first element after another that has not this attribute set.) So, simply adding this attribute to the email element prepends it with the header!

I think that this is a nice example of a presentation that is handled by a simple HTML structure with a few “sprinkles” of specific data attributes, CSS rules or a bit of JavaScript instead of e.g. handling all this logic in a more complex JS code. It’s an approach which uses the combined strengths of the “classic” technologies that, overall, have been here since many many years ago and are thus very well supported, tested and familiar to devs. No need to learn a new big framework every one or two years! This “composition pattern” can be seen …just about everywhere in Hey and I find it very sensible and appealing. And, most importantly, it plays very well with partial page updates via template elements…

The "Mark seen" action analysis

So what goes on when you select a new email and click on the “Mark seen” popup menu item? Let’s have a look!

Not surprisingly, the Mark seen item is a standard HTML form. It triggers a POST action to the server and the only data it sends along is the IDs of the email(s) selected. As Hey employs Turbolinks, the form is submitted asynchronously via AJAX by default.

BTW, the data-remote="true" attribute used to do a similar thing in Rails UJS, included by default in ActionView. Rails UJS, or the corresponding ActionView JS code, is not included among Hey JS scripts so I assume, the new Turbolinks handles the async form posting in a different way and this attribute is actually unused.

Anyway, this is not just a standard async form posting, there is one important detail going on: when Turbolinks code intercepts the form submit event, it first dispatches its own custom event called turbolinks:before-fetch-request. (This can be seen in the turbolinks/dist/fetch_request module, which is probably transpiled from TypeScript or similar language and is unfortunately not properly source-mapped, so it’s harder to grasp…) Still before actually fetching the AJAX request, this custom event is handled by JS code in initializers/page_updates.js and it modifies the Accept header in the request:

What does it mean? The Accept header tells the server what Media types (i.e. “types of data”, often called “MIME types”) the browser expects to receive back in the response. Turbolinks adds the "text/html; page-update" media type. Standards-wise, this is a common text/html media type with a custom parameter denoting that the browser will accept a “page update” response from the server and that the response should generally be a HTML response, of course!

So, to sum it up, the AJAX form submit request adjusted by Turbolinks looks like this in the Network tab:

And the server indeed responds with the “page update” media type as can be seen in the content-type response header:

So what does such response look like? Let’s take a look at its body:

Oh at last, here they are, the <template> elements!

Processing the page update response

Right after Turbolinks receives the response, it again triggers a custom event, turbolinks:before-fetch-response (in fetch_request.js), which allows some special treatment of the response. This event is again handled in the same initializer as was the request event − page_updates.js − in the handlePageUpdates method. This method first checks for the custom media type in the response content-type header and if it is present, it calls the processPageUpdates method from a tiny custom library called page-updater:

I assume the reason why this JS code is not part of Turbolinks but is a separate library is that it is, indeed, independent − you can employ such a thing anywhere, you just need to render proper response content on your server.

The page-updater library is pleasantly small and concise. All it does is it extracts all <template> elements that have the data-page-update attribute set, parses them into individual page update commands and executes them, one by one.

The commands are encoded in the <template> elements in a simple way:

<template data-page-update="command#html-id">
...
</template>

where command is the operation that is about to be run and html-id is the… you guessed it… HTML ID of the element the operation should be run against. Optionally, the <template> element can also have its own content, which is needed for some commands. There are five distinct page update commands defined: append, prepend, replace, update and remove. They are quite self-explaining, I think, maybe I’ll just add that the update command leaves the target element intact and only updates its content whereas replace takes away the content as well as the target element itself. Perhaps, it’ll be best to show a picture instead of a ”thousand words“:

Unfortunately, there is no morph command, i.e. you cannot make only the smallest possible changes to the element (such as changing only its class or attribute), you have to resort to updating the whole thing. I see this as a disadvantage when compared to StimulusReflex which leverages the awesome morphdom library to effectively morph elements on the page.

And by the way, as it turns out, <template> element is defined in the HTML standard and denotes “an element for holding HTML that is not to be rendered immediately when a page is loaded but may be instantiated subsequently during runtime using JavaScript”. It seems like a perfect fit for what this element actually does in Hey!

The “Mark seen” response processing

So, let’ get back to the “Mark seen” action. In the response body image above we can see that the response contains two page update commands:

1. remove the email element from the page,
2. prepend the new version of the email element (given in the response) at the beginning of emails list.

Have you spotted anything weird here? How come the returned mail element is put at the beginning of the emails listing? We already know that we need this email element somewhere in the middle of the list, as it’s a single flat list and we still have some unseen emails at the top!

Eh, you know what? This is getting long and I’m going to cowardly cut this post here. I have a lot more thoughts about this topic! While I generally like this approach very much, I can see some possible caveats, too. Next time, I’ll finish the partial page updates analysis as well as try to discuss this pattern overall, and compare it to the “Turbolinks frames” pattern. Stay tuned and in the mean time you may try to solve the puzzle with mails ordering…! 😉