Reducing system complexity by adding more components

Cover image: "PMA employees and a sophisticated computer system at PMA can be utilised by tenants of the Centre" by Public Record Office Victoria is licensed with CC BY-NC 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc/2.0/

OFTEN when designing and building systems, we are looking to simplify as much as possible, and this leads us to centralize as much of the work as possible into one layer of the stack. This anti-pattern can be seen repeatedly at many different scales in the system. Some examples:

• Putting all the HTML of a page inside frontend component JSX, instead of just the dynamic parts
• Building sophisticated form validation/date picker/etc. libraries in JS component libraries, instead of using HTML <form> or <input type="date"> elements
• Putting model validation logic in the application layer despite using a relational database like PostgreSQL
• (In the Node.js world) using JS-only tools like forever or pm2 to manage services in production
• Repeatedly validating a value (e.g. string is non-empty) despite using a statically-typed language where we could instead parse the string into a more meaningful custom type, like NonEmptyString

I'll go over each of these cases and make an argument for why it actually increases complexity.

Rendering static HTML in client-side components

Of course, there are many different kinds of frontend apps and some of them are actually Single-Page Apps (SPAs) that need to be fully rendered on the frontend. But there are many other pages where this doesn't need to happen. You can tell this is the case when people start talking about optimizing metrics like time to First Meaningful Paint, or Search Engine Optimization (i.e. making sure Google can crawl the site without needing to run JavaScript and thus lowering the page rank in search results).

Typically what happens next is people start talking about implementing things like React Server-Side Rendering (SSR) using Node.js and component hydration. Chances are, you already have a backend like say Python/Django, and now you need to add a Node.js backend on top of that because that's the only realistic way to do it. This is not simple.

Now let's consider not going down the path of throwing all your DOM content inside the frontend app, and instead rendering the static parts of the site (the 'skeleton') in the server side from the very beginning. This lets you start out with much better page load and SEO metrics, because SSR is now an inherent part of the app from the beginning. And you can focus on writing smaller components for the dynamic parts. Bonus: this also reduces your JavaScript payload and burden on the browser's JS engine to load, parse, and run the JS code.

But, you think, what about a page where dynamic components in different and unrelated parts of the DOM need to talk to each other? E.g. a dynamic product/category filter on the left sidebar of the page, which can filter search results in the main content area of the page.

Turns out that's what React Portals do. In fact, if you think about it, this is exactly what React was originally intended to do--make specific components of a page dynamic and allow them to talk to each other. This is going along the grain of React, not against it.

Re-implementing form validation/date pickers in JS

Similar to the previous point, because people have a tendency to pile everything into a single layer of the app that they understand well, when they need to do form validation or pick a date, they immediately reach out for some popular npm library that does those. But modern browsers already ship with sophisticated functionality for exactly these use cases! (Yes, I know some people need to support IE11 or other dead browsers, but most people don't, so...)

You need to do client-side form validation? Try HTML5's built-in form validation. It can:

• Enforce required inputs
• Enforce minimum and maximum lengths and amounts
• Enforce input matches a regular expression
• Enforce correct dates, times, email addresses--you don't need to write a customer regex for that, the browser provides <input type="email">
• Prevent form submission and show nice tooltips with helpful error messages
• And more.

You need a date picker? Try the built-in <input type="date">. It looks like this in Chrome:

Implementing model validation in the application layer

In my opinion, one of the bigger mistakes that people keep making is treating their sophisticated relational database engines like PostgreSQL/MySQL/Maria/etc. like simple key-value stores. Of course much has been written about this already so I won't launch into a full-fledged rant, but it's good to keep in mind that database engines like the above have been developed and optimized for literally decades and are very good at doing what they do:

• Enforce required fields
• Enforce fields match values or patterns, minimum or maximum lengths, etc.
• Enforce relationships between fields in the same table e.g. 'if status is 'error' then status_msg must be non-null and non-empty'
• Enforce relationships between fields across tables i.e. foreign keys
• Do upserts
• Automatically update child table rows when parent table rows are updated
• Automatically delete child table rows when parent table rows are deleted

Not only do they already implement all this functionality, but they also:

• Allow doing it in a declarative way i.e. you declare what should be done instead of providing step-by-step instructions on how to do it, making the process less error-prone
• Carry out multi-step actions like upserts, cascade updates, and deletes in a single transaction, avoiding potential multiple network trips and data inconsistencies.

Finally, you get the benefit of not having to re-solve already-solved problems like all of the above over and over again in your application code.

The argument against all this is usually that it makes the system more complex because it spreads the application logic over several layers. The thing is, we need to be thinking about what kind of complexity we're adding to the system and what the trade-offs are. Putting all validation logic in the application layer is not complexity-free either; as I mentioned above, it forces you to re-implement algorithms like 'check that the result field can have only the values SUCCESS or ERROR' in your application code, that have already been implemented in a general way by e.g. MySQL or Postgres enum types.

Again, it's the complexity of using and learning a system that has been highly optimized for certain tasks, versus trying to re-implement those tasks for the umpteenth time in your codebase.

Segue: SSL termination

Let's look at another case where it's widely accepted that adding the complexity of another layer to the system is the best practice: SSL termination with a reverse proxy like Nginx. Although many web server backends can serve HTTPS directly, we generally accept that it's a better idea in production to put a reverse proxy in front of the server and let it do the SSL termination. Why is that? Simple--the reverse proxy is specialized and highly optimized for that job.

The same argument applies with a relational database engine.

Using JS-only tools to manage services in the Node.js ecosystem

In the Node.js ecosystem, people often use Node-only tools like forever or pm2 to run their services in production, almost always in Linux VMs where the standard service manager, systemd, is available. Why?

I believe it's for the same reason why they try to use tools written in JavaScript for many other tasks, like builds, internationalization, file watching. There is a belief that using JS-only tools will leverage their existing skillset.

But this turns out to be the wrong choice for various reasons--for example, pm2 is AGPL-licensed, and can cause undesirable legal exposure in many corporate environments. Or there is the time pm2 caused build failures because it hit an analytics endpoint that returned a 503 HTTP error. Or just because pm2 just uses systemd under the hood to run itself on Linux.

Instead, it's straightforward to use systemd to manage the service (i.e. ensure restart on crash) and even start multiple instances and load balance between them, a key selling point of pm2.

Again, this is spreading out the overall system into another layer--in this case the OS's userland base system--but again it comes with a host of benefits:

• systemd is a key Linux tool and is actively maintained
• Installed out of the box in mainstream Linux distros, no additional install required
• Specialized for service management and implemented in C for performance
• Free and open source, no telemetry phoning home to unknown servers
• If there are issues with it a sysadmin has a much better chance of being able to help.

Redundant validation in multiple layers

Alexis King already wrote in depth on this one but it's worth repeating. Specifically, I believe it fits in the general pattern seen above--continuing to use a layer of the system (i.e. runtime validation) for a task it's not as optimized for (typechecking) instead of learning and using a different layer (the typechecker). Plenty of examples of this exist already but I'll provide a small one here (in Scala) for completeness--validating that an input string is not empty:

// NonEmptyString.scala

object NonEmptyString {
  def apply(input: String): String = {
    require(input != "", "input must not be empty string")
    input
  }
}

This does the job (at runtime) but it fails to provide any proof that it did the job. I.e., when you use it:

val result = NonEmptyString("hello")

You get back a result string that should have passed the validation, but the moment you pass that string somewhere else, that information is lost in the flow of the program. Instead if you use the type-level to encode the information 'the non-empty check was done':

// NonEmptyString.scala

class NonEmptyString private(override val toString: String) extends AnyVal

object NonEmptyString {
  def apply(input: String): NonEmptyString = {
    require(input != "", "input must not be empty string")
    new NonEmptyString(input)
  }
}

Now when you use this:

val result = NonEmptyString("hello")

You get result: NonEmptyString (or a runtime exception), which encodes at the type level that the validation was done.

(Note: extends AnyVal is a Scala feature that tries to ensure that NonEmptyString doesn't box the input string at runtime.)

Importantly, the only way to get an instance of NonEmptyString at runtime is by going through the validation routine (apply). This provides a strong guarantee that it was done, no matter where you pass the NonEmptyString in the program.

Conclusion

As a craft, software development is young and we developers are forgetful of what came before. We often over-specialize in certain parts of the stack and ignore anything going on slightly outside it. In this post, I try to show that it's worth stepping outside the niche and learning more layers of the stack and what they can do for us.

Of course, being more a craft than a science, the field is full of people with opinions, usually three or more opinions per every two people you might ask. So needless to say, the above is my opinion, from my viewpoint. I hope it will help you form your own through reasoning and experimentation.