Day 22: The Division of Labor Between XML, JSON and MessagePack

The previous post, Business Logic Customization and the Four Plugin Stages, closed by saying the level changes here. From this point the subject is no longer what a definition declares, it is how those definitions and that data travel once they leave memory.

Serialization is two separate jobs. One is persistence: an object has to be stored and read back unchanged next time, which is how definitions are stored and how system data such as SessionUser is stored. The other is transport: an object has to reach the other end, and the other end is not one thing. Desktop, browser, iOS and Android all run in different environments.

This post covers:

1. Which route persistence takes, why transport splits into two, and what the dividing line is drawn on
2. How PayloadFormat decides what a call's value looks like on the way out
3. Why definition style things go out as an XML string rather than a JSON tree
4. Why not one transport type can be left unregistered
5. On the deserialization end, who decides the type, and why only one route needs an allowlist

1. One route for persistence, two for transport

Persistence wants something a human can read, that can go into version control and be edited by hand, and that job goes through XML from end to end.

Transport splits into two, divided on whether the caller is .NET:

• Both ends .NET: value goes through MessagePack
• The front end is not .NET, a pure JavaScript front end or a third party integration: value goes through JSON

The outer layer is the same on both routes: one call sends a JSON-RPC payload, and the protocol says that is JSON, which cannot be swapped out. What there is a choice about is the value slot inside it. Without separate encoding it is simply part of that JSON, and only once it is encoded do the two routes above come into play.

One route for persistence plus two for transport gives three formats, each doing what it is good at:

Whether a format needs an outside package is not a footnote on that table, it is the reason for which layer uses it. A format that needs one cannot sit in the definition layer. The definition layer is at the bottom of the dependency graph, with the database, the repository, the business logic, every front end and the definition file tooling all downstream of it, so any package added there spreads along the dependency chain to every consumer, and not one of them has a use for it.

What that buys is a definition layer that changes not at all when the transport format changes later. The cost is that the transport binding has to be written separately in the API layer, and there is no compiler holding that copy and the definition types together.

2. PayloadFormat decides what value looks like on the way out

When JSON-RPC 2.0: Protocol Choice and the Request Pipeline went through the params slot it mentioned format and type, noting only that they sit on the API payload rather than inside value. What is in those two slots is this section.

format has three values, all deciding the same thing: what form this call's value takes on the way out.

The compression and encryption pipeline itself is the next post's subject.

The same method works under all three values, which is exactly the point of putting the format on the API payload rather than in the method name or the path. Send a Plain API payload to the case study's login and value is readable JSON:

{"jsonrpc":"2.0","method":"System.Login",
 "params":{"format":0,"value":{"userId":"demo","password":"demo"},"type":""},"id":"1"}

What comes back is readable too:

{"jsonrpc":"2.0","method":"System.Login",
 "result":{"format":0,"value":{"accessToken":"033bedcf-...","userName":"Demo User",
 "timeZone":"Asia/Taipei"},"type":""},"id":"1"}

Change format to 1 on the same method, with nothing else touched, and what comes back is an error:

{"jsonrpc":"2.0","method":"System.Login",
 "error":{"code":-32099,"message":"TypeName is missing for deserialization."},"id":"1"}

The type slot is used only by the latter two values, and the reason explains precisely where the difference between the three lies. A Plain value never leaves JSON at any point, so what the server receives is a JsonElement with no type yet, and it is deserialized into the parameter type that method declares only after reflection has found it. The type comes from the method signature, not from the API payload. The other two are different: they pack the object into a string of bytes on the calling end, and bytes carry no type information of their own, so the type name has to be written onto the API payload and sent along. That slot is therefore a field the caller decides, and the server has to check it against a type allowlist before it reaches Type.GetType.

Plain is not a debugging side road, it is the startup path. Probing connectivity, fetching shared settings and establishing a connection all happen before a transport key exists, and at that point there is nothing to encrypt with.

Three components, two ways of deciding

The three components on value are each an interface plus a factory that resolves an implementation by name, and they differ in where the name comes from:

A name with no matching implementation throws, with no quiet fall back to a default.

They split into two because they are not answering the same kind of question. Compression and encryption answer whether this deployment protects value and with what, which is the deploying party's decision. The serializer answers which kind of value this caller is able to write, which is a capability of the caller. Folding the latter into deployment settings too would require every caller on one server to speak the same language, and the two routes in the first section do not speak the same language to begin with.

A call that declares nothing is treated as MessagePack, because that is what callers predating this slot send.

All three components are shut inside two places: the Connector on the calling end, and the API layer on the server. The method that fetches one page of a list has only query fields, conditions, sorting and paging in its signature, with no parameter relating to format at all, and what the business object receives on the far end is a restored object. So swapping in a better suited transport format later changes the serializer slot, and no application code on either end has to change.

3. Why definition style things go out as an XML string

The previous section said value can be a JSON tree. So fetching a FormSchema should give back one too. It does not. Ask the case study for an order's FormSchema with the same Plain API payload, and value has exactly one field in it:

{"result":{"format":0,"value":{"xml":"<?xml version=\"1.0\" encoding=\"utf-8\"?>
<FormSchema ProgId=\"Order\" CategoryId=\"company\" ...>
  <Tables><FormTable TableName=\"Order\" DbTableName=\"ft_order\" ..."},"type":""}}

An XML string, packed inside the JSON tree that is the API payload. Everything definition style takes this route. The API layer only delivers the string, and restoring it into an object is done by the Connector on the calling end, so XmlSerializer has to run on every front end as well.

The reason is that the persistence copy is already there and usable: definitions land on disk as XML to begin with, so sending them out reuses the same serialization at no extra cost.

Letting definition types go through JSON or MessagePack too would cost something continuously. Every additional serialization adds its own set of things to keep track of: whether a round trip loses data, and whether deserialization can be stopped from constructing a type that should not be constructed. Definitions are a nested structure that evolves with each version, so every added property would have to be confirmed again in every format, and swapping in a better suited transport format later would mean revalidating the whole set of definition types once more.

Their present shape is genuinely unsuited to it as well: collection properties are read only, each holding a reference to its own owner. The three serializers happen to split into two camps over that kind of property:

The difference between the two camps is not capability, it is direction. And on the side that builds a new collection, whether it can be assigned to is the same question as whether that member exists at all.

4. Not one transport type can be left unregistered

The rules on the MessagePack side are a separate set. Given that it does not even look at read only properties, the least effort route is to let it find the members itself by reflection, with nothing declared. On a desktop that does run.

The problem is that this route relies on generating code at run time, and not every execution environment can supply that. Where it cannot, an unregistered type is not slower, it throws. So every transport type has to have a formatter registered explicitly, with the members listed out one at a time. The following is from the form family's WireContracts.Form.cs:

WireContract.For<GetListRequest>()
    .Member(nameof(GetListRequest.SelectFields), static x => x.SelectFields, static (x, v) => x.SelectFields = v)
    .Member(nameof(GetListRequest.Filter), static x => x.Filter, static (x, v) => x.Filter = v)
    // remaining members listed one at a time
    .Build();

That nameof slot is not only about readability. What the formatter writes out is a map keyed by property name, an unrecognized key on the way back in is skipped, and a key that never arrived leaves the property at the default declared on it. The one method one parameter object rule from the post on the request pipeline therefore buys one more thing: adding a property to an Args type does not break callers that have not upgraded yet. That is a property this present shape happens to give, not a promise anybody is holding, and no test verifies it.

The verbosity is unavoidable: switching to one generic formatter that reads property types puts you straight back on the route that generates code at run time. So this list can only be written by hand, and there is no generator to rerun.

Collections carry several further traps of their own on this route, some of which behave perfectly normally on a development machine, so testing fine on a desktop cannot be taken as evidence.

Nothing in the compiler holds this list and the types together, and what keeps it from drifting is a set of tests. They start from the same set of message types, walk everything reachable, and compare in both directions: everything reachable must be registered, and anything registered that nothing can reach is reported too. Those tests can also be run on a development machine simulating that kind of environment, rather than waiting for a deployment to find out.

The whole package is kept in one place: the package reference, all the formatters and this list all sit in one folder under Bee.Api.Core, and no other assembly has to know MessagePack exists. That is exactly what the outside package question in the first section was asking.

5. On the deserialization end, who decides the type

Deserialization turns bytes sent from outside into an object inside the process. How dangerous that is depends on who holds the decision about which type it becomes, and there are three different answers here.

The XML route is the most closed. When the server stores a definition back, the type is not read out of the XML content, it is looked up in a fixed table using the DefineType on the request, and a value that is not in the table throws.

The Plain route is decided by the method signature, which is the JsonElement from the second section: the caller does not even have a field it could put a type in.

The encoded route is the only one the caller decides, through the type field from the second section, and it works the same way whichever serializer is used. Bytes carry no type, so the type name can only be written onto the API payload and sent along, which is why there has to be an allowlist before Type.GetType. It filters the whole name string, letting through neither generic arguments nor array element types, and refuses anything it cannot parse. The entire check happens before the object is constructed, not after.

Without it, the caller would in effect be specifying which type the server constructs. The danger is not in the data that arrives, it is that some types do something as soon as they are constructed, and constructing objects is exactly what deserialization does.

The allowlist itself has two layers: a fixed set of basic types, plus namespaces declared when the application starts, with the framework's own as the default. The wider the second layer is opened, the less this check stops, and that is the application's own choice.

A correct type does not mean the content cannot misbehave. The XML side additionally forbids DTDs. XmlSerializer reading a string directly already blocks the fetch an outside resource variety, but abbreviations a document declares for itself still get expanded, and that is enough for a file of a few kilobytes to consume all available memory once expanded. The framework routes all XML in through the same tightened reader, so no individual caller has to remember this.

Back to Northwind

On the case study's server, the code relating to serialization is zero lines. The one JsonSerializer in it is in the table seeding program, reading its own seed data, unrelated to transport.

The front end is the same. Four front ends share one UI project, and that project has not one line handling formats. Even restoring the XML for definitions is done by the Connector. All three serializations are shut inside that layer, and the code above it does not have to know which one is used, how it is compressed, or how it is encrypted.

Zero code does not mean zero cost. Of the four, all but the desktop one carry an entry in their build files granting an exception. The browser one runs on wasm and has to turn reflection based serialization on explicitly, or the very first call cannot be sent, because the framework's message types do not go through source generation and every call passes through that JSON-RPC payload layer. What is caught there is not value, it is the outer layer. What the application pays lands in the build configuration, not in the source code.

In closing

Persistence goes through XML from end to end: definitions have to land on disk anyway, so sending them out reuses the same copy at no extra cost.

Transport splits into two layers. The outer JSON-RPC payload is fixed by the protocol, and the choice is in the value slot inside it: both ends .NET goes through MessagePack, which is also what applies when nothing is declared, while a pure JavaScript front end and third party integrations go through JSON.

And all three components on value are shut inside two places, the Connector on the calling end and the API layer on the server. Replacing any one of them later needs not a single line of application code changed.

The next post, The API Payload Security Pipeline: Order, Protection Levels and Keys, carries on with the other half of value: the compression and encryption that follow serialization, and why that order cannot be swapped.

📚 Part of the thirty-post series ERP Architect's Notes: Designing a Definition-Driven Framework

📘 Original article on HackMD, in Traditional Chinese:
👉 https://hackmd.io/@jeff377/day-22-three-formats

📢 You are welcome to republish this, please credit the source.
📬 Follow for more engineering notes:
Facebook ｜ HackMD ｜ GitHub ｜ NuGet