loading...
Cover image for SMoS: Structured Messaging over Serial (Part 1)

SMoS: Structured Messaging over Serial (Part 1)

chrisdinhnz profile image Chris Dinh ・5 min read

Part 6 of An IoT Odyssey

In the previous post I wrote about how we can use a HM-10 Bluetooth module to send a text message from an mobile application to a terminal on a computer and vice versa. In that scenario there was a context i.e. both ends of the communication link knew the type of data being transmitted and therefore knew how to consume it.

But if we abstract away the context, then all we have left is two devices sending binary data across a serial link. Without any context all we see is 0s and 1s going across the serial link. So to understand the data, we must ask ourselves the following questions:

  • Where does the message starts and where does it ends?
  • What is the message?
  • What should we do with the message?

In this post we will discuss and define a protocol we can use to answer to the aforementioned questions. By having a defined protocol, we enable devices to be less coupled to one another and improve interoperability. This may come across as a bit of an overkill but I think we 'll learn a lot along the way and in the long run, having this protocol may be very well worth it.

Where does the message starts and where does it ends?

Every message has a beginning and an end. Currently all we know is that bytes of data is being sent from one end to the other, bit by bit across the serial link. We needed some kind protocol to make sense of these data. After some browsing around I came across the following protocols:

Of the three, I found the Intel Hex Format's structure to be most suited to our need while not being too overly complicated.

Intel Hex Format

Intel Hex is a binary-to-text encoding format, i.e. it allows binary data to be encoded in a printable (ASCII) format. It has six standard record types:

Intel Hex Record Types

For us we don't intend to send files or program across the serial link therefore we are only interested in the Data record type. We also don't intend to persist the data so the address field will be unused for now (i.e. address field = 0x0000). If you are curious on how the data record is structured, a more detailed explanation can be found here, we will go through some real world scenarios in my next post.

What is the message?

Given we now have the capability to decode the 0s and 1s into a message, our next task is to make sense of the message's content. It could be a string, a number, a toggle state. But how do we know? We need the sender to tag the content for the receiver the make sense of it.

Luckily there are a few well known standard protocols out there that we can model. Bluetooth BLE is one such protocol and is very well documented. We will lean somewhat on their Bluetooth GATT Specifications for the services and data definition so that we don't need to reinvent the wheel. Based on GATT Specifications, every piece of data is associated with a 16-bit assigned number. We will be using the same assigned number to tag the data going across the serial link. However, there will also be option to send generic data in which case the receiver is assumed to know the context in advance. We will go through some real world scenarios in my next post.

What should we do with the message?

Data is meaningless by itself. For the receiver to correctly process the data we need to know the context the data is in. This will be done by tagging the messages with actions (e.g. CRUD operations against databases, GET, POST, PUT, DELETE methods on REST servers). In our case we will model the Constrained Application Protocol (CoAP) as needed to provide the required data context. You can learn more about CoAP here. We will go through some real world scenarios in my next post.

So what does SMoS look like?

Now that we have discussed the questions we wanted to ask, we can start putting the pieces together and see in theory what Structured Messaging over Serial looks like.

The SMoS Structure

The structure of a SMoS message can be broken down as follow:

The SMoS Structure

Start Code
  • Every message begins with a colon (ASCII Hex value $3A)
Byte Count
  • A 2 digit value (1 byte), counting the actual data bytes in the message.
Data Context
  • A 6 digit (3 byte) value, containing various meta data about the message content.
Data Content
  • There can be 0 to 255 data bytes per message.
Checksum
  • This field is a one byte (2 hex digits) 2's complement checksum of the entire record.
The Data Context

Unlike the Intel Hex data record, the address and record type fields are not used. Instead these three bytes will be redefined as follow:

Data Context Structure

Context Type
  • Two MSB bits of the first byte.
  • The values is defined as follow:
    • Confirmable (0x0)
    • Non Confirmable (0x1)
    • Acknowledgement (0x2)
    • Non Acknowledgement (0x3)
Content Type
  • Two bits following Context Type.
  • The values is defined as follow:
    • Generic content (0x0)
    • Bluetooth GATT content (0x1)
    • Reserved (0x2)
    • Reserved (0x3)
Content Type Options
  • Four LSB bits of the first byte.
  • It's values are determined by the Content Type.
Code Class
  • Three MSB bits of the second byte.
  • The values is defined as follow:
    • Request (0x0)
    • Success response (0x2)
    • Failed response, sender error (0x4)
    • Failed response, recipient error (0x5)
Code Detail
  • Five LSB bits of the second byte.
  • If code class is 0x0 and code detail is 0x0, it is an empty message.
  • If code class is 0x0 and code detail is not 0x0, then the values is defined as follow:
    • GET, OBSERVE (0x1)
    • POST (0x2)
    • PUT (0x3)
    • DELETE (0x4)
  • If code class is 0x2, then the values is defined as follow:
    • CREATED (0x1)
    • DELETED (0x2)
    • VALID (0x3)
    • CHANGED (0x4)
    • CONTENT (0x5)
  • If code class is 0x4, then the values is defined as follow:
    • BAD REQUEST (0x0)
    • UNAUTHORIZED (0x1)
    • BAD OPTION (0x2)
    • FORBIDDEN (0x3)
    • NOT FOUND (0x4)
    • METHOD NOT ALLOWED (0x5)
    • NOT ACCEPTABLE (0x6)
    • PRECONDITION FAILED (0x12)
    • REQUEST ENTITY TOO LARGE (0x13)
    • UNSUPPORTED CONTENT FORMAT (0x15)
  • If code class is 0x5, then the values is defined as follow:
    • INTERNAL SERVER ERROR (0x0)
    • NOT IMPLEMENTED (0x1)
    • BAD GATEWAY (0x2)
    • SERVICE UNAVAILABLE (0x3)
    • GATEWAY TIMEOUT (0x4)
    • PROXYING NOT SUPPORTED (0x5)
Message Id
  • Four MSB bits of the third byte.
  • Used to match messages of type Acknowledgement/Reset to messages of type Confirmable/Non-confirmable, and guard against message duplication.
Token Id
  • Four LSB bits of the third byte.
  • Used to correlate notifications with OBSERVE requests.
The Data Content

The actual user/application data with varying length.

Data Content Structure

Discussion

pic
Editor guide