Contents
- Introduction
- Context
- Set-Up Steps
- Exploring the payloads and testing with the AWS IoT Console MQTT Client
- Conclusion
- Author
Introduction
This post is a follow-up to:
Connect the Open Broadband USP Agent to AWS IoT Core using MQTT5, and Validate the Integration with AWS IoT Core Device Advisor
Alina Dima for IoT Builders ・ Dec 9 '22
In this post, you will see how to set up and use the native AWS IoT Rules Engine protobuf decoding SQL function, in order to decode USP records and messages sent by USP agents, on the fly, without the need for additional AWS Lambda functions with bespoke decoding logic.
Context
Protocol Buffers (protobuf) is an open-source data format used to serialize structured data in binary form. It is used for transmitting data over networks or storing it in files. Protobuf allows you to send data in small packet sizes and at a faster rate than other messaging formats. USP specifies Protocol Buffers Version 3 as a mechanism to serialize data to be sent over a Message Transfer Protocol, such as MQTT.
The protobuf decoding support in the AWS IoT Rules Engine was recently introduced (https://aws.amazon.com/about-aws/whats-new/2022/12/aws-iot-core-rules-engine-google-protocol-buffer-messaging-format/) to enable developers to decode protobuf payloads to JSON format directly in the rule, and route them to downstream services, by means of an AWS IoT Rules SQL function: decode(value, decodingScheme) (https://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html#iot-sql-decode-base64).
This feature is interesting because it removes the need to write and maintain custom protobuf decoding logic, which was, prior to this feature, usually done using AWS Lambda functions, invoked either as Rule Actions, or as part of IoT Rule SQL expressions (see the diagram below - before and after the native protobuf support in AWS IoT Rules Engine).
!(https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ssc6t3oihgvwqpp24ggp.png)
Set-Up Steps
Prepare the required protobuf files
As part of the AWS IoT Rule decoding pre-requisites, the protobuf descriptor file needs to be created and uploaded to an S3 bucket.
- Install the Protobuf Compiler (protoc) on your system. On MacOS, you could use Homebrew, as described here.
- Download the 2 proto files record and message from the open source USP GitHub repository, corresponding to the USP version that you want to use in your project. This post will work with the latest version, USP 1.2.
- Create the descriptor file required by the AWS IoT Rules Engine, as described in the docs
You can use the following command:
protoc —descriptor_set_out=usp-1-2.desc —proto_path=<PATH TO YOUR PROTO FILES> —include_imports usp-record-1-2.proto usp-msg-1-2.proto
Create the AWS Cloud resources
You will need to create the AWS IoT Rule, correct Roles and Policies, and S3 Bucket where the descriptor file will be uploded. You can use the AWS CloudFormation template below for this set-up. To create the AWS resources using this template, you can use the AWS CloudFormation CLI commands or the AWS SAM CLI commands.
For the purpose of this example, the Rule will decode the protobuf record, then decode the protobuf message (both using the decode () SQL function), and then republish the resulting JSON object, on a new topic, so we can examine correctness.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
USPProtobufDecodingRule:
Type: AWS::IoT::TopicRule
Properties:
RuleName: USPProtobufDecodingRule
TopicRulePayload:
AwsIotSqlVersion: 2016-03-23
RuleDisabled: False
Sql: SELECT VALUE decode(decode(encode(*, 'base64'), 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-record-1-2.proto', 'Record').noSessionContext.payload, 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-msg-1-2.proto', 'Msg') FROM '/usp/controller'
Actions:
- Republish:
RoleArn: !GetAtt RepublishDecodedRole.Arn
Topic: '/usp/controller/decoded'
RepublishDecodedRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- iot.amazonaws.com
Policies:
- PolicyName: allowRepublish
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: iot:Publish
Resource: !Join [ "", [ "arn:aws:iot:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":topic//usp/controller/decoded" ] ]
USPBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: usp-1-2-bucket
USPBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref USPBucket
PolicyDocument:
Version: 2012-10-17
Statement:
- Action:
- 's3:Get*'
Effect: Allow
Resource: !Join
- ''
- - 'arn:aws:s3:::'
- !Ref USPBucket
- /*
Principal:
Service:
- iot.amazonaws.com
The AWS IoT Rule SQL expression for decoding could look like below:
SELECT VALUE decode(decode(encode(*, 'base64'), 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-record-1-2.proto', 'Record').noSessionContext.payload, 'proto', 'usp-1-2', 'usp-1-2.desc', 'usp-msg-1-2.proto', 'Msg') FROM '/usp/controller'
The SQL rule behaves the following way:
- First, the binary MQTT payload is transformed in a based64-encoded string,
- Then, the base64 encoded string is protobuf-decoded at USP record level, as specified by the descriptor,
- Finally, the USP message inside the record, located in the payload object, is protobuf decoded.
Because USP messages are encoded and wrapped inside encoded USP records, you will need to call the decode function twice.
Upload the descriptor file and start obuspa
- Upload the descriptor file created in step 3 into your S3 bucket. You can do this using the AWS CLI or AWS Console.
- You can now set up and start the obuspa, as described in the following sections: Integrate obuspa with AWS IoT Core over MQTT5 and Verify MQTT 5 Integration/Protobuf Data Publishing.
Once the agent is sending data, the set-up is complete.
Exploring the payloads and testing with the AWS IoT Console MQTT Client
As the protobuf records arrive on the topic, the IoT Rule will pick them up, decode them, and republish the JSON objects on the /usp/controller/decoded
topic. If you subscribe with the test MQTT Client in the AWS IoT Console, you will be able to see the protobuf records on the /usp/controller
topic, and the JSON payloads on the /usp/controller/decoded
topic, as shown in the images below. The first image shows the original protobuf encoded USP record, arriving on the topic, and the second image the decoded USP message, in JSON format.
Conclusion
This post shows how to to decode protobuf records and messages sent by a USP agent, directly in the AWS IoT Rules Engine, using the decode function, as part of the rule expression. This is an AWS IoT Rules Engine feature released recently, making it easier to use protobuf directly in your IoT Rule, without the need to write and maintain custom decoding code in AWS Lambda functions.
Top comments (0)