Introduction
Managing a large fleet of embedded devices can be complex and challenging, particularly when it comes to creating a single image that can be flashed onto multiple devices. These devices must be able to self-provision, utilizing unique information such as their serial number, upon initial boot. In this blog post, we will discuss how AWS IoT Greengrass - Fleet Provisioning can streamline this process for embedded Linux devices, making it more efficient and reliable.
For embedded systems engineers experienced in Embedded Linux and Yocto, we will guide you through building a Raspberry Pi Yocto image with Greengrass with Fleet Provisioning Plugin. This ensures seamless device provisioning and management, as well as automatic registration and configuration.
Please note here that the pre-provisioning lambda is optional but encouraged in order to additional layer of security. We will not be covering it in this post. You can learn more about it here.
With the stage set, let's dive into the prerequisites for setting up AWS IoT Greengrass and Fleet Provisioning for your embedded Linux devices.
Prerequisites
Before diving into the process of preparing the host and configuring the Yocto image build, it's essential to set up AWS IoT Core. This involves creating policies, obtaining claim certificates, and ensuring that the AWS CLI is installed and configured. General information on how to accomplish this can be found in the AWS IoT Greengrass Developer Guide.
In summary, we will need to:
- A token exchange IAM role, which core devices use to authorize calls to AWS services and An AWS IoT role alias that points to the token exchange role.
- An AWS IoT fleet provisioning template. The template must specify information needed for creating thing and policy which will be attached to greengrass core device created. You can either use existing IoT policy name or define the policy on the template.
- An AWS IoT provisioning claim certificate and private key for the fleet provisioning template.
Devices can be manufactured with a provisioning claim certificate and private key embedded in them. When the device connects first time to AWS IoT, it uses the claim certificate to register the new device and exchange it to unique device certificate. Provisioning claim certificate needs to have AWS IoT policy attached which allows devices to register and use the fleet provisioning template.
To make this process more efficient, we can utilize a CloudFormation template that automates most of these steps:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
ProvisioningTemplateName:
Type: String
Default: 'GreengrassFleetProvisioningTemplate'
GGTokenExchangeRoleName:
Type: String
Default: 'GGTokenExchangeRole'
GGFleetProvisioningRoleName:
Type: String
Default: 'GGFleetProvisioningRole'
GGDeviceDefaultPolicyName:
Type: String
Default: 'GGDeviceDefaultIoTPolicy'
GGProvisioningClaimPolicyName:
Type: String
Default: 'GGProvisioningClaimPolicy'
Resources:
GGTokenExchangeRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Ref GGTokenExchangeRoleName
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- credentials.iot.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: '/'
Policies:
- PolicyName: !Sub ${GGTokenExchangeRoleName}Access
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- 'iot:DescribeCertificate'
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
- 'logs:DescribeLogStreams'
- 's3:GetBucketLocation'
Resource: '*'
GGTokenExchangeRoleAlias:
Type: AWS::IoT::RoleAlias
Properties:
RoleArn: !GetAtt GGTokenExchangeRole.Arn
RoleAlias: !Sub ${GGTokenExchangeRoleName}Alias
GGFleetProvisioningRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Ref GGFleetProvisioningRoleName
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- iot.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: '/'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSIoTThingsRegistration'
GGDeviceDefaultPolicy:
Type: AWS::IoT::Policy
Properties:
PolicyName: !Ref GGDeviceDefaultPolicyName
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'iot:Connect'
- 'iot:Publish'
- 'iot:Subscribe'
- 'iot:Receive'
- 'iot:Connect'
- 'greengrass:*'
Resource: '*'
- Effect: Allow
Action:
- 'iot:AssumeRoleWithCertificate'
Resource: !GetAtt GGTokenExchangeRoleAlias.RoleAliasArn
GGFleetProvisionTemplate:
Type: AWS::IoT::ProvisioningTemplate
Properties:
TemplateName: !Ref ProvisioningTemplateName
Description: 'Fleet Provisioning template for AWS IoT Greengrass.'
Enabled: True
ProvisioningRoleArn: !GetAtt GGFleetProvisioningRole.Arn
TemplateBody: !Sub |+
{
"Parameters": {
"ThingName": {
"Type": "String"
},
"ThingGroupName": {
"Type": "String"
},
"AWS::IoT::Certificate::Id": {
"Type": "String"
}
},
"Resources": {
"GGThing": {
"OverrideSettings": {
"AttributePayload": "REPLACE",
"ThingGroups": "REPLACE",
"ThingTypeName": "REPLACE"
},
"Properties": {
"AttributePayload": {},
"ThingGroups": [
{
"Ref": "ThingGroupName"
}
],
"ThingName": {
"Ref": "ThingName"
}
},
"Type": "AWS::IoT::Thing"
},
"GGDefaultPolicy": {
"Properties": {
"PolicyName": "${GGDeviceDefaultPolicyName}"
},
"Type": "AWS::IoT::Policy"
},
"GGCertificate": {
"Properties": {
"CertificateId": {
"Ref": "AWS::IoT::Certificate::Id"
},
"Status": "Active"
},
"Type": "AWS::IoT::Certificate"
}
}
}
GGProvisioningClaimPolicy:
Type: AWS::IoT::Policy
Properties:
PolicyName: !Ref GGProvisioningClaimPolicyName
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- 'iot:Connect'
Resource: '*'
- Effect: Allow
Action:
- 'iot:Publish'
- 'iot:Receive'
Resource:
- !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/certificates/create/*'
- !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/provisioning-templates/${ProvisioningTemplateName}/provision/*'
- Effect: Allow
Action:
- 'iot:Subscribe'
Resource:
- !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/certificates/create/*'
- !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/provisioning-templates/${ProvisioningTemplateName}/provision/*'
Outputs:
GGTokenExchangeRole:
Description: Name of token exchange role.
Value: !Ref GGTokenExchangeRole
GGTokenExchangeRoleAlias:
Description: Name of token exchange role alias.
Value: !Ref GGTokenExchangeRoleAlias
GGFleetProvisionTemplate:
Description: Name of Fleet provisioning template.
Value: !Ref GGFleetProvisionTemplate
GGProvisioningClaimPolicy:
Description: Name of claim certificate IoT policy.
Value: !Ref GGProvisioningClaimPolicy
Save the file and create CloudFormation stack from template.yaml:
aws cloudformation create-stack --stack-name GGFleetProvisoning --template-body file://gg-fp.yaml --capabilities CAPABILITY_NAMED_IAM
Wait few minutes for resources being created. You can check status from CloudFormation console or with command:
aws cloudformation describe-stacks --stack-name GGFleetProvisoning
Create claim certificate
These we will be embedded in our RPi SD Card Image and used to provision our devices.
mkdir claim-certs
export CERTIFICATE_ARN=$(aws iot create-keys-and-certificate \
--certificate-pem-outfile "claim-certs/claim.cert.pem" \
--public-key-outfile "claim-certs/claim.pubkey.pem" \
--private-key-outfile "claim-certs/claim.pkey.pem" \
--set-as-active \
--query certificateArn)
curl -o "claim-certs/claim.root.pem" https://www.amazontrust.com/repository/AmazonRootCA1.pem
Attach the AWS IoT policy to the provisioning claim certificate
As we created IoT policy named GGProvisioningClaimPolicy with CloudFormation we can just use the name to attach the policy:
aws iot attach-policy --policy-name GGProvisioningClaimPolicy --target ${CERTIFICATE_ARN//\"}
Create a Thing Group
Once our devices get provisioned they will become part of this Thing Group allowing us later to target Thing Group Fleet Deployments.
aws iot create-thing-group --thing-group-name EmbeddedLinuxFleet
As of now we should be good to go and build our RPI image.
Building RPi Image
Building a Yocto image for Raspberry Pi requires several steps, including setting up the build environment, cloning the necessary repositories, configuring the build, and finally, building the image itself. Here's a step-by-step guide to help you through the process:
Open a terminal window on your workstation which has all the prerequisits based on the Yocto Project Build Doc.
NOTE For the sake of this tutorial, the variable
BASErefers to the build environment parent directory. Here, this will be set to$HOME. If you are using another partition as the base directory, please set it accordingly.
export BASEDIR=$(pwd)
export DIST=poky-rpi4
export B=kirkstone
Clone the Poky base layer to include OpenEmbedded Core, Bitbake, and so forth to seed the Yocto build environment.
git clone -b $B git://git.yoctoproject.org/poky.git $BASEDIR/$DIST
Clone additional dependent repositories. Note that we are cloning only what is required for AWS IoT Greengrass.
git clone -b $B git://git.openembedded.org/meta-openembedded \
$BASEDIR/$DIST/meta-openembedded
git clone -b $B git://git.yoctoproject.org/meta-raspberrypi \
$BASEDIR/$DIST/meta-raspberrypi
git clone -b $B git://git.yoctoproject.org/meta-virtualization \
$BASEDIR/$DIST/meta-virtualization
git clone -b $B https://github.com/aws4embeddedlinux/meta-aws \
$BASEDIR/$DIST/meta-aws
Source the Yocto environment script. This seeds the build/conf directory.
cd $BASEDIR/$DIST
. ./oe-init-build-env
Add necessary layers to bblayers.conf using bitbake-layer add-layer:
bitbake-layers add-layer ../meta-openembedded/meta-oe
bitbake-layers add-layer ../meta-openembedded/meta-python
bitbake-layers add-layer ../meta-openembedded/meta-filesystems
bitbake-layers add-layer ../meta-openembedded/meta-networking
bitbake-layers add-layer ../meta-virtualization
bitbake-layers add-layer ../meta-raspberrypi
bitbake-layers add-layer ../meta-aws
Configure the local.conf:
Here it is important to note that apart from standard raspberry pi configuration:
MACHINE ?= "raspberrypi4-64"
DISABLE_VC4GRAPHICS = "1"
# Parallelism Options
BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
# Additional image features
USER_CLASSES ?= "buildstats"
# By default disable interactive patch resolution (tasks will just fail instead):
PATCHRESOLVE = "noop"
# Disk Space Monitoring during the build
BB_DISKMON_DIRS = "\
STOPTASKS,${TMPDIR},1G,100K \
STOPTASKS,${DL_DIR},1G,100K \
STOPTASKS,${SSTATE_DIR},1G,100K \
HALT,${TMPDIR},100M,1K \
HALT,${DL_DIR},100M,1K \
HALT,${SSTATE_DIR},100M,1K"
CONF_VERSION = "2"
DISTRO_FEATURES += "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = ""
IMAGE_FSTYPES = "rpi-sdimg"
We should focus on our Greengrass FleetProvisioning configuration part, which should look like this:
IMAGE_INSTALL:append = " greengrass-bin "
GGV2_DATA_EP = "xxx-ats.iot.<your aws region>.amazonaws.com"
GGV2_CRED_EP = "xxx.iot.<your aws region>.amazonaws.com"
GGV2_REGION = "<your aws region>"
GGV2_THING_NAME = "ELThing"
GGV2_TES_RALIAS = "GGTokenExchangeRoleAlias" # we got this from the cloudformation
GGV2_THING_GROUP = "EmbeddedLinuxFleet"
PACKAGECONFIG:pn-greengrass-bin = "fleetprovisioning"
Here it is important to note that we are adding greengrass-bin to our image and then providing additional configuration required by the config.yaml as well as adding PACKAGECONFIG:pn-greengrass-bin = "fleetprovisioning" in order to enable the functionality.
In order to get the AWS region and the IoT endpoints we can do the following:
echo "GGV2_REGION="$(aws configure get region)
echo "GGV2_DATA_EP="$(aws --output text iot describe-endpoint \
--endpoint-type iot:Data-ATS \
--query 'endpointAddress')
echo "GGV2_CRED_EP="$(aws --output text iot describe-endpoint \
--endpoint-type iot:CredentialProvider \
--query 'endpointAddress')
Please note that we will need a unique Thing Name to be generated for each device, so here the Thing Name is taken as a prefix and there is script inside of a greengreass-bin recipe that appends the unique device id to the Thing Name using the MAC address.
#!/bin/sh
file_path="$1"
default_iface=$(busybox route | grep default | awk '{print $8}')
mac_address=$(busybox ifconfig "$default_iface" | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}' | tr ':' '_')
sed -i "s/<unique>/$mac_address/g" "$file_path"
meta-aws
└── recipes-iot
└── aws-iot-greengrass
└──files
└── replace_board_id.sh
Feel free to replace this file with any other way of obtaining the uniqueness such as serial number or similar
Finally we should copy our claim credentials we generated at the beginning to be included in our build:
cp "claim-certs/claim.cert.pem" \
"claim-certs/claim.pkey.pem" \
"claim-certs/claim.root.pem" \
$BASEDIR/$DIST/meta-aws/recipes-iot/aws-iot-greengrass/files/
Please adjust the paths based on the location of generated certs and the recipe.
After all of this we should proceed with building our image.
bitbake core-image-minimal
⌛ Couple of hours later ⌛ the build should be complete, and we can find the resulting image in the following directory:
ls tmp/deploy/images/raspberrypi4-64/*sdimg
To flash the image onto an SD card, use a tool like 'dd'
sudo dd if=tmp/deploy/images/raspberrypi4-64/core-image-minimal-raspberrypi4-64.sdimg of=/dev/sdX bs=4M
Where we need to make sure to replace "/dev/sdX" with the appropriate device identifier for the SD card.
⚠️ Please double check the SD card identifier as a mistake here can wipe your workstation system
Powering the Device for the First Time
Once the SD card is reinserted into the Raspberry Pi with power and internet connected, the device should perform provisioning and appear in the list of Greengrass core devices.:
aws greengrassv2 list-core-devices
{
"coreDevices": [
{
"coreDeviceThingName": "ELThing_11_22_33_44_55_60",
"status": "HEALTHY",
"lastStatusUpdateTimestamp": "2023-04-25T15:39:00.703000+00:00"
},
{
"coreDeviceThingName": "ELThing_11_22_33_44_55_61",
"status": "HEALTHY",
"lastStatusUpdateTimestamp": "2023-03-31T03:11:17.911000+00:00"
},
{
"coreDeviceThingName": "ELThing_11_22_33_44_55_62",
"status": "HEALTHY",
"lastStatusUpdateTimestamp": "2023-02-25T15:17:29.505000+00:00"
},
]
}
Success!
Conclusion
To sum it up, managing a large fleet of embedded Linux devices can be a complex and challenging task, especially when it comes to creating a single image that can be flashed onto multiple devices. However, AWS IoT Greengrass with Fleet Provisioning can streamline this process and make it more efficient and reliable. In this blog post, we have discussed the prerequisites for setting up AWS IoT Greengrass and Fleet Provisioning, including creating policies, obtaining claim certificates, and configuring the Yocto image build. We have also provided a step-by-step guide to building a Yocto image for Raspberry Pi with Greengrass Fleet Provisioning configuration. Using AWS IoT Greengrass - Fleet Provisioning, managing a large fleet of embedded devices can be made easier, more efficient, and secure.
If you have any feedback about this post, or you would like to see more related content, please reach out to me here, or on Twitter or LinkedIn.
Feel free to checkout this video that goes over the mentioned setup: https://youtu.be/Eeo7GLVr0jw
Top comments (0)