AWS IoT Greengrass Components Deployment with GDK using GitHub Actions

As IoT deployments grow, a set of distinct challenges arise when it comes application consistency across a broad spectrum of devices: efficiently managing update rollbacks, and maintaining comprehensive, auditable change logs can all present significant obstacles. However, these challenges can be effectively managed using AWS IoT Greengrass and the Greengrass Development Kit (GDK).

AWS IoT Greengrass enables the deployment of applications directly to your IoT devices, simplifying the management and control of device-side software. The GDK further enhances this capability by streamlining the configuration and packaging of these applications for deployment, making it easier to get your applications onto your devices.

To create a robust and efficient system, we can also introduce GitOps methodologies. GitOps leverages version control, automated deployments, and continuous monitoring to improve the reliability and efficiency of your deployment processes. By using these methodologies with GitHub Actions, we can automate the deployment process, triggering it with every commit or merge to a specified branch.

In this blog post, we'll explore how these technologies and methodologies can be combined to create a powerful, scalable, and automated IoT deployment system. We'll walk through the setup of the GDK, the definition and deployment of Greengrass components, and the setup of a GitHub Actions workflow to automate the entire process.

Let’s dive into and get started with this setup.

Overview

The AWS IoT Greengrass Development Kit Command-Line Interface (GDK CLI) is an open-source tool designed to streamline the creation, building, and publishing of custom Greengrass components. It simplifies the version management process, allows starting projects from templates or community components, and can be customized to meet specific development needs.

It can also be effectively utilized with GitHub Actions to automate the process of building and publishing Greengrass components. This could be a crucial part of a Continuous Integration/Continuous Deployment (CI/CD) pipeline. Here's a general outline of how this could be done:

• Install Dependencies: Create a GitHub Actions workflow file (e.g., .github/workflows/main.yml) and start with a job that sets up the necessary environment. This includes installing Python and pip (since GDK CLI is a Python tool), AWS CLI, and the GDK CLI itself.
• Configure AWS Credentials: Use GitHub Secrets to securely store your AWS credentials (Access Key ID and Secret Access Key). In your workflow, configure AWS CLI with these credentials so that the GDK CLI can interact with your AWS account.
• Build and Publish Components: Use GDK CLI commands in your workflow to build and publish your components. For example, you might have steps that run commands like gdk component build and gdk component publish.
• Integrate with Other Workflows: If you have other workflows in your CI/CD pipeline (such as running tests or deploying to other environments), you can use the output of the GDK CLI commands as inputs to these workflows.

In this way, every time you push a change to your Greengrass component source code on GitHub, the GDK CLI can automatically build and publish the updated component, ensuring that your Greengrass deployments are always using the latest version of your components.

Furthermore, we can initiate a Greengrass deployment based on the deployment template in the next stage of the pipeline, after a successful build and publish. This can target a specific Thing Group, enabling us to reflect changes across a fleet of devices. With this approach, if we have dev and test branches, each of these can be mapped to selected Thing Groups. This allows us to perform field validation on selected devices, providing an efficient way to test changes in a controlled environment before wider deployment.

For further details on creating GitHub Actions workflows, refer to the GitHub Actions documentation. For more information about using the GDK CLI, please refer to the GDK CLI documentation.

Development Project Setup

Based on the high level overview we can structure our project as such:

project
├── .github
│   └── workflows
│       └── main.yml
├── cfn
│   └── github-oidc
│       ├── oidc-provider.yaml
│       └── oidc-role.yaml
└── components
    ├── com.example.hello
    │   ├── gdk-config.json
    │   ├── main.py
    │   └── recipe.yaml
    ├── com.example.world
    │   ├── gdk-config.json
    │   ├── main.py
    │   └── recipe.yaml
    └── deployment.json.template

Where we have the following:

• .github/workflows/main.yml: Which is the GitHub Actions workflow file where the CI/CD pipeline is defined. The GDK CLI and AWS CLI setup, component building, publishing, and deployment tasks are defined here.
• cfn/github-oidc: This directory contains AWS CloudFormation templates (oidc-provider.yaml and oidc-role.yaml) that are used to set up an OIDC provider and role on AWS for authenticating GitHub Actions with AWS.
• components: This directory contains the Greengrass components (com.example.hello and com.example.world) that you are developing. Each component has its own directory with:
  • gdk-config.json: This is the configuration file for the GDK CLI for the specific component.
  • main.py: This is the main Python script file for the component's functionality.
  • recipe.yaml: This is the component recipe that describes the component and its dependencies, lifecycle scripts, etc.
• deployment.json.template: This is a deployment template file for Greengrass deployments. It is used to generate the actual deployment file (deployment.json) that is used when initiating a Greengrass deployment

GitHub Actions and GDK Deployment Role Setup

The CloudFormation template will be used to create an IAM role (oidc-gdk-deployment) that provides the necessary permissions for building and deploying Greengrass components using GDK CLI and GitHub Actions. The role has specific policies attached that allow actions such as describing and creating IoT jobs, interacting with an S3 bucket for Greengrass component artifacts, and creating Greengrass components and deployments.

AWSTemplateFormatVersion: 2010-09-09
Description: 'GitHub OIDC:| Stack: oidc'

Parameters:
  FullRepoName:
    Type: String
    Default: example/gdk-example

Resources:
  Role:
    Type: AWS::IAM::Role
    Properties:
      RoleName: oidc-gdk-deployment
      Policies:
        - PolicyName: iot-thing-group
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - iot:DescribeThingGroup
                  - iot:CreateJob
                Resource:
                  - !Sub arn:aws:iot:${AWS::Region}:${AWS::AccountId}:thinggroup/*
        - PolicyName: iot-jobs
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - iot:DescribeJob
                  - iot:CreateJob
                  - iot:CancelJob
                Resource:
                  - !Sub arn:aws:iot:${AWS::Region}:${AWS::AccountId}:job/*
        - PolicyName: s3-greengrass-bucket
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:CreateBucket
                  - s3:GetBucketLocation
                  - s3:ListBucket
                Resource:
                  - !Sub arn:aws:s3:::greengrass-component-artifacts-${AWS::Region}-${AWS::AccountId}
        - PolicyName: s3-greengrass-components
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:PutObject
                Resource:
                  - !Sub arn:aws:s3:::greengrass-component-artifacts-${AWS::Region}-${AWS::AccountId}/*
        - PolicyName: greengrass-components
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - greengrass:CreateComponentVersion
                  - greengrass:ListComponentVersions
                Resource:
                  - !Sub arn:aws:greengrass:${AWS::Region}:${AWS::AccountId}:components:*
        - PolicyName: greengrass-deployment
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - greengrass:CreateDeployment
                Resource:
                  - !Sub arn:aws:greengrass:${AWS::Region}:${AWS::AccountId}:deployments
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Sub arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com
            Condition:
              StringLike:
                token.actions.githubusercontent.com:sub: !Sub repo:${FullRepoName}:*

Outputs:
  OidcRoleAwsAccountId:
    Value: !Ref AWS::AccountId
  OidcRoleAwsRegion:
    Value: !Ref AWS::Region
  OidcRoleAwsRoleToAssume:
    Value: !GetAtt Role.Arn

The FullRepoName parameter is used to specify the repository that the GitHub Actions workflow will be running in. This is important for the sts:AssumeRoleWithWebIdentity action in the AssumeRolePolicyDocument, which allows GitHub Actions to assume this IAM role for the specified repository.

To deploy this CloudFormation stack, you would use the AWS Management Console, AWS CLI, or an AWS SDK. You would need to specify the FullRepoName parameter as an input when you create the stack. For example, with the AWS CLI, you would use the aws cloudformation deply command and provide the template file and the FullRepoName parameter as inputs:

aws cloudformation deploy \
    --template-file cfn/github-oidc/oidc-role.yaml \
    --stack-name ga-gdk-role \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides FullRepoName=<your org>/<your repo name>

This setup lays the foundation for the GitHub Actions and GDK CLI to work together to automate the building and deployment of Greengrass components.

In scenario that you would like to use OIDC Provider for GitHub actions (suggested) you would also need to set it up in you AWS account. Please not that this is needed only once per account region:

aws cloudformation deploy \
    --template-file cfn/github-oidc/oidc-provider.yaml \
    --stack-name oidc-provider

Now we can go and prepare our Greengrass components.

Greengrass Components GDK Setup

Here we focus here on the configuration and implementation of our Greengrass components. These components form the core of our IoT solution, providing the required functionality on our Greengrass core devices.

The gdk-config.json file is where we configure our Greengrass component.

{
    "component": {
      "com.example.hello": {
        "author": "Example",
        "version": "NEXT_PATCH",
        "build": {
          "build_system": "zip"
        },
        "publish": {
          "bucket": "greengrass-component-artifacts",
          "region": "eu-west-1"
        }
      }
    },
    "gdk_version": "1.2.0"
}

In the provided example, we have a single component named "com.example.hello". This file specifies the author, the build system (which is set to "zip" here), and the AWS S3 bucket details where the component will be published. The version field is set to "NEXT_PATCH", which means GDK will automatically increment the patch version of the component every time it's built.

The recipe.yaml file is the recipe for our Greengrass component.

---
RecipeFormatVersion: "2020-01-25"
ComponentName: "{COMPONENT_NAME}"
ComponentVersion: "{COMPONENT_VERSION}"
ComponentDescription: "This is a simple Hello World component written in Python."
ComponentPublisher: "{COMPONENT_AUTHOR}"
ComponentConfiguration:
  DefaultConfiguration:
    Message: "Hello"
Manifests:
  - Platform:
      os: all
    Artifacts:
      - URI: "s3://BUCKET_NAME/COMPONENT_NAME/COMPONENT_VERSION/com.example.hello.zip"
        Unarchive: ZIP
    Lifecycle:
      Run: "python3 -u {artifacts:decompressedPath}/com.example.hello/main.py {configuration:/Message}"

It contains essential metadata about the component, like its name, version, description, and publisher. It also specifies the default configuration, which, in this case, sets the default message to "Hello". The Manifests section describes the component's artifacts and the lifecycle of the component. In this instance, it specifies the location of the component's zipped artifacts and the command to run the component.

Finally, the main.py file is the Python script that our Greengrass component runs.

import sys

message = f"Hello, {sys.argv[1]}!"

# Print the message to stdout, which Greengrass saves in a log file.
print(message)

This script simply prints out a greeting message. The message is constructed from the argument passed to the script, which comes from the component configuration in the recipe.yaml file. This setup demonstrates how you can pass configuration to your Greengrass components, as we can see in the “com.example.world” example where we provide “World” as a configuration message.

In addition to the component configuration and scripts, we also define a deployment.json.template file.

{
    "targetArn": "arn:aws:iot:$AWS_REGION:$AWS_ACCOUNT_ID:thinggroup/$THING_GROUP",
    "deploymentName": "Main deployment",
    "components": {
        "com.example.hello": {
            "componentVersion": "LATEST",
            "runWith": {}
        },
        "com.example.world": {
            "componentVersion": "LATEST",
            "runWith": {}
        }
    },
    "deploymentPolicies": {
        "failureHandlingPolicy": "ROLLBACK",
        "componentUpdatePolicy": {
            "timeoutInSeconds": 60,
            "action": "NOTIFY_COMPONENTS"
        },
        "configurationValidationPolicy": {
            "timeoutInSeconds": 60
        }
    },
    "iotJobConfiguration": {}
}

This file specifies the deployment configuration for our Greengrass components. The targetArn is the Amazon Resource Name (ARN) of the Greengrass thing group where we aim to deploy our components. In this example, we're deploying two components, com.example.hello and com.example.world, both set to use their latest versions. The deploymentPolicies section sets the policies for failure handling, component update, and configuration validation. This file is vital as it governs how the deployment of our Greengrass components is handled in the target IoT devices.

Please note that this is a template and we will be using this in our pipeline to replace the ARN accordingly.

Taken together, these files form the basis of a Greengrass component. By modifying these templates and scripts, you can create your own custom Greengrass components with GDK. The next step is to set up GitHub Actions to automate the build and deployment of these components.

GitHub Actions Setup

This GitHub workflow file sets up two jobs, namely publish and deploy, which are run when either a push to the main branch occurs or the workflow is manually dispatched.

1. Publish Job

The publish job runs on the ubuntu-latest environment.

jobs:
  publish:
    name: Component publish
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
        ref: ${{ github.head_ref }}
    - uses: actions/setup-python@v3
      with:
        python-version: '3.9'

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        role-to-assume: ${{ secrets.OIDC_ROLE_AWS_ROLE_TO_ASSUME }}
        aws-region: ${{ secrets.OIDC_ROLE_AWS_REGION }}

    - name: Install Greengrass Development Kit
      run: pip install -U git+https://github.com/aws-greengrass/aws-greengrass-gdk-cli.git@v1.2.3

    - name: GDK Build and Publish
      id: build_publish
      run: |

        CHANGED_COMPONENTS=$(git diff --name-only HEAD~1 HEAD | grep "^components/" | cut -d '/' -f 2)

        echo "Components changed -> $CHANGED_COMPONENTS"        

        for component in $CHANGED_COMPONENTS
        do
          cd $component
          echo "Building $component ..."
          gdk component build
          echo "Publishing $component ..."
          gdk component publish
          cd ..
        done

      working-directory: ${{ env.working-directory }}

We start with checking out the code from the repository and setting up Python 3.9. The AWS credentials are then configured using the aws-actions/configure-aws-credentials@v1 action. The credentials used here are fetched from the GitHub secrets OIDC_ROLE_AWS_ROLE_TO_ASSUME and OIDC_ROLE_AWS_REGION. The OIDC_ROLE_AWS_ROLE_TO_ASSUME secret should contain the ARN of the AWS role that the GitHub Actions should assume when executing the workflow. This is the role we created in the firs step we can obtain it by executing the following:

aws cloudformation describe-stacks --stack-name ga-gdk-role --query 'Stacks[0].Outputs[?OutputKey==`OidcRoleAwsRoleToAssume`].OutputValue' --output text

The OIDC_ROLE_AWS_REGION secret should contain the AWS region where your resources are located. After that these variables needs to be added under github.com/<org>/<repo>/settings/secrets/actions .

Next, the Greengrass Development Kit (GDK) is installed using pip. The GDK CLI is used to build and publish any components that have changed between the current and previous commit.

The changed components are identified by looking at the differences between the current and previous commit and extracting the component names. Here it is important that the name of the folder mathces the name of the component.

2. Deploy Job

The deploy job runs after the publish job has completed successfully.

  deploy:
    name: Component deploy
    runs-on: ubuntu-latest
    needs: publish
    permissions:
      id-token: write
      contents: read

    steps:
    - name: Checkout
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
        ref: ${{ github.head_ref }}

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        role-to-assume: ${{ secrets.OIDC_ROLE_AWS_ROLE_TO_ASSUME }}
        aws-region: ${{ secrets.OIDC_ROLE_AWS_REGION }}

    - name: Deploy Greengrass components
      run: |
        export AWS_ACCOUNT_ID=$(aws sts get-caller-identity |  jq -r '.Account')
        export AWS_REGION=${GREENGRASS_REGION}
        # Thing Group is the name of the branch
        export THING_GROUP=${GITHUB_REF#refs/heads/}

        CHANGED_COMPONENTS=$(git diff --name-only HEAD~1 HEAD | grep "^components/" | cut -d '/' -f 2)

        if [ -z "$CHANGED_COMPONENTS" ]; then
          echo "No need to update deployment"
        else
          envsubst < "deployment.json.template" > "deployment.json"

          for component in $CHANGED_COMPONENTS
          do
            version=$(aws greengrassv2 list-component-versions \
              --output text \
              --no-paginate \
              --arn arn:aws:greengrass:${AWS_REGION}:${AWS_ACCOUNT_ID}:components:${component} \
              --query 'componentVersions[0].componentVersion')

            jq '.components[$component].componentVersion = $version' --arg component $component --arg version $version deployment.json > "tmp" && mv "tmp" deployment.json

          done

          # deploy
          aws greengrassv2 create-deployment \
            --cli-input-json file://deployment.json \
            --region ${AWS_REGION}

          echo "Deployment finished!"
        fi

      working-directory: ${{ env.working-directory }}

It follows a similar structure to the publish job, starting with checking out the code from the repository and configuring AWS credentials using the same secrets.

In the deployment step, it first identifies any changed components in a similar way as in the publish job. If no components have changed, it does not proceed with deployment. If there are changed components, it prepares a deployment.json file from the template, replacing placeholders with the actual values. It then gets the version of the changed components from AWS Greengrass and updates the deployment.json file with these versions.

Finally, it creates a deployment using the aws greengrassv2 create-deployment command, providing the deployment.json file as input and setting the region to the one specified in the AWS_REGION environment variable.
Here it is important to note that Thing Group is taken from the name of the branch THING_GROUP=${GITHUB_REF#refs/heads/} as that way we can have different branches related to different Thing Groups as discussed above. In case the thing group is not create you can use the following command:

aws iot create-thing-group --thing-group-name main

At this point, when ever there is a new commit on the main branch a process will kick start and issue a deployment to the specified group of devices.

Conclusion

In this blog post, we've taken a deep dive into AWS IoT Greengrass V2, focusing on the usage of the Greengrass Development Kit (GDK) and how it can be automated with GitHub Actions. We started by setting up the GDK, explored the various components and how they interact, then we moved on to setting up a GitHub Actions workflow to automate the entire process.

By leveraging AWS IoT Greengrass, the GDK, and GitHub Actions, you can create a powerful, scalable, and automated IoT solution. Whether you're managing a small group of IoT devices or a large fleet, this approach offers a robust and efficient way to handle your IoT application deployments.

That's all for this blog post. We hope you found it informative and that it helps you on your journey to creating and managing IoT solutions with AWS IoT Greengrass. Happy coding!

All the above code and setup can be referenced here:
https://github.com/aws-iot-builder-tools/greengrass-continuous-deployments

If you have any feedback about this post, or you would like to see more related content, please reach out to me here, or on Twitter or LinkedIn.