juan jose orjuela for AWS Community Builders

Posted on Dec 18, 2023

AWS Step Functions: Example HTTP Request Call"

#aws #programming #stepfunctions #tutorial

In the dynamic world of cloud computing, integration and automation are keys to success. AWS Step Functions stands as a powerful orchestrator of microservices, enabling developers to weave complex sequences of tasks with ease and efficiency. Among its crown jewels, HTTP Tasks shine on their own, offering unmatched versatility and integration capabilities.

What are HTTP Tasks? Essentially, they are states within an AWS Step Functions state machine designed to interact with third-party APIs. Think of them as bridges connecting your workflow in AWS with the vast universe of available web services. With HTTP Tasks, you can call external APIs like Salesforce, Stripe, or even more customized services, seamlessly integrating into your workflows.

Features:

Connection Flexibility: Whether you need to fetch data, send information, or initiate processes in external systems, HTTP Tasks allow you to do so with just a few clicks.
Robust Security: The integration of AWS IAM and EventBridge ensures that your API calls are secure and managed with AWS's best practices in terms of authentication and authorization.
Advanced Customization: From specifying HTTP methods (GET, POST, and more) to adjusting headers and parameters, you have total control over how you interact with external APIs.
Monitoring and Scalability: Benefit from integrated monitoring and the ability to scale automatically to handle workloads of any size.

This article will guide you through a fascinating journey into the world of HTTP Tasks in AWS Step Functions. We explore realistic use cases, crucial aspects of implementation, and conclude with a practical example: notifying a Slack webhook about the status of a process. Get ready to unlock new possibilities and take your cloud workflows to the next level.

Use Cases:

HTTP Tasks in AWS Step Functions open up a range of possibilities for service integration and process automation. This section explores various practical and powerful use cases where HTTP Tasks can be a transformative tool.

Integration with CRM and ERP Systems

Customer and Order Management: Automate data synchronization between your AWS system and CRM platforms like Salesforce. For instance, you can automatically update customer records in Salesforce when changes occur in your AWS system.
Business Process Automation: Connect your workflows with ERP systems to manage inventory, orders, and logistics, enabling real-time integration that optimizes business operations.

Interactions with Payment Platforms

Payment Processing: Integrate services like Stripe or PayPal to automate payment processing. HTTP Tasks can send transaction data and receive payment confirmations, streamlining the sales cycle.
Subscription Management: Automate the creation and management of subscriptions, as well as updating customer details in online payment systems.

Notifications and Alerts

Sending Notifications: Use services like Twilio or SendGrid to send SMS or emails as part of a workflow, for example, for transaction alerts or order status notifications.
Integration with Slack: Send automatic messages to Slack channels or users to notify about project updates, system alerts, or data summaries.

Data Analysis and Reporting

Integration with Analysis Tools: Extract data from AWS for analysis in external platforms like Google Analytics or BI tools, allowing for deeper understanding and data-driven action.
Automated Reporting: Generate and distribute automatic reports using data processed by your AWS workflow, sending them to reporting tools or storing them in file systems like Google Drive.

Artificial Intelligence and Machine Learning Services

Interaction with AI/ML Models: Integrate external AI services like IBM Watson or Google AI to enrich your data with machine learning capabilities, sentiment analysis, or image recognition.
Machine Learning Process Automation: Orchestrate and manage ML workflows, from data preparation to training and evaluating models, using various ML APIs.

Cloud Resource Management

Orchestration of Cloud Services: Automate the creation, update, and deletion of cloud resources, interacting with services like AWS EC2 or Google Cloud Platform.
Monitoring and Event Response: Automatically respond to cloud system events, such as launching EC2 instances or performance alerts, using workflows that include calls to relevant APIs.

Integration of Social Media and Marketing

Automated Social Media Posts: Schedule and publish content on social media platforms like Twitter or Facebook, as part of digital marketing strategies.

Basic Example:

To illustrate how an HTTP Task is implemented in AWS Step Functions, let's consider a concrete example. Imagine that we need to make a GET call to an external API to fetch information. Below is the code for the HTTP Task and an explanation of each property involved.

{
  "Comment": "Ejemplo de Tarea HTTP en AWS Step Functions",
  "StartAt": "CallExternalAPI",
  "States": {
    "CallExternalAPI": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "ApiEndpoint": "https://api.externalservice.com/data",
        "Method": "GET",
        "Headers": {
          "Content-Type": "application/json"
        },
  "Authentication": {
          "ConnectionArn": "arn:aws:events:region:account-id:event-bus/default"
        }

      },
      "End": true
    }
  }
}

Properties

Comment: A description or comment about the purpose of the state machine. Useful for documentation and clarity.
StartAt: Indicates the first state that will be executed in the state machine. In this case, it's CallExternalAPI.
States: Defines the different states of the machine. Here we only have one state, CallExternalAPI.
- CallExternalAPI: The name of the state we are defining.
- Type: Defines the type of state. In this case, it's a Task, which means it performs a specific task.
- Resource: Specifies the type of resource that the task will use. Here arn:aws:states:::http:invoke is used to indicate that it is an HTTP Task.
- Parameters: Defines the necessary parameters for the HTTP task.
  - ApiEndpoint: The URL of the external API to which the call will be made.
  - Method: The HTTP method to be used. Here it is GET.
  - Headers: Necessary HTTP headers for the request. In this case, it indicates that the content is of JSON type.
- Authentication: This field is used to specify authentication details for the API call.
  - ConnectionArn: The Amazon Resource Name (ARN) of the EventBridge connection that manages authentication credentials. This connection must be pre-configured in EventBridge and contain the necessary information to authenticate with the API (e.g., an API token).
End: Indicates if this is the last state of the state machine. If true, the state machine will end after executing this state.

Practical Example: Simple Notification to Slack via Webhook

In today's digital era, instant communication and notification are essential for efficiency and team collaboration. Slack, one of the leading platforms in business communication, offers robust functionality to integrate personalized notifications through webhooks. Combining this with AWS Step Functions allows us to automate notifications and keep teams informed about key events and processes in real time.

In this section, we will tackle a practical and highly relevant example: sending a simple notification to a Slack channel using a webhook. This task, seemingly straightforward, encapsulates fundamental concepts of integration and automation in the cloud, demonstrating how modern tools can work together efficiently.

Steps for Creating a Webhook in Slack

Create a Channel for Receiving Notifications: Start by creating a channel where you will receive the notifications. In my case, I named it slack-notification-test.

Create a Workflow in Integrations: Next, go to the integrations section and create a workflow. In this example, we'll name it bot-auto-not.

Workflow Creation with Webhook Trigger: In the workflow creation process, choose to start it via a webhook. Following this, you will be asked to define the variables that will be received when consuming the webhook. In this example, the variables are type and text.

Adding an Additional Step for Message Sending: After setting up the webhook, create an additional step to send a message to the channel using the information from the defined variables.

Finding the Webhook URL: In the webhook step, you will find the URL that will be used to send notifications. It will be in this format: https://hooks.slack.com/triggers/11111111/222222222/333333333.
Testing the Webhook: You can test the webhook by making a POST request and sending the parameters defined in the workflow in the body of the request.

curl --location 'https://hooks.slack.com/triggers/1111/22222/33333 \
--header 'Content-type: application/json' \
--data '{
  "text": "Example text",
  "type": "ERROR"
}'

Starting with the Step Function Implementation: With all these components defined, we can now begin with the implementation of the Step Function.

Steps for Creating the Step Function via Terraform:

Create a Role and Policies for the Step Function: The first step is to create a role and policies that will be assumed by the Step Function, enabling it to make HTTP calls.

Role Resource:

resource "aws_iam_role" "step_functions_role" {
  name = "step_functions_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "states.amazonaws.com"
        }
      }
    ]
  })
}

Policies Resource for role:

resource "aws_iam_policy" "sfn_logging_policy" {
  name        = "SFNLoggingPolicy"
  description = "Allow Step Functions to log to CloudWatch Logs."

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "logs:*"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_policy" "sf_invoke_requests_policy" {
  name        = "SFSendInvokeRequestsPolicy"
  description = "Allow Step Functions to invoke requests."

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "states:InvokeHTTPEndpoint"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "events:RetrieveConnectionCredentials"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret"
        ],
        Resource = "*"
      }
    ]
  })
}

There are two key policies to set up:

Log Management Policy: This policy is for the management of logs.
HTTP Invocation and Secrets Access Policy: The second policy grants the Step Function the ability to invoke HTTP services and read secrets, which is essential in cases where the requests require authentication.

Creation of the EventBridge Connection: AWS EventBridge is a serverless event bus service that facilitates the connection of applications with data from different sources. A key feature is the ability to establish "connections" that allow AWS Step Functions to securely communicate with third-party APIs, such as Slack, Salesforce, Stripe, among others. These connections are used, for example, in HTTP Tasks to manage authentications and authorizations securely and efficiently. How it works:
- Creating a Connection: In EventBridge, you define a "connection" that encapsulates the authentication details necessary to communicate with an external API. This connection can include information such as access tokens, API keys, or basic authentication credentials.
- Secure Credential Storage: EventBridge securely stores credentials using AWS Secrets Manager. This means that the credentials are encrypted and protected, and are not exposed in the task definitions or in the code.
- Integration with Step Functions: When defining an HTTP Task in AWS Step Functions, you specify the ARN (Amazon Resource Name) of the EventBridge connection. Step Functions uses this connection to authenticate with the external API during the task execution.
- Authorization Management: The connection handles the authorization process with the external API. This frees developers from the burden of implementing and maintaining custom code for managing tokens or API keys.

Event bridge connection Resource:

resource "aws_cloudwatch_event_connection" "slack_test_webhook" {
  name               = "test-slack-notification-connection"
  description        = "test-slack-notification-connection"
  authorization_type = "API_KEY"

  auth_parameters {
    api_key {
      key   = "not-key"
      value = "None"
    }
  }
}

Necessity of the Connection Even Without Authentication for the Webhook: This connection is required even though the webhook does not handle authentication. For this reason, we input a random API key.

Aws Step function definition:

{
  "Comment": "Send a slack notification via sf",
  "StartAt": "FirstStep",
  "States": {
    "FirstStep": {
      "Type": "Pass",
      "Result": {
        "type": "INFO",
        "text": "Message from Step Functions"
      },
      "ResultPath": "$.notificationData",
      "Next": "SendNotification"
    },
    "SendNotification": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Parameters": {
        "Authentication": {
          "ConnectionArn": "${aws_cloudwatch_event_connection.slack_test_webhook.arn}"
        },
        "RequestBody": {
          "text.$": "$.notificationData.text",
          "type.$": "$.notificationData.type"
        },
        "ApiEndpoint": "https://hooks.slack.com/triggers/11111111/2222222222/33333333333",
        "Method": "POST"
      },
      "Next": "LastStep"
    },
    "LastStep": {
      "Type": "Succeed"
    }
  }
}
EOF
}

Three States in the Step Function:

FirstStep: This state generates the data that will be sent in the notification. The data is sent through the notificationData object.
SendNotification: This state is responsible for sending the notification. Notice that

After going through the process of setting up a simple notification to Slack via a webhook using AWS Step Functions, it's clear that we are dealing with a powerful combination of tools that can revolutionize the way we interact with workflows and team communication.

If you want to see the complete example to run it in Terraform, here's the repository for you to check out.

If you liked this article, don't hesitate to give it a 👏 and a ⭐ on the repository.

Thank you!!

DEV Community