DEV Community

Cover image for Handle optional input parameters in AWS Step Functions without Lambda
Pierre Milliotte for Serverless By Theodo

Posted on

Handle optional input parameters in AWS Step Functions without Lambda

TL;DR

If you want to apply data processing to an input object in your state machine without the well known struggle of dealing with optional parameters, you can use the HandleOptionalParameters CDK construct as follows:

const transformInputValuesTo1ItemArraysTask = new HandleOptionalParameters(
  scope,
  "Handle optional parameters",
  {
      requiredProperties: ["requiredProperty"],
      optionalProperties: ["optionalProperty1", "optionalProperty2"],
      // Apply States.Array intrinsinc function on every input values:
      dataProcessing: "States.Array($.value)",
      // Other examples:

      // Encode every input values and return the object with same input keys but with encoded values:
      // dataProcessing: "States.Base64Encode($.value)",

      // Concatenate the value with the property name, with the JsonPath $.propertyName:
      // dataProcessing: "States.Format('Current key is {} and associated value is {}.', $.propertyName, $.value)",
  }
);

new StateMachine(
  scope,
  "State machine using HandleOptionalParameters construct",
  {
      definition: processObjectValuesTask,
  }
);
Enter fullscreen mode Exit fullscreen mode

πŸ”΄ Context: AWS Step Functions throws if the provided JsonPath is not present

AWS Step Functions lets you do data processing on a state machine execution input thanks to its intrinsic functions.

For example, let's say you need to put every values of an object input in an array: Transform input

To transform a value to an 1-item array containing this value with Amazon State Language, you can use the States.Array() intrinsic function:

Transform one item object

To transform every values of an input object, you can apply States.Array() on every properties:

Transform multi items object

However, when dealing with object inputs, you have to make sure that each property you want to process is present in the input, otherwise the execution will fail ❌:

An error occurred while executing the state 'Transform input values to array' (entered at the event id #2). The function 'States.Array($.optionalProperty)' had the following error: The JsonPath argument for the field '$.optionalProperty' could not be found in the input { "requiredProperty": 1 }

Indeed, if foo is actually undefined:

Failure if some properties are optional

You thus need to handle these optional input parameters.

🟠 Choices: the false good idea for large inputs

Choice is another common flow state of AWS Step Functions that enables you to decide on which branch to continue the state machine execution, based on a condition evaluated at run time. You can thus use a Choice state to evaluate the optional property's presence in the input, and apply the States.Array() function only if the property is defined in the input:

Avoid failure with choices

However, if the input has many optional values, your step function may look like this πŸ˜“:

Choices solution is not scalable

This causes scalability issues:

  • πŸ§‘β€πŸ’» DX: as you can see in the execution graph view, debugging is painful in the AWS console
  • πŸ›‘ AWS limitations: in my case, Cloudformation template hard size limit of 1MB is reached with 10 optional properties
  • πŸ’Έ Cost: in a standard workflow, billing depends on the number of state transition which is proportional to the number of optional properties
  • πŸš€ Performance: Choices are not parallelised which leads speed to decrease with the number of optional properties

You thus need to handle the scalability of your optional parameters.

🟒 Let the HandleOptionalParameters CDK construct handle the optional parameters for you

The HandleOptionalParameters CDK construct processes every defined values with whatever data processing you provide, while ignoring the undefined ones. And it does it in 9 common flow tasks, regardless of the number of optional parameters πŸŽ‰:

Handle optional parameters the scalable way

Let's break it down:

CDK construct explained

Conclusion

From now on, stop using Lambda for handling large state machine inputs with many optional parameters ! πŸ˜‰

Top comments (0)