AWS Step Functions with Callback Pattern

This blog describes the Step Function callback pattern with a simple use case. It also covers Step function and Lambda integration with API Gateway and DynamoDB CRUD operations.

In a callback pattern, Step Functions pauses execution of the workflow until your application returns a token through the Step Functions application programming interface. This helps you automate workflows for applications that need human activities such as review and approval of documents or creating and managing work items in a customer support application.

Use Case:

A user places an order with a list of order items and associated costs. If the overall cost is more than 100 then the order needs external approval. The order will be processed/rejected based on the external approval status.

Workflow:

1. A user places an order using a RESTful API exposed through API Gateway. The API Gateway in turn invokes a step function to process the order.

2. An order can be either auto-approved or requires external approval. Step function Callback is used in case of external approval. If an order requires additional approval then the step function will be paused until the order is approved/rejected through a separate RESTful API.

3. To keep the order on hold step function “waitForTaskToken” is used in the required step and saves the token in the DynamoDB.

4. A separate API (a Lambda function) will be exposed through API Gateway which accepts order ID and state (either Approved or Rejected). The lambda function retrieves step function Callback token from the Dynamo DB using order ID.

5. The Lambda function invokes the step function using the Callback Token.

6. The Step function resumes execution.

You can get the code from below Git hub repo.

srikarG81/AWS-Step-Functions-with-Callback-Pattern

Code walkthrough:

The state machine is defined in the state-machine.json file. This file contains the syntax to define a step and corresponding lambda function to handle the logic for the step.

“https://github.com/srikarG81/AWS-Step-Functions-with-Callback-Pattern/blob/main/OrderOrchestration/state-machine.json”

Example: Below code snippet defines a step “Validate Order” and logic for the step is implemented in the Lambda function referred in the “Resource” attribute.

Validate Order lambda function:It validates the order and saves the order information in the DB.

File: “https://github.com/srikarG81/AWS-Step-Functions-with-Callback-Pattern/blob/main/OrderOrchestration/StepFunctionTasks.cs”

Below code snippet defines another step “WaitForApproval” and its corresponding Lambda function is specified in the “Resource” attribute.
“waitForTaskToken” will instruct step-function to generate Token for this step and the step function will be on hold/paused after executing the associated Lambda function.

Step definition in the “state-machine.json” file and corresponding lambda:

The payload in the above step represents the input to the associated Lambda function {WaitForApprovalTask}. The function saves the token & Order ID in the Dynamo DB order table.

The waiting step will resume execution only when external services invoke the function as follows. In the below code snippet “amazonStepFunctionsClient” invokes the step function with the callback token. The callback token is retrived from the DB using the order ID.

Code Deployment :

Create DynamoDB table with following AWS CLI command: Command creates a table with two attributes/{like columns in RDBMS world}.

The primary key for Order consists of two attributes (OrderID and Cost). Each item in the table must have these two attributes. The combination of Order ID and Cost distinguishes each item in the table from all of the others.

Other than the primary key, the Order table is schema less, which means that neither the attributes nor their data types need to be defined beforehand. Each item can have its own distinct attributes.

aws dynamodb create-table \
    --table-name Order \
    --attribute-definitions AttributeName=OrderID,AttributeType=S AttributeName=Cost,AttributeType=N \
   --key-schema AttributeName=OrderID,KeyType=HASH AttributeName=Cost,KeyType=RANGE \
    --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Deploy AWS stepfunction and Lambda using the AWS plugin in Visual studio as follows.

Integrate the above step function in API Gateway:

1. Navigate to API Gateway in AWS console.
2. Select Create API -> REST API-> Build.
3. In Create new API screen, Enter API name, Endpoint Type field values.
4. Create a new post method and integrate the step function as follows. You can refer the AWS documentation like for additional details (https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-api-gateway.html)

Demo:

Post an order request using PostMan as follows:

Post the order request to step function using API gateway post method url. As the cost of the order is more than 100, The order will be on hold as shown in the following step function screenshot.

Order process step function state for the above request.

The order is on hold as it needs approval from external service. The approval process is simulated by another API calls as follows.

Step Function state after extenal approval:

Conclusion:

AWS Step function callback patterns can be useful to automate workflows for applications with human activities and custom integrations with third-party services.