Scheduling and delaying queue messages

This article mostly talks about different approaches to scheduling and delaying queue messages and their pros and cons with examples and code.

Recently I have been using Amazon SQS as a queue to produce and consume messages as follows:

Now everything seems to be perfect until when we need to delay messages dynamically on a specific queue.

Now certainly when you end up delaying most of the messages inside all queues, apparently using a queue in the first place is not the ideal technique in this situation.

But sometimes in some odd circumstances, we need to delay some specific messages and hide them from the consumer.

Scenario

Suppose you have an ordering and logistics system.
You order a burger from your favorite restaurant. The restaurant accepts the order. After accepting the order, a driver should get assigned to deliver the order to the customer.

Now let's imagine this process happening over microservices that are listening to respective queues.

What happens is that the restaurant acts as a producer of the message when accepting an order. A logistics microservice that is responsible for assigning orders will consume the produced message immediately.

What's wrong with that?

What if the restaurant takes 30 minutes to prepare the burger. In such a case, assigning a driver to this order will be a bad decision because the driver will be wrongly utilized by going to the restaurant immediately and waiting for the burger to be prepared.

Therefore in this case we want to delay a message for x mins until it reaches y time and then execute the operation.

Different Approaches to apply this delay

Unfortunately, AWS SQS allows to delay messages up to 15 mins. But what if we need to delay it a bit more by x mins dynamically.

1. Simplest solution that might come to our minds, is just to set an in-memory timeout function. If using javascript, then setTimeout. In the producer, we calculate how much we need to delay a specific message and we put it as the interval of the setTimeout function.

Cons:

• The function inside the timeout might be blocked by other operations that need to be executed.
• If the server restarts for any reason, the delay and timeout will be flushed. For example, pushing a piece of code might cause the server to restart and boot again.

2. Another solution might be sending a timestamp in the producer in which when the message should get consumed.
In the consumer, we check if the timestamp reached what we need. If yes, consume the message. If not push the message back to the queue and don't consume it.

• Again this is a bad practice because there will be a loop in the consumer, and some messages will be pushed back to the queue multiple times.

Step functions and state machines.

Now, what if we forward the messages to a state machine instead of the queue immediately. state machines have waiting states and we can specify either the timestamp or duration of how much a message should wait in a state machine.

A simple state machine can achieve the following:

• Get a message from the producer.
• Wait for x duration.
• forward messages to specified queues.

Amazon has a tool called step functions where we can create state machines.

1. Create a State machine in amazon step functions.

2. Add a wait state and specify the waiting time by adding a fixed interval or dynamic one.

3. Step functions allow to trigger other AWS services. In this case SQS, so we add another state that defines how to trigger the SQS by providing the queue URL.

After this step, we will have a working State machine that gets messages, keeping them in a waiting state then forwards them to SQS.

A JSON definition of this state machine would look something like the following:

{
  "Comment": "A description of my state machine",
  "StartAt": "Wait",
  "States": {
    "Wait": {
      "Type": "Wait",
      "Next": "SQS SendMessage",
      "Seconds": 0
    },
    "SQS SendMessage": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage",
      "Parameters": {
        "MessageBody.$": "$"
      },
      "End": true
    }
  }
}

The interval can be put dynamic by changing the Seconds param in the waiting state to SecondsPath. Now we can pass it a dynamic variable from the message we send.

"Wait": {
      "Type": "Wait",
      "SecondsPath": "$.expiryInterval", // dynamic variable we pass in the message
      "Next": "SQS SendMessage"
    },

Triggering Step functions from node.js application.

const express = require('express');
const AWS = require('aws-sdk');


const app = express();

app.listen(3000, () => {
    console.log('first server is running on port 3000');
})

AWS.config.update({
    region: 'eu-central-1',
    accessKeyId: process.env.ACCESS_KEY_ID,
    secretAccessKey: process.env.SECRET_ACCESS_KEY,
});

const params = {
    stateMachineArn: process.env.STATE_MACHINE_ARN,
    input: JSON.stringify({ input: 'evan testing', expiryInterval: 10 })
};

const stepfunctions = new AWS.StepFunctions()


stepfunctions.startExecution(params, (err, data) => {
    if (err) {
        console.log(err, err.stack);
    } else {
        console.log(data);
    }
}
);

Conclusion

Queues are not meant to schedule events and actions in the first place. But sometimes such behavior is needed. The easiest and most efficient way that I found is using amazon state machines.