Neil Killen

In my previous post, I looked at message queues and how they can be utilized in your solution architecture. In the second part of this series, I will explore how you can add messages to a Message queue (Message Publisher) and Consume messages (Consumer) from a message queue with Sitecore and AWS SQS.

Using the example of an Events Booking site where a user finds an event they are interested in complete an event registration form which gets submitted to Event Bookings Application via an API. The Booking Application is known to go down and it is not uncommon to run into connection issue with the API causing booking requests to get lost. The Booking Application is old and unreliable and occasionally goes down.

By adding a Message Queue we decouple the website from the booking application and provide a level of resilience. If an outage does occur or there is an API connection issue instead of messages being lost they will be persisted on the queue. When the booking application comes back online or connection has been restored the messages on the queue will get processed.

Create a Message Queue in AWS SQS

Creating a standard message queue in AWS SQS is relatively straightforward.

1. Once logged into AWS locate Simple Queue Service within Application Integration category of services. All your queues will be listed here and you can monitor, configure an existing queue or create new queues.
2. To add a new queue Click Create Queue.
3. Enter the name of the Queue and Select Standard Queue then Click Configure Queue.
4. You will be presented with the configuration settings for the queue or you could just simply hit Quick Create Queue to create the queue with the default configuration and configure later if required.
   • Default Visibility Timeout – is the length of time (in seconds) that a message received from a queue will be invisible to other receiving components. (between 0 seconds and 12 hours) Default: 30 seconds.
   • Message Retention Period – is the amount of time SQS will retain a message if it does not get deleted (between 1 minute and 14 days) Default: 4 days.
   • Maximum Message Size – is the size limit of a message before it is rejected (between 1KB and 256KB) Default: 256KB.
   • Delivery Delay – is the amount of time to delay the first delivery of all messages added to this queue (between 0 seconds and 15 minutes) Default: 0 seconds.
   • Receive Message Wait Time – is the maximum amount of time that a long polling receive call will wait for a message to become available before returning an empty response (between 0 and 20 seconds) Default: 0 seconds.
   • Use Redrive Policy – send messages into a dead letter queue after exceeding the Maximum Receives, Default: disabled. I’ll go into more detail on this in the next post.
   • Dead Letter Queue – The name of the existing queue that will serve as the dead letter queue.
   • Maximum Receives – The maximum number of times a message can be received before it is sent to the Dead Letter Queue.
   • Use SSE – When this option is selected, SQS encrypts all messages sent to this queue, Default: disabled.
   • AWS KMS Customer Master Key (CMK) – SQS uses a KMS CMK to generate data keys for encryption and decryption of multiple SQS messages during the data key reuse period.
   • Data Key Reuse Period – The time period during which Amazon SQS can reuse a data key to encrypt or decrypt messages before calling KMS again (between 1 minute and 24 hours).
5. Once your Queue is created you can easily access and view information about the queue. The following details you will need to be able to connect to your queue: 
   • URL: Amazon SQS assigns each queue you create an identifier called a queue URL that includes the queue name and other Amazon SQS components like your AWS account number. Whenever you want to perform an action on a queue, you provide its queue URL.
   • ARN: the ARN (Amazon Resource Name) uniquely identify AWS resources. ARns are used when you need to specify a resource unambiguously across all of AWS, such as in IAM policies, and API calls.

SQS Security Credentials

When you first create an AWS account, you begin with a single sign-in identity known as AWS account root user. This account has complete access to all AWS services and resources available to you’re account. Instead of sharing your root user credentials with others, it is recommended you can create an individual IAM user or a group of users with access to the SQS service by assigning the appropriate security policy i.e. AmazonSQSFullAccess policy and configure access keys which can be used for connecting to your AWS SQS service.

Sending a Message to an AWS SQS Queue

Once your queue and security credentials for accessing the queue are configured sending a message to an AWS SQS queue is relatively straightforward, using the AWS .NET SDK you can do this with just a few lines of C#.

Consuming messages from an SQS Queue

What if we need to consume messages from an SQS Queue? In an ideal world, the booking application would periodically connect to the booking queue and retrieve the booking messages, however, what if the booking application was in the process of being replaced or we did not have access to the source code and we need to push messages to its exposed API from the message queue.

We could easily consume messages in a queue by creating a Sitecore scheduled task. The scheduled task would be configured to run periodically and poll the message queue and read any messages contained in the queue. For each message contained in the queue, we would attempt to process the message by forwarding it to the booking API. If the message is processed successfully we would then delete the message from the queue.  If the message is not processed successfully we leave the message on the queue and it will be processed on subsequent polls of the queue.

1. Sitecore Scheduled Task configured to periodically poll message queue.
2. Read messages contained in the queue.
3. Process each message by attempting to forward to the booking API.
4. On successful processing of message delete message from queue.
5. If the message is not processed successfully it remains on the queue.

A few things to note:

• When a consumer receives and processes a message from a queue, the message remains in the queue, SQS doesn’t automatically delete the message. The consumer must delete the message from the queue after receiving and processing it. To prevent other consumers from processing the message at the same time, SQS starts the visibility timeout, preventing other consumers from receiving and processing the message.
• The SQS ReceiveMessage request can return 1 to 10 messages in the response by configuring MaxNumberOfMessages request parameter default: 1.
• You can configure AWS SQS Long polling to help reduce costs by eliminating the number of empty responses – when there are no messages available for a ReceiveMessage request. By default SQS uses short polling, querying only a subset of its servers, to determine whether any messages are available for a response.
• Messages in the queue are not persisted indefinitely messages will be automatically removed from the queue after the configurable retention period of 1 to 14 days. You need to create a mechanism for handling failed messages.
• If you are not familiar with Sitecore Scheduled tasks Brian Pederson has written a couple of great posts I recommend checking them out:
  • Sitecore Scheduled Task – Schedule time format and other quirks
  • Run Sitecore scheduled task at the same time every day
• If you need your scheduled tasks to run at a specific time then I recommend the Akshay Sura’s Sitecore module Sitecron available on the marketplace.

Additional Resources

• Creating an AWS SQS Queue
• AWS Identity and Access Management (IAM) Best Practices
• AWS SDK for .NET
• Send Message, Receive Message, Delete Message
• Sitecron Module

 

In the next post in this series, I will look at handling failed messages using a dead-letter queue and Sitecore workflow.