AWS SQS service

Amazon Simple Queue Service (Amazon SQS) is a reliable, highly-scalable hosted queue for storing messages as they travel between applications or microservices. Amazon SQS moves data between distributed application components and helps you decouple these components.

There are two types of queues:

1. Standard queue

2. FIFO queue 

 

 

SQS offers two types of queues – Standard & FIFO queues

Message Order

• Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent. occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order
• FIFO queues offer first-in-first-out delivery and exactly-once processing: the order in which messages are sent and received is strictly preserved

Delivery

• Standard queues guarantee that a message is delivered at least once and duplicates can be introduced into the queue
• FIFO queues ensure a message is delivered exactly once and remains available until a consumer processes and deletes it; duplicates are notintroduced into the queue

Transactions Per Second (TPS)

• Standard queues allow nearly-unlimited number of transactions per second
• FIFO queues are limited to 300 transactions per second per API action

Regions

• Standard queues are available in all the regions
• FIFO queues are currently available in limited regions including the US West (Oregon), US East (Ohio), US East (N. Virginia), and EU (Ireland)

SQS Buffered Asynchronous Client

• FIFO queues aren’t currently compatible with the SQS Buffered Asynchronous Client, where messages are buffered at client side and send as a single request to the SQS queue to reduce cost.

AWS Services Supported

• Standard Queues are supported by all AWS services
• FIFO Queues are currently not supported by all AWS services like
  • CloudWatch Events
  • S3 Event Notifications
  • SNS Topic Subscriptions
  • Auto Scaling Lifecycle Hooks
  • AWS IoT Rule Actions
  • AWS Lambda Dead Letter Queues

Use Cases

• Standard queues can be used in any scenarios, as long as the application can process messages that arrive more than once and out of order
  • Decouple live user requests from intensive background work: Let users upload media while resizing or encoding it.
  • Allocate tasks to multiple worker nodes: Process a high number of credit card validation requests.
  • Batch messages for future processing: Schedule multiple entries to be added to a database.
• FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated
  • Ensure that user-entered commands are executed in the right order.
  • Display the correct product price by sending price modifications in the right order.
  • Prevent a student from enrolling in a course before registering for an account.

 

 

SQS APIs: two major categories, queue management and message management

 

 

 

 

FIFO queue specific API parameters:

 

MessageDeduplicationId

This parameter applies only to FIFO (first-in-first-out) queues.

The token used for deduplication of messages within a 5-minute minimum deduplication interval. If a message with a particular MessageDeduplicationId is sent successfully, subsequent messages with the same MessageDeduplicationIdare accepted successfully but aren't delivered. For more information, see Exactly-Once Processing in the Amazon Simple Queue Service Developer Guide.

• Every message must have a unique MessageDeduplicationId,

  • You may provide a MessageDeduplicationId explicitly.

  • If you aren't able to provide a MessageDeduplicationId and you enable ContentBasedDeduplication for your queue, Amazon SQS uses a SHA-256 hash to generate the MessageDeduplicationId using the body of the message (but not the attributes of the message).

  • If you don't provide a MessageDeduplicationId and the queue doesn't have ContentBasedDeduplication set, the action fails with an error.

  • If the queue has ContentBasedDeduplication set, your MessageDeduplicationId overrides the generated one.

• When ContentBasedDeduplication is in effect, messages with identical content sent within the deduplication interval are treated as duplicates and only one copy of the message is delivered.

• If you send one message with ContentBasedDeduplication enabled and then another message with a MessageDeduplicationId that is the same as the one generated for the first MessageDeduplicationId, the two messages are treated as duplicates and only one copy of the message is delivered.

Note

The MessageDeduplicationId is available to the recipient of the message (this can be useful for troubleshooting delivery issues).

If a message is sent successfully but the acknowledgement is lost and the message is resent with the sameMessageDeduplicationId after the deduplication interval, Amazon SQS can't detect duplicate messages.

Amazon SQS continues to keep track of the message deduplication ID even after the message is received and deleted.

The length of MessageDeduplicationId is 128 characters. MessageDeduplicationId can contain alphanumeric characters (a-z, A-Z, 0-9) and punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~).

 

MessageGroupId

This parameter applies only to FIFO (first-in-first-out) queues.

The tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are processed in a FIFO manner (however, messages in different message groups might be processed out of order). To interleave multiple ordered streams within a single queue, use MessageGroupId values (for example, session data for multiple users). In this scenario, multiple readers can process the queue, but the session data of each user is processed in a FIFO fashion.

• You must associate a non-empty MessageGroupId with a message. If you don't provide a MessageGroupId, the action fails.

• ReceiveMessage might return messages with multiple MessageGroupId values. For each MessageGroupId, the messages are sorted by time sent. The caller can't specify a MessageGroupId.

The length of MessageGroupId is 128 characters. Valid values are alphanumeric characters and punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~).

 

Increasing Throughput with Horizontal Scaling and Batching

Amazon SQS queues can deliver very high throughput (many thousands of messages per second). To achieve this throughput, you must scale message producers and consumers horizontally (add more producers and consumers).

In addition to horizontal scaling, batching provides a throughput with fewer threads, connections, and requests than would be required by individual message requests. You can use batched Amazon SQS API actions to send, receive, or delete up to 10 messages at a time. Because Amazon SQS charges by the request instead of by the message, batching can also substantially reduce costs.

 

 

Limits Related to Queues

The following table lists limits related to queues.

Limit	Description
Inflight messages per queue	For standard queues, there can be a maximum of 120,000 inflight messages per queue. If you reach this limit, Amazon SQS returns the OverLimit error message. To avoid reaching the limit, you should delete messages from the queue after they're processed. You can also increase the number of queues you use to process your messages.
	For FIFO queues, there can be a maximum of 20,000 inflight messages per queue. If you reach this limit, Amazon SQS returns no error messages.
Queue name	A queue name can have up to 80 characters. The following characters are accepted: alphanumeric characters, hyphens (-), and underscores (_). Note Queue names are case-sensitive (for example, Test-queue and test-queue are different queues).
	The name of a FIFO queue must end with the .fifo suffix. The suffix counts towards the 80-character queue name limit. To determine whether a queue is FIFO, you can check whether the queue name ends with the suffix.
Queue tag	We don't recommend adding more than 50 tags to a queue.
	The tag Key is required, but the tag Value is optional.
	The tag Key and tag Value are case-sensitive.
	The tag Key and tag Value can include Unicode alphanumeric characters in UTF-8 and whitespaces. The following special characters are allowed: _ . : / = + - @
	The tag Key or Value must not include the reserved prefix aws: (you can't delete tag keys or values with this prefix).
	The maximum tag Key length is 128 Unicode characters in UTF-8. The tag Key must not be empty or null.
	The maximum tag Value length is 256 Unicode characters in UTF-8. The tag Value may be empty or null.
	Tagging API actions are limited to 5 TPS per AWS account. If your application requires a higher throughput, file a technical support request.

Limits Related to Messages

The following table lists limits related to messages.

Limit	Description
Message attributes	A message can contain up to 10 metadata attributes.
Message batch	A single message batch request can include a maximum of 10 messages. For more information, see Advanced Configuration in the Amazon SQS Batch API Actions section.
Message content	A message can include only XML, JSON, and unformatted text. The following Unicode characters are allowed: #x9 | #xA | #xD | #x20 to #xD7FF | #xE000 to #xFFFD | #x10000 to #x10FFFF Any characters not included in this list are rejected. For more information, see the W3C specification for characters.
Message retention	By default, a message is retained for 4 days. The minimum is 60 seconds (1 minute). The maximum is 1,209,600 seconds (14 days).
Message throughput	Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.
	FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second. To request a limit increase, file a support request.
Message size	The minimum message size is 1 byte (1 character). The maximum is 262,144 bytes (256 KB). To send messages larger than 256 KB, you can use the Amazon SQS Extended Client Library for Java. This library allows you to send an Amazon SQS message that contains a reference to a message payload in Amazon S3. The maximum payload size is 2 GB.
Message visibility timeout	The default visibility timeout for a message is 30 seconds. The maximum is 12 hours.
Policy information	The maximum limit is 8,192 bytes, 20 statements, 50 principals, or 10 conditions. For more information, see Limits Related to Policies.

 

 

Using Amazon SQS Dead-Letter Queues

Amazon SQS supports dead-letter queues. A dead-letter queue is a queue that other (source) queues can target for messages that can't be processed (consumed) successfully. Dead-letter queues are useful for debugging your application or messaging system. Dead-letter queues allow you to isolate problematic messages to determine why their processing doesn't succeed.

 

essages Using Short Polling

The behavior of consuming messages from the queue depends on whether you use short (standard) polling, the default behavior, or long polling. For more information about long polling, see Amazon SQS Long Polling.

When you consume messages from the queue using short polling, Amazon SQS samples a subset of the servers (based on a weighted random distribution) and returns messages from just these servers. Thus, a particular receive request might not return all of your messages. However, if you have a small number of messages in your queue (fewer than 1,000), one particular request might not return any of your messages, whereas a subsequent request returns them. If you keep consuming from your queues, Amazon SQS samples all of the servers, and you receive all your messages.

The following figure shows the short-polling behavior of messages returned after one of your system components makes a receive request. Amazon SQS samples several of the servers (in gray) and returns the messages from those servers (Message A, C, D, and B). Message E isn't returned to this particular request, but is returned to a subsequent request.

 

Amazon SQS Long Polling

Long polling helps reduce your cost of using Amazon SQS by reducing the number of empty responses (when there are no messages available to return in reply to a ReceiveMessage request sent to an Amazon SQS queue) and eliminating false empty responses (when messages are available in the queue but aren't included in the response):

• Long polling reduces the number of empty responses by allowing Amazon SQS to wait until a message is available in the queue before sending a response. Unless the connection times out, the response to the ReceiveMessage request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage action.

• Long polling eliminates false empty responses by querying all (rather than a limited number) of the servers.

• Long polling returns messages as soon any message becomes available.

 

Message ID

Each message receives a system-assigned message ID that Amazon SQS returns to you in the SendMessage response. This identifier is useful for identifying messages. (However, to delete a message you need the message's receipt handle.) The maximum length of a message ID is 100 characters.

Receipt Handle

Every time you receive a message from a queue, you receive a receipt handle for that message. This handle is associated with the action of receiving the message, not with the message itself. To delete the message or to change the message visibility, you must provide the receipt handle (not the message ID). Thus, you must always receive a message before you can delete it (you can't put a message into the queue and then recall it). The maximum length of a receipt handle is 1024 characters.

Important

If you receive a message more than once, each time you receive it, you get a different receipt handle. You must provide the most recently received receipt handle when you request to delete the message (otherwise, the message might not be deleted).

 

Additional Identifiers for FIFO Queues

For more information about the following identifiers, see Exactly-Once Processing and the Amazon Simple Queue Service API Reference.

Message Deduplication ID

The token used for deduplication of sent messages. If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication interval.

Message Group ID

The tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are always processed one by one, in a strict order relative to the message group (however, messages that belong to different message groups might be processed out of order).

Sequence Number

The large, non-consecutive number that Amazon SQS assigns to each message.

 

Visibility Timeout

When a consumer receives and processes a message from a queue, the message remains in the queue. Amazon SQS doesn't automatically delete the message. Because Amazon SQS is a distributed system, there's no guarantee that the consumer actually receives the message (for example, due to a connectivity issue, or due to an issue in the consumer application). Thus, the consumer must delete the message from the queue after receiving and processing it.

Immediately after the message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. The default visibility timeout for a message is 30 seconds. The maximum is 12 hours.

 

Managing Large Amazon SQS Messages Using Amazon S3

You can manage Amazon SQS messages with Amazon S3. This is especially useful for storing and consuming messages with a message size of up to 2 GB. To manage Amazon SQS messages with Amazon S3, use the Amazon SQS Extended Client Library for Java. Specifically, you use this library to:

• Specify whether messages are always stored in Amazon S3 or only when the size of a message exceeds 256 KB.

• Send a message that references a single message object stored in an Amazon S3 bucket.

• Get the corresponding message object from an Amazon S3 bucket.

• Delete the corresponding message object from an Amazon S3 bucket.