[SAA + SAP] 21. SQS

SAA

• Max retention days 14 days
• Max size 256 KB

• Cross Account Access
• S3 push notification 

• MaximumReceive API
• Set threshold

• Request Queue
• Response Queues for each request (2 requests, then create 2 response queues)
• SQS Temporary Queue Client

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-temporary-queues.html

 

FIFO Queue

When creating a FIFI queue, Queue name should end with `.fifo`

Has GroupID and Deduplication ID

 

• Need to create a Custom Metric
• Queue length: ApproximateNumberOfMessages

 

SAP

• SQS can be used as a write buffer for DynamoDB
  • Because SQS auto-scale
  • write an application to read from SQS then insert to DynamoDB
  • to prevent PrevisionThroughtExpection error

• SQS might be processed twice
• Need to make sure consumer is idempotency
• For example, two EC2 happens to handle the same SQS message
• Then need to insert data to DynamoDB
• this is not idempotent, because two record will be the same
• Instead, we can upsert into DynamoDB based on primary key
• so that, two operations are still idempotency

idempotency is a Web API design principle defined as the ability to apply the same operation multiple times without changing the result beyond the first try.

 

Upsert: An operation that inserts rows into a database table if they do not already exist, or updates them if they do.

 

• Lambda read 10 message in batch
• Using long polling for efficiency
• DLQ should be set only on SQS side
• Invoke DLQ is async, because Lambda doesn't need response from DLQ
• Use Lambda destination
• 6x the timeout of your Lambda function

Kinesis and SQS FIFO are both in order, but FIFO only can process 3,000 messages per second

 

 

Idempotent need to be handeld in application side