[AWS] Amazon Simple Queue Service (SQS)
初步了解
一、是什么
Ref: https://aws.amazon.com/cn/sqs/
Amazon Simple Queue Service
适用于微服务、分布式系统和无服务器应用程序的完全托管的消息队列
二、官方示范
Ref: Python Code Samples for Amazon SQS
三、实践出真知
有命令行步骤：python boto AWS SQS connection [可以实践下]
-
boto是什么？
Boto 是AWS的基于python的SDK（当然还支持其他语言的SDK，例如Ruby, Java等)，Boto允许开发人员编写软件时使用亚马逊等服务像S3和EC2等，Boto提供了简单，面向对象的API，也提供了低等级的服务接入。
$ git clone git://github.com/boto/boto.git
$ cd boto
$ sudo python setup.py install
$ sudo pip install msgpack
$ pip install boto3
-
配置boto安全证书
这里我们用配置文件，首先新建一个 ~/.boto 文件，内容如下
[Credentials] aws_access_key_id=YOURACCESSKEY aws_secret_access_key=YOURSECRETKEY
现在可以使用boto了。
-
boto3的安装和配置
Ref: boto3用法
安装boto3和awscli：pip install boto3 awscli
配置aws：aws configure
根据提示输入access_key_id, secret_access_key和 region。
其中access_key_id, secret_access_key的默认存储位置为：~/.aws/credentials：
[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
region的存储位置为~/.aws/config:
[default]
region=us-east-1
-
创建与SQS的连接
import boto.sqs # 链接SQS conn=boto.sqs.connect_to_region('cn-north-1') q=conn.create_queue('demo-sqs') conn.get_all_queues() my_queue=conn.get_queue('queue-name')
链接成功后，就是消息发送操作了呢。
from boto.sqs.message import Message # 发送消息 m=Message() m.set_body('This is my first message!') q.write(m) # 获得消息们 rs=q.get_messages() len(rs) # 获得个别消息 m=rs[0] m.get_body()
四、消息定义
官方示范
Ref: Python Code Samples for Amazon SQS
一、Queue 操作
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # SPDX-License-Identifier: Apache-2.0 """ Purpose Demonstrate basic queue operations in Amazon Simple Queue Service (Amazon SQS). Learn how to create, get, and remove standard, FIFO, and dead-letter queues. Usage is shown in the test/test_queue_wrapper.py file. Prerequisites - You must have an AWS account, and have your default credentials and AWS Region configured as described in the [AWS Tools and SDKs Shared Configuration and Credentials Reference Guide](https://docs.aws.amazon.com/credref/latest/refdocs/creds-config-files.html). - Python 3.6 or later - Boto 3 1.11.10 or later - PyTest 5.3.5 or later (to run unit tests) Running the tests The best way to learn how to use this service is to run the tests. For instructions on testing, see the docstring in test/test_queue_wrapper.py. Running the code Run individual functions in the Python shell to make calls to your AWS account. > python >>> import queue_wrapper >>> queue_wrapper.create_queue("My-test-queue") sqs.Queue(url='https://us-west-2.queue.amazonaws.com/1234EXAMPLE/My-test-queue') >>> queue = queue_wrapper.get_queue("My-test-queue") >>> queue_wrapper.remove_queue(queue) Additional information Running this code might result in charges to your AWS account. """ import logging import boto3 from botocore.exceptions import ClientError logger = logging.getLogger(__name__) sqs = boto3.resource('sqs', region_name='us-east-2') def create_queue(name, attributes=None): """ Creates an Amazon SQS queue. Usage is shown in usage_demo at the end of this module. :param name: The name of the queue. This is part of the URL assigned to the queue. :param attributes: The attributes of the queue, such as maximum message size or whether it's a FIFO queue. :return: A Queue object that contains metadata about the queue and that can be used to perform queue operations like sending and receiving messages. """ if not attributes: attributes = {} try: queue = sqs.create_queue( QueueName=name, Attributes=attributes ) logger.info("Created queue '%s' with URL=%s", name, queue.url) except ClientError as error: logger.exception("Couldn't create queue named '%s'.", name) raise error else: return queue def get_queue(name): """ Gets an SQS queue by name. Usage is shown in usage_demo at the end of this module. :param name: The name that was used to create the queue. :return: A Queue object. """ try: queue = sqs.get_queue_by_name(QueueName=name) logger.info("Got queue '%s' with URL=%s", name, queue.url) except ClientError as error: logger.exception("Couldn't get queue named %s.", name) raise error else: return queue def get_queues(prefix=None): """ Gets a list of SQS queues. When a prefix is specified, only queues with names that start with the prefix are returned. Usage is shown in usage_demo at the end of this module. :param prefix: The prefix used to restrict the list of returned queues. :return: A list of Queue objects. """ if prefix: queue_iter = sqs.queues.filter(QueueNamePrefix=prefix) else: queue_iter = sqs.queues.all() queues = list(queue_iter) if queues: logger.info("Got queues: %s", ', '.join([q.url for q in queues])) else: logger.warning("No queues found.") return queues def remove_queue(queue): """ Removes an SQS queue. When run against an AWS account, it can take up to 60 seconds before the queue is actually deleted. Usage is shown in usage_demo at the end of this module. :param queue: The queue to delete. :return: None """ try: queue.delete() logger.info("Deleted queue with URL=%s.", queue.url) except ClientError as error: logger.exception("Couldn't delete queue with URL=%s!", queue.url) raise error def usage_demo(): """Demonstrates some ways to use the functions in this module.""" prefix = 'sqs-usage-demo-' river_queue = create_queue( prefix + 'peculiar-river', { 'MaximumMessageSize': str(1024), 'ReceiveMessageWaitTimeSeconds': str(20) } ) print(f"Created queue with URL: {river_queue.url}.") lake_queue = create_queue( prefix + 'strange-lake.fifo', { 'MaximumMessageSize': str(4096), 'ReceiveMessageWaitTimeSeconds': str(10), 'VisibilityTimeout': str(300), 'FifoQueue': str(True), 'ContentBasedDeduplication': str(True) } ) print(f"Created queue with URL: {lake_queue.url}.") stream_queue = create_queue(prefix + 'boring-stream') print(f"Created queue with URL: {stream_queue.url}.") alias_queue = get_queue(prefix + 'peculiar-river') print(f"Got queue with URL: {alias_queue.url}.") remove_queue(stream_queue) print(f"Removed queue with URL: {stream_queue.url}.", ) queues = get_queues(prefix=prefix) print(f"Got {len(queues)} queues.") for queue in queues: remove_queue(queue) print(f"Removed queue with URL: {queue.url}.") def main(): go = input("Running the usage demonstration uses your default AWS account " "credentials and might incur charges on your account. Do you want " "to continue (y/n)? ") if go.lower() == 'y': print("Starting the usage demo. Enjoy!") usage_demo() else: print("Thanks anyway!") if __name__ == '__main__': main()
二、Message 操作
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # SPDX-License-Identifier: Apache-2.0 """ Purpose Demonstrate basic message operations in Amazon Simple Queue Service (Amazon SQS). Learn how to send, receive, and delete messages from a queue. Usage is shown in the test/test_message_wrapper.py file. Prerequisites - You must have an AWS account, and have your default credentials and AWS Region configured as described in the [AWS Tools and SDKs Shared Configuration and Credentials Reference Guide](https://docs.aws.amazon.com/credref/latest/refdocs/creds-config-files.html). - Python 3.6 or later - Boto 3 1.11.10 or later - PyTest 5.3.5 or later (to run unit tests) Running the tests The best way to learn how to use this service is to run the tests. For instructions on testing, see the docstring in test/test_message_wrapper.py. Running the code Run individual functions in the Python shell to make requests to your AWS account. > python >>> import queue_wrapper >>> queue = queue_wrapper.create_queue("My-test-queue") >>> message_wrapper.send_message(queue, "Test message") >>> messages = message_wrapper.receive_messages(queue, 10, 10) >>> messages[0].body 'Test message' >>> queue_wrapper.remove_queue(queue) Additional information Running this code might result in charges to your AWS account. """ import logging import sys import boto3 from botocore.exceptions import ClientError import queue_wrapper logger = logging.getLogger(__name__) sqs = boto3.resource('sqs', region_name='us-east-2') def send_message(queue, message_body, message_attributes=None): """ Send a message to an Amazon SQS queue. Usage is shown in usage_demo at the end of this module. :param queue: The queue that receives the message. :param message_body: The body text of the message. :param message_attributes: Custom attributes of the message. These are key-value pairs that can be whatever you want. :return: The response from SQS that contains the assigned message ID. """ if not message_attributes: message_attributes = {} try: response = queue.send_message( MessageBody=message_body, MessageAttributes=message_attributes ) except ClientError as error: logger.exception("Send message failed: %s", message_body) raise error else: return response def send_messages(queue, messages): """ Send a batch of messages in a single request to an SQS queue. This request may return overall success even when some messages were not sent. The caller must inspect the Successful and Failed lists in the response and resend any failed messages. Usage is shown in usage_demo at the end of this module. :param queue: The queue to receive the messages. :param messages: The messages to send to the queue. These are simplified to contain only the message body and attributes. :return: The response from SQS that contains the list of successful and failed messages. """ try: entries = [{ 'Id': str(ind), 'MessageBody': msg['body'], 'MessageAttributes': msg['attributes'] } for ind, msg in enumerate(messages)]
response = queue.send_messages(Entries=entries)
if 'Successful' in response: for msg_meta in response['Successful']: logger.info( "Message sent: %s: %s", msg_meta['MessageId'], messages[int(msg_meta['Id'])]['body'] ) if 'Failed' in response: for msg_meta in response['Failed']: logger.warning( "Failed to send: %s: %s", msg_meta['MessageId'], messages[int(msg_meta['Id'])]['body'] ) except ClientError as error: logger.exception("Send messages failed to queue: %s", queue) raise error else: return response def receive_messages(queue, max_number, wait_time): """ Receive a batch of messages in a single request from an SQS queue. Usage is shown in usage_demo at the end of this module. :param queue: The queue from which to receive messages. :param max_number: The maximum number of messages to receive. The actual number of messages received might be less. :param wait_time: The maximum time to wait (in seconds) before returning. When this number is greater than zero, long polling is used. This can result in reduced costs and fewer false empty responses. :return: The list of Message objects received. These each contain the body of the message and metadata and custom attributes. """ try: messages = queue.receive_messages( MessageAttributeNames=['All'], MaxNumberOfMessages=max_number, WaitTimeSeconds=wait_time ) for msg in messages: logger.info("Received message: %s: %s", msg.message_id, msg.body) except ClientError as error: logger.exception("Couldn't receive messages from queue: %s", queue) raise error else: return messages def delete_message(message): """ Delete a message from a queue. Clients must delete messages after they are received and processed to remove them from the queue. Usage is shown in usage_demo at the end of this module. :param message: The message to delete. The message's queue URL is contained in the message's metadata. :return: None """ try: message.delete() logger.info("Deleted message: %s", message.message_id) except ClientError as error: logger.exception("Couldn't delete message: %s", message.message_id) raise error def delete_messages(queue, messages): """ Delete a batch of messages from a queue in a single request. Usage is shown in usage_demo at the end of this module. :param queue: The queue from which to delete the messages. :param messages: The list of messages to delete. :return: The response from SQS that contains the list of successful and failed message deletions. """ try: entries = [{ 'Id': str(ind), 'ReceiptHandle': msg.receipt_handle } for ind, msg in enumerate(messages)] response = queue.delete_messages(Entries=entries) if 'Successful' in response: for msg_meta in response['Successful']: logger.info("Deleted %s", messages[int(msg_meta['Id'])].receipt_handle) if 'Failed' in response: for msg_meta in response['Failed']: logger.warning( "Could not delete %s", messages[int(msg_meta['Id'])].receipt_handle ) except ClientError: logger.exception("Couldn't delete messages from queue %s", queue) else: return response def usage_demo(): """ Demonstrates some ways to use the functions in this module. This demonstration reads the lines from this Python file and sends the lines in batches of 10 as messages to a queue. It then receives the messages in batches until the queue is empty. It reassembles the lines of the file and verifies they match the original file. """ def pack_message(msg_path, msg_body, msg_line): return { 'body': msg_body, 'attributes': { 'path': {'StringValue': msg_path, 'DataType': 'String'}, 'line': {'StringValue': str(msg_line), 'DataType': 'String'} } } def unpack_message(msg): return (msg.message_attributes['path']['StringValue'], msg.body, int(msg.message_attributes['line']['StringValue'])) queue = queue_wrapper.create_queue('sqs-usage-demo-message-wrapper') with open(__file__) as file: lines = file.readlines() line = 0 batch_size = 10 received_lines = [None]*len(lines) print(f"Sending file lines in batches of {batch_size} as messages.") while line < len(lines): messages = [pack_message(__file__, lines[index], index) for index in range(line, min(line + batch_size, len(lines)))] line = line + batch_size send_messages(queue, messages) print('.', end='') sys.stdout.flush() print(f"Done. Sent {len(lines) - 1} messages.") print(f"Receiving, handling, and deleting messages in batches of {batch_size}.") more_messages = True while more_messages: received_messages = receive_messages(queue, batch_size, 2) print('.', end='') sys.stdout.flush() for message in received_messages: path, body, line = unpack_message(message) received_lines[line] = body if received_messages: delete_messages(queue, received_messages) else: more_messages = False print('Done.') if all([lines[index] == received_lines[index] for index in range(len(lines))]): print(f"Successfully reassembled all file lines!") else: print(f"Uh oh, some lines were missed!") queue.delete() def main(): go = input("Running the usage demonstration uses your default AWS account " "credentials and might incur charges on your account. Do you want " "to continue (y/n)? ") if go.lower() == 'y': print("Starting the usage demo. Enjoy!") usage_demo() else: print("Thanks anyway!") if __name__ == '__main__': main()
End.
