消息中间件RabbitMQ

Rabbitmq

​ 之前我是使用redis当做消息队列来使用，因为队列是先进先出的数据结构，有个方法 blpop阻塞式弹出实现队列，但是使用redis当做中间件是不专业的，而Rabbitmq来作为消息中间件是专业的

消息队列解决了

1.应用解耦 多个应用使用消息队列做缓存.任务存放在消息队列中。应用从消息队列中取

2.流量削峰 在高峰期 操作过多 系统处理不了，只能限制超过数量，固定死。 消息队列用来做缓冲，应用去消费。

3 消息分发 比如A生产消息放在消息中间件里，其他B,C,D只要订阅了就都可以拿。

4 异步消息 与应用解耦 A处理任务，A调用B的接口，B把结果放在消息中间件里，A在同时间去处理别的任务，不需要在原地等结果。

下载

windows环境

需要配合Erlang环境

Erlang下载地址 OTP Versions Tree (erlang.org)

Rabbitmq下载地址在视窗上安装 — 兔子MQ (rabbitmq.com)

对应版本关系RabbitMQ Erlang Version Requirements — RabbitMQ

Linux

安装erlang
yum -y install erlang
安装Rabbitmq
yum -y install Rabbitmq-server

docker

docker pull rabbitmq:management
# 指定默认用户名与密码，做端口映射，1个端口是链接服务的端口，一个是Web管理界面的端口
docker run -id --name Myrabbitmq -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin -p 15672:15672 -p 5672:5672 rabbitmq:management

运行起来了以后通过虚拟机的 10.0.0.200:15672 链接到Web管理界面修改密码

官方使用文档RabbitMQ Tutorials — RabbitMQ

基本使用

下载pika模块

pip3.8 install pika

生产者消费者

send.py 代码创建队列存放消息

import pika
# 1.拿到链接  无密码
# connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672))
# 有密码
credentials = pika.PlainCredentials('admin','admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672,credentials=credentials))

#2. 链接channal 通道
channel = connection.channel()

# 3 创建队列 队列名字为hello，可以创建多个队列
channel.queue_declare(queue='hello')

# 4 向队列中发送消息
"""
routing_key 队列名
body        消息内容
"""
channel.basic_publish(exchange='',routing_key='hello',body='Hello World!1')
print(" Sent 'Hello World!'")
# 5 关闭链接
connection.close()

receiver.py 阻塞取消息

import pika, sys, os

# 1. 密码拿到链接
credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

# 拿到通道创建队列
channel = connection.channel()
channel.queue_declare(queue='hello')  # 如果队列不存在，则创建队列，如果存在，则不创建

# 回调函数
def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

# 去hello中接收消息，回调函数为callback
"auto_ack 消息确认机制"
channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
# 开始接收消息 一直阻塞，队列里有消息就会拿出来
channel.start_consuming()

这里可能会出现问题，在消息队列中拿到数据后开始处理业务逻辑，过程中可能会报错，但是消息拿到之后队列里就没了这条数据。所以有一个消息确认机制

auto_ack =True 消息拿出来就自动删了。

​ =False 消息一直都在，我们要告诉他用完了，才会删除

优化后，消息安全之ack

import pika, sys, os

# 1. 密码拿到链接
credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

# 拿到通道创建队列
channel = connection.channel()
channel.queue_declare(queue='hello')  # 如果队列不存在，则创建队列，如果存在，则不创建

# 回调函数
def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
    ch.basic_ack(delivery_tag=method.delivery_tag)  # 手动确认消息已经被消费
# 去hello中接收消息，回调函数为callback
channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=False)
print(' [*] Waiting for messages. To exit press CTRL+C')
# 开始接收消息 一直阻塞，队列里有消息就会拿出来
channel.start_consuming()

消息安全之durable持久化

服务挂掉消息就没了，需要对队列和消息做持久化

对队列做持久化，创建队列时加个参数 durable=True

send.py

import pika

credentials = pika.PlainCredentials('admin','admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672,credentials=credentials))

#2. 链接channal 通道
channel = connection.channel()

# 3 创建队列 队列名字为hello，把queue持久化
channel.queue_declare(queue='hello',durable=True)

# 4 向队列中发送消息
"""
routing_key 队列名
body        消息内容
"""
channel.basic_publish(exchange='',routing_key='hello',body='heelo world')
print(" Sent 'Hello World!'")
# 5 关闭链接
connection.close()

receiver.py

import pika, sys, os

# 1. 密码拿到链接
credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

# 拿到通道创建队列
channel = connection.channel()
channel.queue_declare(queue='hello',durable=True)  # 如果队列不存在，则创建队列，如果存在，则不创建

# 回调函数
def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
    ch.basic_ack(delivery_tag=method.delivery_tag)  # 手动确认消息已经被消费
# 去hello中接收消息，回调函数为callback
channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=False)
print(' [*] Waiting for messages. To exit press CTRL+C')
# 开始接收消息 一直阻塞，队列里有消息就会拿出来
channel.start_consuming()

手动停掉服务以后在启动，队列还在，但是队列里的消息没了，现在对消息做持久化

在存入消息的时候增加参数properties=pika.BasicProperties(delivery_mode=2,)

send.py

import pika
# 1.拿到链接  无密码
# connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672))
# 有密码
credentials = pika.PlainCredentials('admin','admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672,credentials=credentials))

#2. 链接channal 通道
channel = connection.channel()

# 3 创建队列 队列名字为hello，把queue持久化
channel.queue_declare(queue='hello',durable=True)

# 4 向队列中发送消息
"""
routing_key 队列名
body        消息内容
"""
channel.basic_publish(exchange='',routing_key='hello',body='heelo world',properties=pika.BasicProperties(delivery_mode=2,))  # delivery_mode=2 消息持久化
print(" Sent 'Hello World!'")
# 5 关闭链接
connection.close()

闲置消费

​ 多个消费者是按照顺序来，第一个消息给第一个消费者，第二给消息给第二个消费者，但是可能第一个消费者处理消息耗时久，一直没有结束，就可以让第二个消费者优先获得闲置的消息。

在接收消息之前添加配置channel.basic_qos(prefetch_count=1)

import time

import pika, sys, os

# 1. 密码拿到链接
credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

# 拿到通道创建队列
channel = connection.channel()
channel.queue_declare(queue='hello',durable=True)  # 如果队列不存在，则创建队列，如果存在，则不创建

# 回调函数
def callback(ch, method, properties, body):
    time.sleep(10)
    print(" [x] Received %r" % body)
    ch.basic_ack(delivery_tag=method.delivery_tag)  # 手动确认消息已经被消费

channel.basic_qos(prefetch_count=1)  # 谁闲置谁去消费
# 去hello中接收消息，回调函数为callback
channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=False)
print(' [*] Waiting for messages. To exit press CTRL+C')
# 开始接收消息 一直阻塞，队列里有消息就会拿出来
channel.start_consuming()

发布订阅

​ 一个发布者可以有多个订阅者。只要订阅了都可以取出来，同一条消息订阅者都能收到。和闲置消费可不一样哦。 闲置消费只是个队列，拿出来一个就没了

send.py

import pika

credentials = pika.PlainCredentials('admin','admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672,credentials=credentials))
channel = connection.channel()
# 声明exchange
channel.exchange_declare(exchange='logs',exchange_type='fanout')

channel.basic_publish(exchange='logs',routing_key='',body='heelo world')
connection.close()

receiver.py

import pika, sys, os

credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

channel = connection.channel()
# 声明exchange，秘书  exchange_type='fanout'表示将消息发送给所有队列
channel.exchange_declare(exchange='logs', exchange_type='fanout')

# 随机生成队列名
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
print(queue_name)

# 将队列绑定到exchange上
channel.queue_bind(exchange='logs', queue=queue_name)


def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)

channel.start_consuming()

发布订阅高级

之前使用的是fanout 模式。 表示把消息发给所有队列

还有很多模式

direct模式，可以让订阅者监听特定的数据.

​ 应用场景 产生日志，日志有不同的级别，订阅者订阅不同的级别。

send.py

import pika

credentials = pika.PlainCredentials('admin','admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672,credentials=credentials))
channel = connection.channel()
# 声明exchange
channel.exchange_declare(exchange='m1',exchange_type='direct')
# 发送不同的routing_key
channel.basic_publish(exchange='m1',routing_key='nb',body='123')
connection.close()

receiver.py

import pika, sys, os

credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

channel = connection.channel()
# 声明exchange，秘书
channel.exchange_declare(exchange='m1', exchange_type='direct')

# 随机生成队列名
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
print(queue_name)

# 接收指定的routing_key消息
channel.queue_bind(exchange='m1', queue=queue_name,routing_key='lxj')


def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)

channel.start_consuming()

topic模式 模糊匹配

# 代表一个或多部分

*代表一个部分

send.py

import pika

credentials = pika.PlainCredentials('admin','admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200',port=5672,credentials=credentials))
channel = connection.channel()
# 声明exchange
channel.exchange_declare(exchange='m1',exchange_type='topic')
# 发送不同的routing_key
"""# 可以接收lxj开头后面多部分lxj.handsome.xx，*只能接收lxj开头 后面一部分lxj.handsome """
channel.basic_publish(exchange='m1',routing_key='lxj.handsome.xx',body='a123')
connection.close()

receiver.py

import pika, sys, os

credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

channel = connection.channel()
# 声明exchange，
channel.exchange_declare(exchange='m1', exchange_type='topic')

# 随机生成队列名
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
print(queue_name)

# 将队列绑定到exchange上，订阅
channel.queue_bind(exchange='m1', queue=queue_name,routing_key='lxj.#')


def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)

channel.start_consuming()

receiver1.py

import pika, sys, os

credentials = pika.PlainCredentials('admin', 'admin')
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.0.0.200', port=5672, credentials=credentials))

channel = connection.channel()
# 声明exchange，秘书
channel.exchange_declare(exchange='m1', exchange_type='topic')

# 随机生成队列名
result = channel.queue_declare(queue='', exclusive=True)
queue_name = result.method.queue
print(queue_name)

# 接收指定的routing_key消息
channel.queue_bind(exchange='m1', queue=queue_name,routing_key='lxj.*')


def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)

channel.start_consuming()