说明:
几个概念说明: Broker:简单来说就是消息队列服务器实体。 Exchange:消息交换机,它指定消息按什么规则,路由到哪个队列。 Queue:消息队列载体,每个消息都会被投入到一个或多个队列。 Binding:绑定,它的作用就是把exchange和queue按照路由规则绑定起来。 Routing Key:路由关键字,exchange根据这个关键字进行消息投递。 vhost:虚拟主机,一个broker里可以开设多个vhost,用作不同用户的权限分离。 producer:消息生产者,就是投递消息的程序。 consumer:消息消费者,就是接受消息的程序。 channel:消息通道,在客户端的每个连接里,可建立多个channel,每个channel代表一个会话任务。 消息队列的使用过程大概如下: (1)客户端连接到消息队列服务器,打开一个channel。 (2)客户端声明一个exchange,并设置相关属性。 (3)客户端声明一个queue,并设置相关属性。 (4)客户端使用routing key,在exchange和queue之间建立好绑定关系。 (5)客户端投递消息到exchange。 exchange接收到消息后,就根据消息的key和已经设置的binding,进行消息路由,将消息投递到一个或多个队列里。 exchange也有几个类型,完全根据key进行投递的叫做Direct交换机,例如,绑定时设置了routing key为”abc”,那么客户端提交的消息,只有设置了key为”abc”的才会投递到队列。
对key进行模式匹配后进行投递的叫做Topic交换机,符号”#”匹配一个或多个词,符号”*”匹配正好一个词。例如”abc.#”匹配”abc.def.ghi”,”abc.*”只匹配”abc.def”。
还有一种不需要key的,叫做Fanout交换机,它采取广播模式,一个消息进来时,投递到与该交换机绑定的所有队列。 RabbitMQ支持消息的持久化,也就是数据写在磁盘上,为了数据安全考虑,我想大多数用户都会选择持久化。消息队列持久化包括3个部分: (1)exchange持久化,在声明时指定durable => 1 (2)queue持久化,在声明时指定durable => 1 (3)消息持久化,在投递时指定delivery_mode => 2(1是非持久化) 如果exchange和queue都是持久化的,那么它们之间的binding也是持久化的。如果exchange和queue两者之间有一个持久化,一个非持久化,就不允许建立绑定。
安装:
下载: wget www.rabbitmq.com/releases/erlang/erlang-18.3-1.el7.centos.x86_64.rpm wget http://repo.iotti.biz/CentOS/7/x86_64/socat-1.7.3.2-5.el7.lux.x86_64.rpm wget www.rabbitmq.com/releases/rabbitmq-server/v3.6.5/rabbitmq-server-3.6.5-1.noarch.rpm 安装: rpm -ivh erlang-18.3-1.el7.centos.x86_64.rpm rpm -ivh rabbitmq-server-3.6.5-1.noarch.rpm rpm -ivh socat-1.7.3.2-1.1.el7.x86_64.rpm 配置文件: vim /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/ebin/rabbit.app 比如修改密码、配置等等,例如:loopback_users 中的 <<"guest">>,只保留guest 服务启动和停止: 启动 rabbitmq-server start & 停止 rabbitmqctl app_stop rabbitmqctl start_app rabbitmq-plugins enable rabbitmq_management rabbitmqctl stop 管理插件:rabbitmq-plugins enable rabbitmq_management 访问地址:http://192.168.11.76:15672/
本地测试环境移除所有数据:
rabbitmqtl reset 移除所有数据 要在 rabbitmqctl stop_app之后使用
[root@localhost ~]# rabbitmq rabbitmqctl rabbitmq-plugins rabbitmq-server
[root@localhost soft]# rabbitmqctl list_queues Listing queues ... task_queue123 0 [root@localhost soft]# rabbitmqctl list_vhosts Listing vhosts ... / /sunlong
[root@localhost soft]# rabbitmqctl list_exchanges Listing exchanges ... amq.fanout fanout amq.match headers direct amq.rabbitmq.trace topic amq.headers headers amq.direct direct amq.rabbitmq.log topic amq.topic topic
1 [root@localhost soft]# rabbitmqctl status 2 Status of node rabbit@localhost ... 3 [{pid,2521}, 4 {running_applications, 5 [{rabbitmq_management,"RabbitMQ Management Console","3.6.5"}, 6 {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.5"}, 7 {rabbit,"RabbitMQ","3.6.5"}, 8 {os_mon,"CPO CXC 138 46","2.4"}, 9 {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.5"}, 10 {webmachine,"webmachine","1.10.3"}, 11 {mochiweb,"MochiMedia Web Server","2.13.1"}, 12 {compiler,"ERTS CXC 138 10","6.0.3"}, 13 {ssl,"Erlang/OTP SSL application","7.3"}, 14 {public_key,"Public key infrastructure","1.1.1"}, 15 {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"}, 16 {amqp_client,"RabbitMQ AMQP Client","3.6.5"}, 17 {asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"}, 18 {inets,"INETS CXC 138 49","6.2"}, 19 {rabbit_common,[],"3.6.5"}, 20 {xmerl,"XML parser","1.3.10"}, 21 {mnesia,"MNESIA CXC 138 12","4.13.3"}, 22 {syntax_tools,"Syntax tools","1.7"}, 23 {crypto,"CRYPTO","3.6.3"}, 24 {sasl,"SASL CXC 138 11","2.7"}, 25 {stdlib,"ERTS CXC 138 10","2.8"}, 26 {kernel,"ERTS CXC 138 10","4.2"}]}, 27 {os,{unix,linux}}, 28 {erlang_version, 29 "Erlang/OTP 18 [erts-7.3] [source] [64-bit] [async-threads:64] [hipe] [kernel-poll:true]\n"}, 30 {memory, 31 [{total,59824960}, 32 {connection_readers,0}, 33 {connection_writers,0}, 34 {connection_channels,0}, 35 {connection_other,2680}, 36 {queue_procs,19328}, 37 {queue_slave_procs,0}, 38 {plugins,1009776}, 39 {other_proc,18123920}, 40 {mnesia,70096}, 41 {mgmt_db,750320}, 42 {msg_index,43856}, 43 {other_ets,1399424}, 44 {binary,28720}, 45 {code,27824046}, 46 {atom,1000601}, 47 {other_system,9552193}]}, 48 {alarms,[]}, 49 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]}, 50 {vm_memory_high_watermark,0.4}, 51 {vm_memory_limit,768196608}, 52 {disk_free_limit,50000000}, 53 {disk_free,16176914432}, 54 {file_descriptors, 55 [{total_limit,65435}, 56 {total_used,2}, 57 {sockets_limit,58889}, 58 {sockets_used,0}]}, 59 {processes,[{limit,1048576},{used,229}]}, 60 {run_queue,0}, 61 {uptime,95}, 62 {kernel,{net_ticktime,60}}]
rabbitmq php扩展amqp安装
①安装rabbitmq-c-0.7.1
没有安装就会提示上面的错误
下载地址:https://github.com/alanxz/rabbitmq-c
我选择的是最新版本0.7.1
wget https://github.com/alanxz/rabbitmq-c/releases/download/v0.7.1/rabbitmq-c-0.7.1.tar.gz tar zxf rabbitmq-c-0.7.1.tar.gz cd rabbitmq-c-0.7.1 ./configure --prefix=/usr/local/rabbitmq-c-0.7.1 make && make install
②安装amqp
下载地址https://pecl.php.net/package/amqp
我选择的是1.6.1
wget https://pecl.php.net/get/amqp-1.6.1.tgz tar zxf amqp-1.6.1.tgz cd amqp-1.6.1 /usr/local/php/bin/phpize ./configure --with-php-config=/usr/local/php/bin/php-config --with-amqp --with-librabbitmq-dir=/usr/local/rabbitmq-c-0.7.1
注意:这里的/usr/local/rabbitmq-c-0.7.1
要跟上面rabbitmq-c
安装的地址一样
make && make install
③添加php模块
vi /usr/local/php/etc/php.ini
最后添加一行
extension = /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/amqp.so
重启php
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); //声明队列 $channel->queue_declare('hello', false, false, false, false); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; }; //监听队列 $channel->basic_consume('hello', '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; use PhpAmqpLib\Message\AMQPMessage; //获取连接 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); //从连接中创建通道 $channel = $connection->channel(); //声明队列 $channel->queue_declare('hello', false, false, false, false); $msg = new AMQPMessage('Hello World!'); $channel->basic_publish($msg, '', 'hello'); echo " [x] Sent 'Hello World!'\n"; $channel->close(); $connection->close();
channel->basic_consume("TestQueue", "", false, false, false, false, $callback);
顺序 | 参数名 | 默认值 | 作用 |
---|---|---|---|
1 | queue | 消息要取得消息的队列名 | |
2 | consumer_tag | 消费者标签 | |
3 | no_local | false | 这个功能属于AMQP的标准,但是rabbitMQ并没有做实现. |
4 | no_ack | false | 收到消息后,是否不需要回复确认即被认为被消费 |
5 | exclusive | false | 排他消费者,即这个队列只能由一个消费者消费.适用于任务不允许进行并发处理的情况下.比如系统对接 |
6 | nowait | false | 不返回执行结果,但是如果排他开启的话,则必须需要等待结果的,如果两个一起开就会报错 |
7 | callback | null | 回调函数 |
8 | ticket | null | |
9 | arguments | null |
queue_declare( $queue = '', $passive = false, $durable = false,//是否持久化 $exclusive = false,//独占 保证顺序消费 $auto_delete = true,//如果和交换机没有绑定关系 则自动删除 $nowait = false, $arguments = array(), $ticket = null )
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; use PhpAmqpLib\Message\AMQPMessage; //获取连接 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); //从连接中创建通道 $channel = $connection->channel(); //声明队列 $channel->queue_declare('task_queue123', false, true, false, false); for($i=0;$i<=50;$i++){ $data = implode(' ', array_slice($argv, 1)); if (empty($data)) { $data = "Hello World! ".$i; } $msg = new AMQPMessage($data); $channel->basic_publish($msg, '', 'task_queue123'); } $channel->close(); $connection->close();
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); //声明队列 $channel->queue_declare('task_queue123', false, true, false, false); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; // sleep(2); //手动确认 $msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']); echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //手动确认 第四个参数为true $channel->basic_consume('task_queue123', '', false, false, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); //声明队列 $channel->queue_declare('task_queue123', false, true, false, false); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; sleep(1); echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //自动确认 第四个参数为true $channel->basic_consume('task_queue123', '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; use PhpAmqpLib\Message\AMQPMessage; //获取连接 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); //从连接中创建通道 $channel = $connection->channel(); $channel->exchange_declare('logs', 'fanout', false, false, false); for($i=0;$i<=50;$i++){ $data = implode(' ', array_slice($argv, 1)); if (empty($data)) { $data = " hi ,Hello World! ".$i; } $msg = new AMQPMessage($data); $channel->basic_publish($msg, 'logs'); } $channel->close(); $connection->close();
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->exchange_declare('logs', 'fanout', false, false, false); list($queue_name, ,) = $channel->queue_declare("", false, false, true, false); $channel->queue_bind($queue_name, 'logs'); //声明队列 echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //手动确认 第四个参数为true $channel->basic_consume($queue_name, '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->exchange_declare('logs', 'fanout', false, false, false); list($queue_name, ,) = $channel->queue_declare("", false, false, true, false); $channel->queue_bind($queue_name, 'logs'); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; // sleep(2); //手动确认 echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //手动确认 第四个参数为true $channel->basic_consume($queue_name, '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; use PhpAmqpLib\Message\AMQPMessage; //获取连接 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); //从连接中创建通道 $channel = $connection->channel(); $channel->exchange_declare('direct_logs', 'direct', false, false, false); for($i=0;$i<=50;$i++){ $data = implode(' ', array_slice($argv, 1)); if (empty($data)) { $data = " hi ,Hello World! ".$i; } $msg = new AMQPMessage($data); $channel->basic_publish($msg, 'direct_logs','info'); } $channel->close(); $connection->close();
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->exchange_declare('direct_logs', 'direct', false, false, false); list($queue_name, ,) = $channel->queue_declare("", false, false, true, false); $channel->queue_bind($queue_name, 'direct_logs','info'); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //手动确认 第四个参数为true $channel->basic_consume($queue_name, '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->exchange_declare('direct_logs', 'direct', false, false, false); list($queue_name, ,) = $channel->queue_declare("", false, false, true, false); $channel->queue_bind($queue_name, 'direct_logs','info.hello'); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //手动确认 第四个参数为true $channel->basic_consume($queue_name, '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; use PhpAmqpLib\Message\AMQPMessage; //获取连接 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); //从连接中创建通道 $channel = $connection->channel(); $channel->exchange_declare('topic_logs', 'topic', false, false, false); for($i=0;$i<=5;$i++){ $data = implode(' ', array_slice($argv, 1)); if (empty($data)) { $data = " info.hello ".$i; } $msg = new AMQPMessage($data); $channel->basic_publish($msg, 'topic_logs','info.hello'); } for($i=0;$i<=5;$i++){ $data = implode(' ', array_slice($argv, 1)); if (empty($data)) { $data = " info.helloword ".$i; } $msg = new AMQPMessage($data); $channel->basic_publish($msg, 'topic_logs','info.helloword'); } $channel->close(); $connection->close();
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->exchange_declare('topic_logs', 'topic', false, false, false); list($queue_name, ,) = $channel->queue_declare("", false, false, true, false); $channel->queue_bind($queue_name, 'topic_logs','info.#'); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //手动确认 第四个参数为true $channel->basic_consume($queue_name, '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->exchange_declare('topic_logs', 'topic', false, false, false); list($queue_name, ,) = $channel->queue_declare("", false, false, true, false); $channel->queue_bind($queue_name, 'topic_logs','info.hello'); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; echo " [x] Done\n"; }; $channel->basic_qos(null, 1, null); //手动确认 第四个参数为true $channel->basic_consume($queue_name, '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
消息的可靠性投递方案:
1,中小型公司
1,消息入库 status=0,2,生产端发送消息到booker,3,ack异步确认 修改db消息状态
如果异步确认出现rpc中断 闪断现象 可以用定时任务弥补,查找状态为0的 5分钟还没消费的,重新发送
2,大型公司 高并发
可靠性投递下订单后发送消息1,接着发送延时消息2 (补偿消息),消息1处理后入库(mysql),处理消息2判断db是否有这条消息 是否已经处理ok,如果
消息1处理失败,则rpc调用,重新请求service
什么是幂等性?
可能你要对一件事情执行一个操作,要操作一百次 一千次,最终我们执行结果要相同的,那样就是一个幂等性。
比如我们执行一条sql,无论我执行多少次 这个结果都是相同的,这个就是幂等性的保障。
高并发情况下,大量消息 投递 和 消费,可能会出现重复投递,网络原因导致闪断,导致booker重发消息,这个时候我们不去做幂等就会出现重复消费
,我们消息做幂等,就为了让消息不被重复消费,即使我们收到很多同样的消息,我们也只会对这条消息消费一次,可能我们代码会跑多次,但是我们消息
只会执行这一步操作。
指纹码:时间搓或者外部 内部的唯一码;
利用hash算法 分库分表,分摊流量压力
sexex
如果要入库的话,怎么保证数据一致性? redis有事务,mysql也有事务 怎么去处理?会存在redis写成功,mysql写失败。。。。。。思考。。。。
利用加version版本号 来仿造乐观锁,当第二个人进来后 version=1 已经查询不到了,保证数据唯一
boolean setSuccess = redis.setnx(request.serializeToString(),"");//原子操作 if(setSuccess){ doBusiness(); //执行业务 }else{ doNothing(); //什么都不做 }
1,conrim机制:
正常情况下,如果消息经过交换器进入队列就可以完成消息的持久化,但如果消息在没有到达broker之前出现意外,那就造成消息丢失,有没有办法可以解决这个问题?
RabbitMQ有两种方式来解决这个问题:
- 通过AMQP提供的事务机制实现;
- 使用发送者确认模式实现;
一、事务使用
事务的实现主要是对信道(Channel)的设置,主要的方法有三个:
-
channel.txSelect()声明启动事务模式;
-
channel.txComment()提交事务;
-
channel.txRollback()回滚事务;
从上面的可以看出事务都是以tx开头的,tx应该是transaction extend(事务扩展模块)的缩写,如果有准确的解释欢迎在博客下留言。
我们来看具体的代码实现:
// 创建连接 ConnectionFactory factory = new ConnectionFactory(); factory.setUsername(config.UserName); factory.setPassword(config.Password); factory.setVirtualHost(config.VHost); factory.setHost(config.Host); factory.setPort(config.Port); Connection conn = factory.newConnection(); // 创建信道 Channel channel = conn.createChannel(); // 声明队列 channel.queueDeclare(_queueName, true, false, false, null); String message = String.format("时间 => %s", new Date().getTime()); try { channel.txSelect(); // 声明事务 // 发送消息 channel.basicPublish("", _queueName, MessageProperties.PERSISTENT_TEXT_PLAIN, message.getBytes("UTF-8")); channel.txCommit(); // 提交事务 } catch (Exception e) { channel.txRollback(); } finally { channel.close(); conn.close(); }
注意:用户需把config.xx配置成自己Rabbit的信息。
从上面的代码我们可以看出,在发送消息之前的代码和之前介绍的都是一样的,只是在发送消息之前,需要声明channel为事务模式,提交或者回滚事务即可。
了解了事务的实现之后,那么事务究竟是怎么执行的,让我们来使用wireshark抓个包看看,如图所示:
输入ip.addr==rabbitip && amqp查看客户端和rabbit之间的通讯,可以看到交互流程:
- 客户端发送给服务器Tx.Select(开启事务模式)
- 服务器端返回Tx.Select-Ok(开启事务模式ok)
- 推送消息
- 客户端发送给事务提交Tx.Commit
- 服务器端返回Tx.Commit-Ok
以上就完成了事务的交互流程,如果其中任意一个环节出现问题,就会抛出IoException移除,这样用户就可以拦截异常进行事务回滚,或决定要不要重复消息。
那么,既然已经有事务了,没什么还要使用发送方确认模式呢,原因是因为事务的性能是非常差的。事务性能测试:
事务模式,结果如下:
- 事务模式,发送1w条数据,执行花费时间:14197s
- 事务模式,发送1w条数据,执行花费时间:13597s
- 事务模式,发送1w条数据,执行花费时间:14216s
非事务模式,结果如下:
- 非事务模式,发送1w条数据,执行花费时间:101s
- 非事务模式,发送1w条数据,执行花费时间:77s
- 非事务模式,发送1w条数据,执行花费时间:106s
从上面可以看出,非事务模式的性能是事务模式的性能高149倍,我的电脑测试是这样的结果,不同的电脑配置略有差异,但结论是一样的,事务模式的性能要差很多,那有没有既能保证消息的
可靠性又能兼顾性能的解决方案呢?那就是接下来要讲的Confirm发送方确认模式。
扩展知识
我们知道,消费者可以使用消息自动或手动发送来确认消费消息,那如果我们在消费者模式中使用事务(当然如果使用了手动确认消息,完全用不到事务的),会发生什么呢?
消费者模式使用事务
假设消费者模式中使用了事务,并且在消息确认之后进行了事务回滚,那么RabbitMQ会产生什么样的变化?
结果分为两种情况:
- autoAck=false手动应对的时候是支持事务的,也就是说即使你已经手动确认了消息已经收到了,但在确认消息会等事务的返回解决之后,在做决定是确认消息还是重新放回队列,如果你
- 手动确认现在之后,又回滚了事务,那么已事务回滚为主,此条消息会重新放回队列;
- autoAck=true如果自定确认为true的情况是不支持事务的,也就是说你即使在收到消息之后在回滚事务也是于事无补的,队列已经把消息移除了;
2、Confirm发送方确认模式
Confirm发送方确认模式使用和事务类似,也是通过设置Channel进行发送方确认的。
Confirm的三种实现方式:
方式一:channel.waitForConfirms()普通发送方确认模式;
方式二:channel.waitForConfirmsOrDie()批量确认模式;
方式三:channel.addConfirmListener()异步监听发送方确认模式;
方式一:普通Confirm模式
// 创建连接 ConnectionFactory factory = new ConnectionFactory(); factory.setUsername(config.UserName); factory.setPassword(config.Password); factory.setVirtualHost(config.VHost); factory.setHost(config.Host); factory.setPort(config.Port); Connection conn = factory.newConnection(); // 创建信道 Channel channel = conn.createChannel(); // 声明队列 channel.queueDeclare(config.QueueName, false, false, false, null); // 开启发送方确认模式 channel.confirmSelect(); String message = String.format("时间 => %s", new Date().getTime()); channel.basicPublish("", config.QueueName, null, message.getBytes("UTF-8")); if (channel.waitForConfirms()) { System.out.println("消息发送成功" ); }
看代码可以知道,我们只需要在推送消息之前,channel.confirmSelect()声明开启发送方确认模式,再使用channel.waitForConfirms()等待消息被服务器确认即可。
方式二:批量Confirm模式
// 创建连接 ConnectionFactory factory = new ConnectionFactory(); factory.setUsername(config.UserName); factory.setPassword(config.Password); factory.setVirtualHost(config.VHost); factory.setHost(config.Host); factory.setPort(config.Port); Connection conn = factory.newConnection(); // 创建信道 Channel channel = conn.createChannel(); // 声明队列 channel.queueDeclare(config.QueueName, false, false, false, null); // 开启发送方确认模式 channel.confirmSelect(); for (int i = 0; i < 10; i++) { String message = String.format("时间 => %s", new Date().getTime()); channel.basicPublish("", config.QueueName, null, message.getBytes("UTF-8")); } channel.waitForConfirmsOrDie(); //直到所有信息都发布,只要有一个未确认就会IOException System.out.println("全部执行完成");
以上代码可以看出来channel.waitForConfirmsOrDie(),使用同步方式等所有的消息发送之后才会执行后面代码,只要有一个消息未被确认就会抛出IOException异常。
方式三:异步Confirm模式
// 创建连接 ConnectionFactory factory = new ConnectionFactory(); factory.setUsername(config.UserName); factory.setPassword(config.Password); factory.setVirtualHost(config.VHost); factory.setHost(config.Host); factory.setPort(config.Port); Connection conn = factory.newConnection(); // 创建信道 Channel channel = conn.createChannel(); // 声明队列 channel.queueDeclare(config.QueueName, false, false, false, null); // 开启发送方确认模式 channel.confirmSelect(); for (int i = 0; i < 10; i++) { String message = String.format("时间 => %s", new Date().getTime()); channel.basicPublish("", config.QueueName, null, message.getBytes("UTF-8")); } //异步监听确认和未确认的消息 channel.addConfirmListener(new ConfirmListener() { @Override public void handleNack(long deliveryTag, boolean multiple) throws IOException { System.out.println("未确认消息,标识:" + deliveryTag); } @Override public void handleAck(long deliveryTag, boolean multiple) throws IOException { System.out.println(String.format("已确认消息,标识:%d,多个消息:%b", deliveryTag, multiple)); } });
异步模式的优点,就是执行效率高,不需要等待消息执行完,只需要监听消息即可,以上异步返回的信息如下:
以看出,代码是异步执行的,消息确认有可能是批量确认的,是否批量确认在于返回的multiple的参数,此参数为bool值,如果true表示批量执行了deliveryTag这个值以前的所有消息,如果为false的话表示单条确认。
Confirm性能测试
测试前提:与事务一样,我们发送1w条消息。
方式一:Confirm普通模式
- 执行花费时间:2253s
- 执行花费时间:2018s
- 执行花费时间:2043s
方式二:Confirm批量模式
- 执行花费时间:1576s
- 执行花费时间:1400s
- 执行花费时间:1374s
方式三:Confirm异步监听方式
- 执行花费时间:1498s
- 执行花费时间:1368s
- 执行花费时间:1363s
总结
综合总体测试情况来看:Confirm批量确定和Confirm异步模式性能相差不大,Confirm模式要比事务快10倍左右。
phpcode:
1 <?php 2 require_once __DIR__ . '/../vendor/autoload.php'; 3 use PhpAmqpLib\Connection\AMQPStreamConnection; 4 use PhpAmqpLib\Message\AMQPMessage; 5 6 7 $queue = "test_confirm"; 8 $exchange = "test_confirm"; 9 //获取连接 10 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); 11 //从连接中创建通道 12 $channel = $connection->channel(); 13 14 15 $channel->queue_declare($queue, false, true, false, false); 16 $channel->exchange_declare($exchange, 'direct', false, true, false); 17 $channel->queue_bind($queue, $exchange); 18 19 //在channel上开启确认模式 20 $channel->confirm_select(); 21 22 23 $channel->set_ack_handler( 24 function (AMQPMessage $message) { 25 echo "Message acked with content " . $message->body . PHP_EOL; 26 } 27 ); 28 $channel->set_nack_handler( 29 function (AMQPMessage $message) { 30 echo "Message nacked with content " . $message->body . PHP_EOL; 31 } 32 ); 33 34 35 $channel->wait_for_pending_acks(); 36 37 38 39 for($i=0;$i<=3;$i++){ 40 $data = implode(' ', array_slice($argv, 1)); 41 if (empty($data)) { 42 $data = " info.hello ".$i; 43 } 44 $msg = new AMQPMessage($data); 45 $channel->basic_publish($msg, $exchange,'info.hello'); 46 } 47 48 $channel->wait_for_pending_acks(); 49 50 //$channel->set_return_listener(function(){ 51 // echo 5; 52 //}); 53 54 55 $channel->close(); 56 $connection->close();
1 <?php 2 require_once __DIR__ . '/../vendor/autoload.php'; 3 use PhpAmqpLib\Connection\AMQPStreamConnection; 4 5 6 $queue = "test_confirm"; 7 $exchange = "test_confirm"; 8 9 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); 10 $channel = $connection->channel(); 11 $channel->queue_declare($queue, false, true, false, false); 12 $channel->exchange_declare($exchange, 'direct', false, true, false); 13 14 $channel->queue_bind($queue, $exchange,'info.hello'); 15 16 17 echo " [*] Waiting for messages. To exit press CTRL+C\n"; 18 19 $callback = function ($msg) { 20 echo ' [x] Received ', $msg->body, "\n"; 21 echo " [x] Done\n"; 22 }; 23 //$channel->basic_qos(null, 1, null); 24 25 26 //自动签收 第四个参数为true 27 $channel->basic_consume($queue, '', false, true, false, false, $callback); 28 29 while (count($channel->callbacks)) { 30 $channel->wait(); 31 }
生产者confirm确认消息模式,消费者自动签收
[root@localhost Conifrm]# php7 ./worker.php [*] Waiting for messages. To exit press CTRL+C [x] Received info.hello 0 [x] Done [x] Received info.hello 1 [x] Done [x] Received info.hello 2 [x] Done [x] Received info.hello 3 [x] Done [root@localhost Conifrm]# php7 ./task.php Message acked with content info.hello 0 Message acked with content info.hello 1 Message acked with content info.hello 2 Message acked with content info.hello 3
什么时候回触发NACK呢?
比如磁盘写满了,比如mq出现异常,queue容量到达上限;
或者消费者发送ack过程中出现网络闪断,需要通过定时任务抓取中间状态,重发或者补偿
3.return消息机制
return Listener用于处理一些不可路由的消息!
对一些不可达的消息第一时间进行监听
生产者通过指定一个exchange 和 routingkey 把消息送达到某个队列中去,然后消费者监听队列,进行消费处理。但是在某些情况下,如果我们在发送消息时,当前的exchange 不存在或者指定的routingkey路由不到,这个时候如果要监听这种不可达的消息,就要使用 return Listener。流程图如下所示
实现return消息机制
在基础API中有一个关键的配置项 Mandatory:如果为true,则监听器会接收到路由不可达的消息,然后进行后续处理,如果为false,则broker端自动删除该消息。
phpcode:
1 <?php 2 require_once __DIR__ . '/../vendor/autoload.php'; 3 use PhpAmqpLib\Connection\AMQPStreamConnection; 4 use PhpAmqpLib\Message\AMQPMessage; 5 6 7 $queue = "test_return"; 8 $exchange = "test_return"; 9 //获取连接 10 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); 11 //从连接中创建通道 12 $channel = $connection->channel(); 13 14 15 $channel->queue_declare($queue, false, true, false, false); 16 $channel->exchange_declare($exchange, 'topic', false, true, false); 17 $channel->queue_bind($queue, $exchange); 18 19 //在channel上开启确认模式 20 $channel->confirm_select(); 21 22 23 $channel->set_ack_handler( 24 function (AMQPMessage $message) { 25 echo "Message acked with content " . $message->body . PHP_EOL; 26 } 27 ); 28 $channel->set_nack_handler( 29 function (AMQPMessage $message) { 30 echo "Message nacked with content " . $message->body . PHP_EOL; 31 } 32 ); 33 34 35 $channel->wait_for_pending_acks(); 36 37 38 39 $wait = true; 40 $returnListener = function ( 41 $replyCode, 42 $replyText, 43 $exchange, 44 $routingKey, 45 $message 46 ) use ($wait) { 47 $GLOBALS['wait'] = false; 48 echo "return: ", 49 "replyCode:".$replyCode, "\n", 50 "replyText:".$replyText, "\n", 51 "exchange:".$exchange, "\n", 52 "routingKey:".$routingKey, "\n", 53 "message:".$message->body, "\n"; 54 }; 55 $channel->set_return_listener($returnListener); 56 57 for($i=0;$i<=3;$i++){ 58 $data = implode(' ', array_slice($argv, 1)); 59 if (empty($data)) { 60 $data = " msg info.hello ".$i; 61 } 62 $msg = new AMQPMessage($data); 63 # 默认Mandatory=false 则会监听路由不可达消息 booker会自动删除该消息 64 // $channel->basic_publish($msg, $exchange,'info.hello'); 65 66 # Mandatory=false 则会监听路由不可达消息 67 $channel->basic_publish($msg, $exchange,'aa.bb',true); 68 } 69 70 $channel->wait_for_pending_acks(); 71 72 //$channel->set_return_listener(function(){ 73 // echo 5; 74 //}); 75 76 77 78 while ($wait) { 79 $channel->wait(); 80 } 81 82 $channel->close(); 83 $connection->close();
<?php require_once __DIR__ . '/../vendor/autoload.php'; use PhpAmqpLib\Connection\AMQPStreamConnection; $queue = "test_return"; $exchange = "test_return"; $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); $channel = $connection->channel(); $channel->queue_declare($queue, false, true, false, false); $channel->exchange_declare($exchange, 'topic', false, true, false); $channel->queue_bind($queue, $exchange,'info.#'); echo " [*] Waiting for messages. To exit press CTRL+C\n"; $callback = function ($msg) { echo ' [x] Received ', $msg->body, "\n"; echo " [x] Done\n"; }; //$channel->basic_qos(null, 1, null); //自动签收 第四个参数为true $channel->basic_consume($queue, '', false, true, false, false, $callback); while (count($channel->callbacks)) { $channel->wait(); }
task:
[root@localhost return]# php7 ./task.php Message acked with content msg info.hello 0 Message acked with content msg info.hello 1 Message acked with content msg info.hello 2 Message acked with content msg info.hello 3 return: replyCode:312 replyText:NO_ROUTE exchange:test_return routingKey:aa.bb message: msg info.hello 0
4,消息端的手工ACK和NACK与重回队列
很多地方都没有说清楚怎么去手动ack,其实手动ack就是在当前channel里面调用basicAsk的方法,并传入当前消息的tagId就可以了。
注意如果抛异常或unack(并且requeue为true),消息会一直重新入队列,一不小心就会xxxxx一大堆消息不断重复~。
//消息的标识,false只确认当前一个消息收到,true确认所有consumer获得的消息 (正常消费)
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
//ack返回false,并重新回到队列,api里面解释得很清楚 (本地异常)
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, true);
//拒绝消息
channel.basicReject(message.getMessageProperties().getDeliveryTag(), true);
最后一个参数为true就重回队列
phpcode:
1 <?php 2 require_once __DIR__ . '/../vendor/autoload.php'; 3 use PhpAmqpLib\Connection\AMQPStreamConnection; 4 use PhpAmqpLib\Message\AMQPMessage; 5 6 7 $queue = "test_ack_queue"; 8 $exchange = "test_ack_queue"; 9 //获取连接 10 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); 11 //从连接中创建通道 12 $channel = $connection->channel(); 13 14 15 $channel->queue_declare($queue, false, true, false, false); 16 $channel->exchange_declare($exchange, 'topic', false, true, false); 17 $channel->queue_bind($queue, $exchange); 18 19 //在channel上开启确认模式 20 $channel->confirm_select(); 21 22 23 $channel->set_ack_handler( 24 function (AMQPMessage $message) { 25 echo "Message acked with content " . $message->body . PHP_EOL; 26 } 27 ); 28 $channel->set_nack_handler( 29 function (AMQPMessage $message) { 30 echo "Message nacked with content " . $message->body . PHP_EOL; 31 } 32 ); 33 34 35 $channel->wait_for_pending_acks(); 36 37 38 39 $wait = true; 40 $returnListener = function ($replyCode, $replyText, $exchange, $routingKey, $message) use ($wait) { 41 $GLOBALS['wait'] = false; 42 echo "return: ", 43 "replyCode:".$replyCode, "\n", 44 "replyText:".$replyText, "\n", 45 "exchange:".$exchange, "\n", 46 "routingKey:".$routingKey, "\n", 47 "message:".$message->body, "\n"; 48 }; 49 //路由不可达 监听 50 $channel->set_return_listener($returnListener); 51 52 for($i=0;$i<=0;$i++){ 53 $data = implode(' ', array_slice($argv, 1)); 54 if (empty($data)) { 55 $data = " msg info.hello ".$i; 56 } 57 $msg = new AMQPMessage($data); 58 59 60 $headers = new \PhpAmqpLib\Wire\AMQPTable(array( 61 'x-foo'=>'bar', 62 'table'=>['figuf', 'ghf'=>5, 5=>675], 63 // 'ack_type' => 'good', 64 'date' => new DateTime(), 65 )); 66 67 $headers->set('shortshort', -5, \PhpAmqpLib\Wire\AMQPTable::T_INT_SHORTSHORT); 68 $headers->set('short', -1024, \PhpAmqpLib\Wire\AMQPTable::T_INT_SHORT); 69 $headers->set('ack_type','good1'); 70 71 // var_dump($headers->getNativeData()); 72 echo PHP_EOL; 73 $msg->set('application_headers', $headers); 74 75 # 默认Mandatory=false 则会监听路由不可达消息 booker会自动删除该消息 76 $channel->basic_publish($msg, $exchange,'info.hello',true); 77 } 78 79 $channel->wait_for_pending_acks(); 80 81 82 83 84 85 while ($wait) { 86 $channel->wait(); 87 } 88 89 $channel->close(); 90 $connection->close();
1 <?php 2 require_once __DIR__ . '/../vendor/autoload.php'; 3 use PhpAmqpLib\Connection\AMQPStreamConnection; 4 use \PhpAmqpLib\Message\AMQPMessage; 5 6 $queue = "test_ack_queue"; 7 $exchange = "test_ack_queue"; 8 9 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); 10 $channel = $connection->channel(); 11 $channel->queue_declare($queue, false, true, false, false); 12 $channel->exchange_declare($exchange, 'topic', false, true, false); 13 14 $channel->queue_bind($queue, $exchange,'info.#'); 15 16 17 echo " [*] Waiting for messages. To exit press CTRL+C\n"; 18 19 $callback = function ($msg) { 20 echo ' [x] Received ', $msg->body, "\n"; 21 echo " [x] Done\n"; 22 }; 23 24 $callback = function (AMQPMessage $message) { 25 echo PHP_EOL . ' [x] ', $message->delivery_info['routing_key'], ':', $message->body, "\n"; 26 echo 'Message headers follows' . PHP_EOL; 27 $header = $message->get('application_headers')->getNativeData(); 28 // var_dump($header); 29 echo PHP_EOL; 30 31 32 if ($header['ack_type'] == 'good') { 33 $message->delivery_info['channel']->basic_ack($message->delivery_info['delivery_tag']); 34 } else { 35 sleep(2); 36 //第二个参数,批量确认,第三个参数:重回队列 37 $message->delivery_info['channel']->basic_nack($message->delivery_info['delivery_tag'],false,true); 38 } 39 40 // Send a message with the string "quit" to cancel the consumer. 41 if ($message->body === 'quit') { 42 $message->delivery_info['channel']->basic_cancel($message->delivery_info['consumer_tag']); 43 } 44 45 }; 46 47 //限流每次只接一个消息 48 $channel->basic_qos(null, 1, null); 49 50 //自动签收 第四个参数为true no_ack=false 51 $channel->basic_consume($queue, '', false, false, false, false, $callback); 52 53 while (count($channel->callbacks)) { 54 $channel->wait(); 55 }
配注:
生产者设置头部 $headers->set('ack_type','good1');
消费端通过判断头部变量ack_type
确认签收:
$message->delivery_info['channel']->basic_ack($message->delivery_info['delivery_tag']);
不签收重回队列:
sleep(2); //第二个参数,批量确认,第三个参数:重回队列 $message->delivery_info['channel']->basic_nack($message->delivery_info['delivery_tag'],false,true);
打印结果:
[root@localhost ack]# php7 ./task.php Message acked with content msg info.hello 0
消费端没没隔2秒会消费一次,因为没签收 也设置了重回队列
[root@localhost ack]# php7 ./worker.php [*] Waiting for messages. To exit press CTRL+C [x] info.hello: msg info.hello 0 Message headers follows [x] info.hello: msg info.hello 0 Message headers follows [x] info.hello: msg info.hello 0 Message headers follows [x] info.hello: msg info.hello 0 Message headers follows
5,TTL
两种TTL:如果两种都设置了,按照TTL小的那个处理
1.通过队列属性设置,通过队列发送出去的消息都遵循这个TTL;
2.通过对消息本身单独设置
第一种,一旦消息过期直接丢弃;第二种,即使消息过期,也不一定立刻丢弃,因为只有当消息被投递的时候,才能判断该消息是否过期。
设置队列TTL:
如果不设置TTL,则表示此消息不会过期,如果将TTL设置为0,则表示除非此时可以直接将消息投递到消费者,否则该消息立即被丢弃。
ConnectionFactory factory = new ConnectionFactory(); factory.HostName = IP_ADDRESS; factory.Port = PORT; factory.UserName = USER_NAME; factory.Password = PASSWORD; con = factory.CreateConnection(); channel = con.CreateModel(); channel.ExchangeDeclare(EXCHANGE_NAME, "topic", true, false, null); Dictionary<string, object> agres = new Dictionary<string, object>(); //消息TTL01.Queue设置 agres.Add("x-message-ttl", 6000); //队列TTL设置:1800000ms agres.Add("x-expires", 1800000); channel.QueueDeclare(QUEUE_NAME, true, false, false, agres); channel.QueueBind(QUEUE_NAME, EXCHANGE_NAME, BINDING_KEY, null);//channel.ExchangeBind() string message = "Hello Word!"; var body = Encoding.UTF8.GetBytes(message); var properties = channel.CreateBasicProperties(); properties.Persistent = true;
设置消息TTL:
ConnectionFactory factory = new ConnectionFactory(); factory.HostName = IP_ADDRESS; factory.Port = PORT; factory.UserName = USER_NAME; factory.Password = PASSWORD; con = factory.CreateConnection(); channel = con.CreateModel(); channel.ExchangeDeclare(EXCHANGE_NAME, "topic", true, false, null); //Dictionary<string, object> agres = new Dictionary<string, object>(); ////消息TTL01.Queue设置 //agres.Add("x-message-ttl", 6000); ////队列TTL设置:1800000ms //agres.Add("x-expires", 1800000); //channel.QueueDeclare(QUEUE_NAME, true, false, false, agres); channel.QueueBind(QUEUE_NAME, EXCHANGE_NAME, BINDING_KEY, null);//channel.ExchangeBind() string message = "Hello Word!"; var body = Encoding.UTF8.GetBytes(message); var properties = channel.CreateBasicProperties(); properties.Persistent = true; //消息TTL02.每条消息设置设置,如果两个都设置了,则按照TTL小的那个 properties.Expiration = "6000";//TTL = 6000ms channel.BasicPublish(EXCHANGE_NAME, ROUTING_KEY, properties, body);
6,死信队列
DLX,Dead-Letter-Exchange
利用DLX,当消息在一个队列中变成死信(dead message)之后,它能被重新publish到另一个Exchange,这个Exchange就是Dlx
一 进入死信队列(进入死信的三种方式)
1.消息被拒绝(basic.reject or basic.nack)并且requeue=false
2.消息TTL过期过期时间
3.队列达到最大长度
DLX也是一下正常的Exchange同一般的Exchange没有区别,它能在任何的队列上被指定,实际上就是设置某个队列的属性,当这个队列中有死信时,RabbitMQ就会自动的将这个消息重新发布到设置的Exchange中去,进而被路由到另一个队列, publish可以监听这个队列中消息做相应的处理, 这个特性可以弥补R abbitMQ 3.0.0以前支持的immediate参数中的向publish确认的功能。
如果消息在队列中到达TTL,将被丢弃。这时候,消息变成死信(dead letter).过期是导致死信的原因之一,在RabbitMQ中,以下情况都会产生死信:
- 消息过期
- 消息被消费着拒绝(reject/nack),并且设置requeue参数为false
- 队列到达最大长度
消息在队列中变成死信默认将被丢弃,为了处理死信,可以使用死信交换器(DLX)。
死信交换器可以认为是队列的备胎,当队列中产生死信时,死信被发送到死信交换器,由死信交换器重新路由到与之绑定的队列,这些队列被成为死信队列。
声明队列时,可以通过x-dead-letter-exchange参数设置该队列的死信交换器,也可以通过policy方式设定队列的死信交换器。
Map<String,Object> params = new HashMap<String, Object>(); params.put("x-dead-letter-exchange","dlx-exchange"); channel.queueDeclare("myqueue",false,false,false,params);
这样,当myquue队列中产生死信时,死信将被发送到dlx-exchange交换器,与它重新路由。
消息到路由键是后生产者发送是设置到,在死信被发送到死信交换器时,我们有机会修改消息到路由键。在声明队列是,指定x-dead-letter-routing-key参数即可。
params.put("x-dead-letter-routing-key","deadKey");
这样,当死信被发送到死信交换器时,它到路由键变为deadKey,后续在死信交换器中将根据该路由键进行路由。通过这种在队列上为死信统一更新路由键到方式,使得在某些
情况下可以统一将死信路由到指定队列,方便对死信统一处理。
7,消费端的限流
假如单个生产者一分钟生产几百条数据,但是单个消费端一分钟只能消费60条数据,这个时候生产端和消费端肯定是不平衡的;高并发情况下,生产端我们没法做限制,消费端肯定要做消峰,
这个时候我们消费端就要做限流操作,目的就是为了让消费端更稳定,不然超出了最大负载 可能会导致消费端的资源耗尽。
在我们mq中有两种签收模式,自动,手动;在高并发情况下,我们一定不能设置为自动签收,在工作没有人会自动签收;
签收设置为手工确定,只要你消息没有被确定前,是不会有新的消息到达consume端的,这是mq的机制,这也是给消费者减压减负;
消费者端: BasicQos( prefetchSize:0 #消息大小限制,一般为0,不限制 perfetchCount:1 #一次最多处理多少消息,一般设置为1。会告诉mq不要同时给一个消费者推送多余N个消息,即一旦有N个消息还没有ack 则该consumer将block掉,直到有消息ack 简单来说:就是不要一次给我太多消息 我扛不住 global:true/false # 是否将上面设置应用于channel,就是上面限制是channel级别还是consumer级别 限流策略在什么时候应用,true:channel通道上限制 false:消费者上做限制 ) 注意:prefetchSize和global这两项,mq还没有实现,暂时不研究,prefetch_count在no_ask=false情况下生效,即在自动应答的情况下这两个值不生效
消费者添加:
$channel->basic_qos(null, 1, null);
1 <?php 2 require_once __DIR__ . '/../vendor/autoload.php'; 3 use PhpAmqpLib\Connection\AMQPStreamConnection; 4 use PhpAmqpLib\Message\AMQPMessage; 5 6 7 $queue = "qos_queue"; 8 $exchange = "qos_queue"; 9 //获取连接 10 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); 11 //从连接中创建通道 12 $channel = $connection->channel(); 13 14 15 $channel->queue_declare($queue, false, true, false, false); 16 $channel->exchange_declare($exchange, 'topic', false, true, false); 17 $channel->queue_bind($queue, $exchange); 18 19 //在channel上开启确认模式 20 $channel->confirm_select(); 21 22 23 $channel->set_ack_handler( 24 function (AMQPMessage $message) { 25 echo "Message acked with content " . $message->body . PHP_EOL; 26 } 27 ); 28 $channel->set_nack_handler( 29 function (AMQPMessage $message) { 30 echo "Message nacked with content " . $message->body . PHP_EOL; 31 } 32 ); 33 34 35 $channel->wait_for_pending_acks(); 36 37 38 39 $wait = true; 40 $returnListener = function ( 41 $replyCode, 42 $replyText, 43 $exchange, 44 $routingKey, 45 $message 46 ) use ($wait) { 47 $GLOBALS['wait'] = false; 48 echo "return: ", 49 "replyCode:".$replyCode, "\n", 50 "replyText:".$replyText, "\n", 51 "exchange:".$exchange, "\n", 52 "routingKey:".$routingKey, "\n", 53 "message:".$message->body, "\n"; 54 }; 55 $channel->set_return_listener($returnListener); 56 57 for($i=0;$i<=3;$i++){ 58 $data = implode(' ', array_slice($argv, 1)); 59 if (empty($data)) { 60 $data = " msg info.hello ".$i; 61 } 62 $msg = new AMQPMessage($data); 63 # 默认Mandatory=false 则会监听路由不可达消息 booker会自动删除该消息 64 $channel->basic_publish($msg, $exchange,'info.hello',true); 65 66 # Mandatory=false 则会监听路由不可达消息 67 // $channel->basic_publish($msg, $exchange,'aa.bb',true); 68 } 69 70 $channel->wait_for_pending_acks(); 71 72 //$channel->set_return_listener(function(){ 73 // echo 5; 74 //}); 75 76 77 78 while ($wait) { 79 $channel->wait(); 80 } 81 82 $channel->close(); 83 $connection->close();
1 <?php 2 require_once __DIR__ . '/../vendor/autoload.php'; 3 use PhpAmqpLib\Connection\AMQPStreamConnection; 4 5 6 $queue = "qos_queue"; 7 $exchange = "qos_queue"; 8 9 $connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest'); 10 $channel = $connection->channel(); 11 $channel->queue_declare($queue, false, true, false, false); 12 $channel->exchange_declare($exchange, 'topic', false, true, false); 13 14 $channel->queue_bind($queue, $exchange,'info.#'); 15 16 17 echo " [*] Waiting for messages. To exit press CTRL+C\n"; 18 19 $callback = function ($msg) { 20 echo ' [x] Received ', $msg->body, "\n"; 21 echo " [x] Done\n"; 22 }; 23 $channel->basic_qos(null, 1, null); 24 25 //自动签收 第四个参数为true 26 $channel->basic_consume($queue, '', false, true, false, false, $callback); 27 28 while (count($channel->callbacks)) { 29 $channel->wait(); 30 }
消息持久化:
如果我们希望即使在RabbitMQ服务重启的情况下,也不会丢失消息,我们可以将Queue与Message都设置为可持久化的(durable),这样可以保证绝大部分情况下我们的RabbitMQ消息不会丢失。当然还是会有一些小概率事件会导致消息丢失。
持久化队列:我们就hello队列持久化
在声明队列名称时,持久化队列,生产端和消费端都要
channel.queue_declare(queue='hello', durable=True)
我们重复上面的操作,但是给hello队列做持久化,而hello1不做,并重启rabbitmq
可以看到重启后,hello队列还在,hello1队列消失了,但是原本hello中的一条消息也没有保存下来。所以在这边我们仅仅做到了消息队列的持久化,还没有做消息持久化。
消息持久化:
我们刚才实现了在rabbitmq崩溃的情况下,就队列本身保存下来,重启后队列还在。接下来我们要将消息也保存下来,即消息的持久化
channel.basic_publish(exchange='', routing_key='hello', body='hello', properties=pika.BasicProperties( delivery_mode=2, # make message persistent )) # 增加properties,这个properties 就是消费端 callback函数中的properties # delivery_mode = 2 持久化消息
- 队列持久化需要在声明队列时添加参数 durable=True,这样在rabbitmq崩溃时也能保存队列
- 仅仅使用durable=True ,只能持久化队列,不能持久化消息
- 消息持久化需要在消息生成时,添加参数 properties=pika.BasicProperties(delivery_mode=2)
更多demo参考
https://github.com/sunlongv520/php-amqplib/tree/master/demo
golang使用rabbitmq监听消息 并实现rbmq断开断线重连
本文来自博客园,作者:孙龙-程序员,转载请注明原文链接:https://www.cnblogs.com/sunlong88/articles/9742020.html