MQ
初始MQ
同步通讯的优缺点
微服务间基于Feign的调用就属于同步方式,存在一些问题:
- 耦合度高:每次加入新的需求,都要修改原来的代码
- 性能下降: 调用者需要等待服务提供者响应,如果调用链过长则响应时间等于每次调用的时间之和
- 资源浪费: 调用链中每个服务在等待响应过程中,不能释放请求占用的资源,高并发场景下会嫉妒浪费系统资源
- 级联失败: 如果服务提供者出现问题,所有调用方都会跟着出问题,如同多米诺骨牌一样,迅速导致整个微服务群故障
异步同步的优缺点
异步调用常见实现就是时间驱动模式
异步通信的优点:
- 耦合度低
- 吞吐量提升
- 故障隔离
- 流量削峰
异步通信的缺点:
- 依赖于Broker的可靠性、安全性、吞吐能力
- 架构复杂了,业务没有明显的流程线,不好追踪管理
MQ 主要应用场景
- 应用解耦(异步)
系统的耦合性越高,容错性就越低。以电商应用为例,用户创建订单后,如果耦合调用库存系统、物流系统、支付系统,任何一个子系统出现故障或者因为升级等原因暂时不可用,都会造成下单操作异常,影响用户使用体验
使用消息队列解耦合,系统的耦合性就会提高了。比如物流系统发生故障,需要几分钟才能来修复,在这段时间内,物流系统要处理的数据被缓存在消息队列中,用户的下单操作正常完成。当物流系统恢复后,补充处理存在消息队列中的订单消息即可,终端系统感知不到物流系统发生过几分钟故障
- 流量削峰
应用系统如果遇到系统请求流量的瞬间猛增,有可能会将系统压垮。有了消息队列可以将大量请求缓存起来,分散很长一段时间处理,这样可以大大提高系统的稳定性和用户体验
一般情况,为了保证系统的稳定性,如果系统负载超过阈值,就会阻止用户请求,这会影响用户体验,而如果使用消息队列将请求缓存起来,等待系统处理完毕后通知用户下单完毕,这样比不能下单体验要好。
处于经济考量目的:
业务系统正常时段的QPS如果是1000,流量最高峰是10000,为了应对流量高峰配置高性能的服务器显然不划算,这是可以使用消息队列对峰值流量削峰
- 数据分发
通过消息队列可以让数据在多个系统之间进行流通,数据的产生方不需要关心谁来使用数据,只需要将数据发送到消息队列,数据使用方直接在消息队列中直接获取数据即可
MQ产品对比
MQ(MessageQueue) ,中文是消息队列,字面来看就是存放消息的队列。也就是事件驱动架构中的Broker。
安装
方式一:docker pull rabbitmq:3-management
方式二:下载好tar,上传到服务器
docker load -i mq.tar
运行MQ容器:
docker run \
-e RABBITMQ_DEFAULT_USER=root \
-e RABBITMQ_DEFAULT_PASS=123456 \
--name mq \
--hostname mq1 \
-p 15672:15672 \
-p 5672:5672 \
-d \
rabbitmq:3-management
-p 15672:15672
: 控制台UI界面端口
-p 5672:5672
:消息通信的端口
RabbitMQ 快速入门
RabbitMQ 的结构和概念:
- channel: 操作MQ的工具
- exchangef: 路由消息到队列中
- queue:缓存消息
- virtual host: 虚拟主机,是对queue、exchange等资源的逻辑分组
常见的消息模型:
- 基本消息队列(BasicQueue)
- 工作消息队列(WorkQueue)
- 发布订阅(Publish、Subscribe),又根据交换机类型不同分为三种:
- Fanout Exchange:广播
- Direct Exchange: 路由
- Topic ExchangeL:主题
helloworld 案例 - 简单队列模型
- publisher:
- queue: 消息队列,负责接收并缓存消息
- consumer: 订阅队列,处理队列中的消息
publisher:
package cn.itcast.mq.helloworld;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import org.junit.Test;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
public class PublisherTest {
@Test
public void testSendMessage() throws IOException, TimeoutException {
// 1.建立连接
ConnectionFactory factory = new ConnectionFactory();
// 1.1.设置连接参数,分别是:主机名、端口号、vhost、用户名、密码
factory.setHost("192.168.184.152");
factory.setPort(5672);
factory.setVirtualHost("/");
factory.setUsername("root");
factory.setPassword("123456");
// 1.2.建立连接
Connection connection = factory.newConnection();
// 2.创建通道Channel
Channel channel = connection.createChannel();
// 3.创建队列
String queueName = "simple.queue";
channel.queueDeclare(queueName, false, false, false, null);
// 4.发送消息
String message = "hello, rabbitmq!";
channel.basicPublish("", queueName, null, message.getBytes());
System.out.println("发送消息成功:【" + message + "】");
// 5.关闭通道和连接
channel.close();
connection.close();
}
}
查看连接:
查看通道:
查看队列:
查看具体消息:
consumer:
package cn.itcast.mq.helloworld;
import com.rabbitmq.client.*;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
public class ConsumerTest {
public static void main(String[] args) throws IOException, TimeoutException {
// 1.建立连接
ConnectionFactory factory = new ConnectionFactory();
// 1.1.设置连接参数,分别是:主机名、端口号、vhost、用户名、密码
factory.setHost("192.168.184.152");
factory.setPort(5672);
factory.setVirtualHost("/");
factory.setUsername("root");
factory.setPassword("123456");
// 1.2.建立连接
Connection connection = factory.newConnection();
// 2.创建通道Channel
Channel channel = connection.createChannel();
// 3.创建队列
String queueName = "simple.queue";
channel.queueDeclare(queueName, false, false, false, null);
// 4.订阅消息
channel.basicConsume(queueName, true, new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope,
AMQP.BasicProperties properties, byte[] body) throws IOException {
// 5.处理消息
String message = new String(body);
System.out.println("接收到消息:【" + message + "】");
}
});
System.out.println("等待接收消息。。。。");
}
}
接收消息成功:
SpringAMQP
什么是AMQP?
Advanced Message Queuing Protocol,是用于在应用程序或之间传递业务消息的开放标准。该协议与语言和平台无关,更符合微服务中独立性的要求
什么是SpringAMQP?
Spring AMQP 是基于AMQP协议定义的一套API规范,提供了模板来发送和接收消息。包含两部分,其中spring-amqp是基础抽象,spring-rabbit是底层的默认实现
官方文档:https://spring.io/projects/spring-amqp
简单队列模型: 基于SpringAMQP 发送消息:
1.引入依赖:
<!--AMQP依赖,包含RabbitMQ-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
2.application.yaml 中配置MQ信息:
spring:
rabbitmq:
host: 192.168.184.152
port: 5672
username: root
password: 123456
virtual-host: /
3.基于RabbitTamplete 发送消息:
@RunWith(SpringRunner.class)
@SpringBootTest
public class SpringTest {
@Autowired
private RabbitTemplate rabbitTemplate;
@Test
public void testSendMesToRabbit(){
String message = "hello chuangzhou";
String queue = "simple.queue";
rabbitTemplate.convertAndSend(queue,message);
}
}
简单队列模型: 基于SpringAMQP 接收消息:
1.引入依赖
2.application.yaml 配置MQ信息
3.新建一个监听器类
@Component
public class RabbitListenerBean {
@RabbitListener(queues = "simple.queue")
public void listenerRabbitMQ(String msg){
System.out.println("监听到rabbit mq:" + msg);
}
}
WorkQueue 模型
Work queue, 工作队列,可以提高消息处理速度,避免队列消息堆积
案例:模拟WorkQueue 实现一个队列绑定多个消费者
1.publisher 服务中每秒产生50条消息,发送到semple.queue
2.consumer服务中定义两个消息监听者
3.消费者1 每秒处理50条消息,消费者二每秒处理10条消息
发送者:
@Test
public void testSendMes2WorkQueue() throws InterruptedException {
String message = "hello chuangzhou_";
String queue = "simple.queue";
for (int i = 0; i < 50; i++) { //1s 发送50个消息
rabbitTemplate.convertAndSend(queue,message + i);
Thread.sleep(20);
}
}
消费者:
@RabbitListener(queues = "simple.queue")
public void listenerWorkQueue1(String msg) throws InterruptedException {
System.out.println("消费者1,监听到消息:" + msg + "," + LocalTime.now());
Thread.sleep(20);
}
@RabbitListener(queues = "simple.queue")
public void listenerWorkQueue2(String msg) throws InterruptedException {
System.err.println("消费者2.......监听到消息:" + msg + "," + LocalTime.now());
Thread.sleep(100); //模拟处理消息比较慢
}
消费结果:
监听者2.......监听到消息:hello chuangzhou_0,21:40:11.035537500
监听者1,监听到消息:hello chuangzhou_1,21:40:11.050540400
监听者1,监听到消息:hello chuangzhou_3,21:40:11.094549400
监听者1,监听到消息:hello chuangzhou_5,21:40:11.138256400
监听者1,监听到消息:hello chuangzhou_7,21:40:11.181169900
监听者1,监听到消息:hello chuangzhou_9,21:40:11.226042400
监听者2.......监听到消息:hello chuangzhou_2,21:40:11.238651
监听者1,监听到消息:hello chuangzhou_11,21:40:11.269784400
监听者1,监听到消息:hello chuangzhou_13,21:40:11.313025600
监听者1,监听到消息:hello chuangzhou_15,21:40:11.357462100
监听者1,监听到消息:hello chuangzhou_17,21:40:11.401029800
监听者2.......监听到消息:hello chuangzhou_4,21:40:11.440045400
监听者1,监听到消息:hello chuangzhou_19,21:40:11.448047100
监听者1,监听到消息:hello chuangzhou_21,21:40:11.491060900
监听者1,监听到消息:hello chuangzhou_23,21:40:11.534028100
监听者1,监听到消息:hello chuangzhou_25,21:40:11.581554300
监听者1,监听到消息:hello chuangzhou_27,21:40:11.625249200
监听者2.......监听到消息:hello chuangzhou_6,21:40:11.642340400
监听者1,监听到消息:hello chuangzhou_29,21:40:11.668994900
监听者1,监听到消息:hello chuangzhou_31,21:40:11.710513700
监听者1,监听到消息:hello chuangzhou_33,21:40:11.755104800
监听者1,监听到消息:hello chuangzhou_35,21:40:11.797723600
监听者1,监听到消息:hello chuangzhou_37,21:40:11.840960400
监听者2.......监听到消息:hello chuangzhou_8,21:40:11.845519
监听者1,监听到消息:hello chuangzhou_39,21:40:11.886014100
监听者1,监听到消息:hello chuangzhou_41,21:40:11.930745400
监听者1,监听到消息:hello chuangzhou_43,21:40:11.972472400
监听者1,监听到消息:hello chuangzhou_45,21:40:12.018647400
监听者2.......监听到消息:hello chuangzhou_10,21:40:12.047705300
监听者1,监听到消息:hello chuangzhou_47,21:40:12.062546800
监听者1,监听到消息:hello chuangzhou_49,21:40:12.107575100
监听者2.......监听到消息:hello chuangzhou_12,21:40:12.248889900
监听者2.......监听到消息:hello chuangzhou_14,21:40:12.450119
监听者2.......监听到消息:hello chuangzhou_16,21:40:12.653348500
监听者2.......监听到消息:hello chuangzhou_18,21:40:12.855420100
监听者2.......监听到消息:hello chuangzhou_20,21:40:13.058581700
监听者2.......监听到消息:hello chuangzhou_22,21:40:13.260466200
监听者2.......监听到消息:hello chuangzhou_24,21:40:13.462077600
监听者2.......监听到消息:hello chuangzhou_26,21:40:13.664272
监听者2.......监听到消息:hello chuangzhou_28,21:40:13.865059500
监听者2.......监听到消息:hello chuangzhou_30,21:40:14.069590900
监听者2.......监听到消息:hello chuangzhou_32,21:40:14.272157700
监听者2.......监听到消息:hello chuangzhou_34,21:40:14.475049600
监听者2.......监听到消息:hello chuangzhou_36,21:40:14.678456700
监听者2.......监听到消息:hello chuangzhou_38,21:40:14.880600400
监听者2.......监听到消息:hello chuangzhou_40,21:40:15.081184600
监听者2.......监听到消息:hello chuangzhou_42,21:40:15.284129300
监听者2.......监听到消息:hello chuangzhou_44,21:40:15.485866500
监听者2.......监听到消息:hello chuangzhou_46,21:40:15.688105500
监听者2.......监听到消息:hello chuangzhou_48,21:40:15.890066100
从结果可以发现,两个消费者消费的数量相等,这是因为rabbitmq 内部的预取机制
因此要加一限制,让处理能力高的多处理,处理能力低的少处理
可以通过prefetch 来限制预取的数量:
spring:
rabbitmq:
host: 192.168.184.152
port: 5672
username: root
password: 123456
virtual-host: /
listener: # 预取限制
simple:
prefetch: 1 # 每次只能获取一条消息,处理完成才能获取下一个
限制后的处理结果:接收者1处理40条,接收者2处理了10条
消费者1,监听到消息:hello chuangzhou_0,21:44:35.240829200
消费者2.......监听到消息:hello chuangzhou_1,21:44:35.254832600
消费者1,监听到消息:hello chuangzhou_2,21:44:35.275837100
消费者1,监听到消息:hello chuangzhou_3,21:44:35.299842700
消费者1,监听到消息:hello chuangzhou_4,21:44:35.323848900
消费者1,监听到消息:hello chuangzhou_5,21:44:35.347585700
消费者2.......监听到消息:hello chuangzhou_6,21:44:35.359841800
消费者1,监听到消息:hello chuangzhou_7,21:44:35.381834600
消费者1,监听到消息:hello chuangzhou_8,21:44:35.405432200
消费者1,监听到消息:hello chuangzhou_9,21:44:35.427001600
消费者1,监听到消息:hello chuangzhou_10,21:44:35.450992500
消费者2.......监听到消息:hello chuangzhou_11,21:44:35.468943100
消费者1,监听到消息:hello chuangzhou_12,21:44:35.491340
消费者1,监听到消息:hello chuangzhou_13,21:44:35.515337
消费者1,监听到消息:hello chuangzhou_14,21:44:35.539248900
消费者1,监听到消息:hello chuangzhou_15,21:44:35.562228400
消费者2.......监听到消息:hello chuangzhou_16,21:44:35.575936
消费者1,监听到消息:hello chuangzhou_17,21:44:35.596546800
消费者1,监听到消息:hello chuangzhou_18,21:44:35.619077100
消费者1,监听到消息:hello chuangzhou_19,21:44:35.643939600
消费者1,监听到消息:hello chuangzhou_20,21:44:35.666689100
消费者2.......监听到消息:hello chuangzhou_21,21:44:35.681717600
消费者1,监听到消息:hello chuangzhou_22,21:44:35.702930600
消费者1,监听到消息:hello chuangzhou_23,21:44:35.725872500
消费者1,监听到消息:hello chuangzhou_24,21:44:35.749369400
消费者1,监听到消息:hello chuangzhou_25,21:44:35.773150600
消费者2.......监听到消息:hello chuangzhou_26,21:44:35.790152200
消费者1,监听到消息:hello chuangzhou_27,21:44:35.812955500
消费者1,监听到消息:hello chuangzhou_28,21:44:35.835982600
消费者1,监听到消息:hello chuangzhou_29,21:44:35.861004
消费者1,监听到消息:hello chuangzhou_30,21:44:35.884358800
消费者2.......监听到消息:hello chuangzhou_31,21:44:35.896332900
消费者1,监听到消息:hello chuangzhou_32,21:44:35.917354
消费者1,监听到消息:hello chuangzhou_33,21:44:35.938549100
消费者1,监听到消息:hello chuangzhou_34,21:44:35.960244200
消费者1,监听到消息:hello chuangzhou_35,21:44:35.982279900
消费者2.......监听到消息:hello chuangzhou_36,21:44:36.002253200
消费者1,监听到消息:hello chuangzhou_37,21:44:36.022921100
消费者1,监听到消息:hello chuangzhou_38,21:44:36.044534100
消费者1,监听到消息:hello chuangzhou_39,21:44:36.068426600
消费者1,监听到消息:hello chuangzhou_40,21:44:36.091488300
消费者2.......监听到消息:hello chuangzhou_41,21:44:36.109492
消费者1,监听到消息:hello chuangzhou_42,21:44:36.129921500
消费者1,监听到消息:hello chuangzhou_43,21:44:36.152451900
消费者1,监听到消息:hello chuangzhou_44,21:44:36.173456300
消费者1,监听到消息:hello chuangzhou_45,21:44:36.195584700
消费者2.......监听到消息:hello chuangzhou_46,21:44:36.215832200
消费者1,监听到消息:hello chuangzhou_47,21:44:36.236337200
消费者1,监听到消息:hello chuangzhou_48,21:44:36.260252600
消费者1,监听到消息:hello chuangzhou_49,21:44:36.282758700
发布订阅模型介绍
发布订阅模式与BasicQueu、WorkQueue 的区别就是允许将同一消息发给多个消费者,实现方式是加入了exchange
(BasicQueu、WorkQueue的一条消息只能被一个消费者消费)
FanoutExchange
Fanout Exchange 会将接收到的消息路由到每一个跟其绑定的queue
案例实现步骤:
步骤1 : 在consumer 服务中,声明交换机、队列,并将两者绑定
package cn.itcast.mq.config;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.FanoutExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class FanoutExchangeConfig {
//创建交换机
@Bean
public FanoutExchange fanoutExchange(){
return new FanoutExchange("chuangzhou.fanout");
}
//创建队列1
@Bean
public Queue fanoutQueue1(){
return new Queue("fanout.queue1");
}
//创建队列2
@Bean
public Queue fanoutQueue2(){
return new Queue("fanout.queue2");
}
//将队列1绑定到交换机
@Bean
public Binding binding1(Queue fanoutQueue1,FanoutExchange fanoutExchange){
return BindingBuilder.bind(fanoutQueue1).to(fanoutExchange);
}
//将队列2绑定到交换机
@Bean
public Binding binding2(Queue fanoutQueue2,FanoutExchange fanoutExchange){
return BindingBuilder.bind(fanoutQueue2).to(fanoutExchange);
}
}
重启consumer服务后就可以在控制台看到exchange及queue 绑定成功:
步骤2. 在consumer服务中,编写两个消费者方法,分别监听fanout.queue1 和 fanout.queue2
package cn.itcast.mq.listener;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
import java.time.LocalTime;
@Component
public class RabbitListenerBean {
@RabbitListener(queues = "fanout.queue1")
public void listenerFanoutQueue1(String msg) throws InterruptedException {
System.out.println("fanout.queue1 监听到消息:" + msg + "," + LocalTime.now());
}
@RabbitListener(queues = "fanout.queue2")
public void listenerFanoutQueue2(String msg) throws InterruptedException {
System.out.println("fanout.queue2 监听到消息:" + msg + "," + LocalTime.now());
}
}
- 在publisher中编写测试方法,向交换机发送消息
@Test
public void testSendMes2FanoutExchange(){
String fanouName = "chuangzhou.fanout";
String message = "hello fanouExchange";
rabbitTemplate.convertAndSend(fanouName,"",message);
}
消费者监听到消息:
DirectExchange
DirectExchange 会将收到的消息根据规则路由到指定的queue, 因此称为路由模式
- 每一个Queue 都与Exchange 设置一个BindingKey
- 发布者发送消息时,指定消息的RoutingKey
- Exchage将消息路由到BindKey与消息RoutingKey一致的队列
案例:
步骤1.:consumer 利用 @RabbitListener 进行绑定并设置BindKey
@RabbitListener(bindings = @QueueBinding(
value = @Queue("direct.queue1"),
exchange = @Exchange(value = "chuangzhou.direct",type = ExchangeTypes.DIRECT),
key = {"blue","red"}
))
public void listenerDirectQueue1(String msg) throws InterruptedException {
System.out.println("direct.queue1 监听到消息:" + msg + "," + LocalTime.now());
}
@RabbitListener(bindings = @QueueBinding(
value = @Queue("direct.queue2"),
exchange = @Exchange(value = "chuangzhou.direct",type = ExchangeTypes.DIRECT),
key = {"blue","black"}
))
public void listenerDirectQueue2(String msg) throws InterruptedException {
System.out.println("direct.queue2 监听到消息:" + msg + "," + LocalTime.now());
}
步骤2: 发送者发送消息时指定routingKey
@Test
public void testSendMes2DirectExchange(){
String fanouName = "chuangzhou.direct";
String message = "hello directExchange";
rabbitTemplate.convertAndSend(fanouName,"blue",message);
}
总结:
Direct 与 Fanout 交换机的差异?
- Fanout 交换机将消息路由给每一个与之绑定的队列
- Direct 交换机根据RoutingKey 判断路由给哪个队列
- 如果多个队列具有相同的RoutingKey,则与Fanout 功能类似
基于@RabbitListener 注解声明队列和交换机有哪些常见注解?
- @QueueBind
- @Queue
- @Exchange
TopicExchange
TopicExchange 与 DirectExchange 类似,区别在于routingKey必须是多个单词的列表,并且以.分割。
consumer:
@RabbitListener(bindings = @QueueBinding(
value = @Queue("topic.queue1"),
exchange = @Exchange(value = "chuangzhou.topic",type = ExchangeTypes.TOPIC),
key = "china.#"
))
public void listenerTopicQueue1(String msg) throws InterruptedException {
System.out.println("topic.queue1 监听到消息:" + msg + "," + LocalTime.now());
}
@RabbitListener(bindings = @QueueBinding(
value = @Queue("topic.queue2"),
exchange = @Exchange(value = "chuangzhou.topic",type = ExchangeTypes.TOPIC),
key = "#.news"
))
public void listenerTopicQueue2(String msg) throws InterruptedException {
System.out.println("topic.queue2 监听到消息:" + msg + "," + LocalTime.now());
}
发送消息:
@Test
public void testSendMes2TopicExchange(){
String exchangeName = "chuangzhou.topic";
String message = "中国的光刻机研发出来了";
rabbitTemplate.convertAndSend(exchangeName,"china.news",message);
}
两个队列都收到消息:
使用routingKeyId weather.news 发送消息:
@Test
public void testSendMes2TopicExchange(){
String exchangeName = "chuangzhou.topic";
String message = "今天的天气太好了";
rabbitTemplate.convertAndSend(exchangeName,"weather.news",message);
}
消息转换器
引入案例:
声明一个简单队列:
@Bean
public Queue objectQueue(){
return new Queue("object.queue");
}
向队列发送一个实体对象,控制台查看消息:
@Test
public void sendMes2ObjQueue(){
Map<String, Object> map = new HashMap<>();
map.put("name", "wangcai");
map.put("age", "1");
rabbitTemplate.convertAndSend("object.queue",map);
}
spring 对消息的处理对象是由org.springframework.amqp.support.converter.MessageConverter,而默认实现是SimpleMessageConverter,基于JDK的
ObjectOutputStrean 完成序列化。
如果要修改只需要定义一个MessageConverter 类型的Bean即可。推荐使用JSON序列化,步骤如下:
发送者:
1.父工程中引入依赖
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
2.创建转换器对象
@Bean
public MessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
消费者:
1.创建转换器对象
@Bean
public MessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
@RabbitListener(queues = "object.queue")
public void listenerObejctQueue(Map<String,Object> msg) throws InterruptedException {
System.out.println("object.queue 监听到消息:" + msg + "," + LocalTime.now());
}
总结:
SpringAMQP 消息的序列化和反序列化是怎么实现的?
- 利用MessageConverter实现的,默认是JDK的序列化
- 注意发送发与接收方必须使用相同的MessageConverter
本文来自博客园,作者:chuangzhou,转载请注明原文链接:https://www.cnblogs.com/czzz/p/16630190.html