7.Kafka,构建TB级异步消息系统
1.阻塞队列
- BlockingQueue
-
- 解决线程通信的问题。
- 阻塞方法:put、take。
- 生产者消费者模式
-
- 生产者:产生数据的线程。
- 消费者:使用数据的线程。
- 实现类
-
- ArrayBlockingQueue
- LinkedBlockingQueue
- PriorityBlockingQueue、SynchronousQueue、DelayQueue等。
面试题:写一个生产者消费者实现
public class Test { public static void main(String[] args) { BlockingQueue<String> queue = new LinkedBlockingDeque<>(10); Producer p = new Producer(queue); Consumer c = new Consumer(queue); new Thread(p,"producer").start(); new Thread(c,"consumer").start(); } } class Consumer implements Runnable { private BlockingQueue<String> queue; public Consumer(BlockingQueue<String> queue) { this.queue = queue; } @Override public void run() { try { while(true){ Thread.sleep(20); System.out.println("消费者消费了:" + queue.take()); } }catch (InterruptedException e) { e.printStackTrace(); } } } class Producer implements Runnable{ private BlockingQueue<String> queue; public Producer(BlockingQueue<String> queue){ this.queue = queue; } @Override public void run() { try { for (int i = 0; i < 5; i++) { String tmp = "a product " + i + " from:" + Thread.currentThread().getName(); System.out.println("生产者生产了:" + tmp); queue.put(tmp); Thread.sleep(20); } }catch (InterruptedException e) { e.printStackTrace(); } } }
2.kafka入门
- Kafka简介
-
- Kafka是一个分布式的消息队列。
- 应用:消息系统、日志收集、用户行为追踪、流式处理。
- Kafka特点
-
- 高吞吐量、消息持久化、高可靠性、高扩展性。
- Kafka术语
Kafka相关链接:https://kafka.apache.org/
Windows下使用Kafka
在2.8以前,kafka安装前需要安装zookeeper。如果不需要额外使用zookeeper其他功能,可以安装2.8以后的版本。
在启动前改一下相关配置:
(1)解压kafka压缩包
(2)config下的zookeeper.properties
(3)config下的server.properties
注意启动的时候先zookeeper后kafka,停止的时候先kafka后zookeeper。
(4) 启动zookeeper
cd E:\Software\Kafka\kafka_2.12-3.4.0
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
报错:INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider),需要改zookeeper.properties
(5)启动kafka
bin\windows\kafka-server-start.bat config\server.properties
启动完成后会出现配置的文件夹
(6) 创建主题
bin\windows\kafka-topics.bat --create --topic topicDemo --bootstrap-server localhost:9092
(7)显示所有topic列表
bin\windows\kafka-topics --list --bootstrap-server localhost:9092
(8) 向主题发送消息
E:\Software\Kafka\kafka_2.12-3.4.0>bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic topicDemo
(9)消费消息
bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic topicDemo --from-beginning
3.Spring整合Kafka
(1)引入依赖
这里记得spring-kafka的版本对应一下你下载的版本
<dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> <version>2.8.3</version> </dependency>
(2)配置Kafka
在application.properties中配置server、consumer
#KafkaProperties spring.kafka.bootstrap-servers=localhost:9092 #kafka安装目录下config下consumer.properties中找 spring.kafka.consumer.group-id=test-consumer-group #是否自动提交消费者offset,消费者是按offset读取数据,这个选项表明要记录offset spring.kafka.consumer.enable-auto-commit=true #自动提交的频率,设置为3s spring.kafka.consumer.auto-commit-interval=3000
(3)访问Kafka
- 生产者
kafkaTemplate.send(topic, data); - 消费者
@KafkaListener(topics = {“test”})
public void handleMessage(ConsumerRecord record) {}
public class KafkaTest { @Autowired KafkaProducer kafkaProducer; @Autowired KafkaConsumer kafkaConsumer; @Test public void testKafka(){ kafkaProducer.sendMessage("test","hello world"); kafkaProducer.sendMessage("test","I love java"); try { Thread.sleep(1000*10); } catch (InterruptedException e) { e.printStackTrace(); } } } @Component class KafkaProducer{ @Autowired private KafkaTemplate kafkaTemplate; public void sendMessage(String topic,String content){ kafkaTemplate.send(topic,content); } } @Component class KafkaConsumer{ @KafkaListener(topics = {"test"}) public void handleMessage(ConsumerRecord record){ System.out.println(record.value()); } }
4.发送系统通知
- 触发事件
- 评论后,发布通知
- 点赞后,发布通知
- 关注后,发布通知
- 处理事件
- 封装事件对象
- 开发事件的生产者
- 开发事件的消费者
(1)封装事件对象
@Data public class Event { //张三给李四点赞-->userId是张三,entityUserId是李四 private String topic; private int userId; private int entityId;//哪个实体 private int entityType;//评论/点赞/关注 private int entityUserId;//发布实体的人 private Map<String,Object> data=new HashMap<>(); public Event setData(String key, Object value) {//注意set方法的修改是为了可以类似sb.append(" ").append(“1”);的操作 this.data.put(key, value); return this; } }
(2)事件生产者
新建event包,在包下新建EventProducer类
@Component public class EventProducer { @Autowired private KafkaTemplate kafkaTemplate; //处理事件-->发送消息 public void fireEvent(Event event){ //将event发送到指定topic kafkaTemplate.send(event.getTopic(), JSONObject.toJSONString(event)); } }
(3)事件消费者
@Slf4j @Component public class EventConsumer{ //消息最终是要往message表中插入数据 @Autowired private MessageService messageService; //三种类型新消息 //follow队列 关注 //comment队列 评论 //like队列 点赞 @KafkaListener(topics = {TOPIC_LIKE,TOPIC_COMMENT,TOPIC_FOLLOW}) public void handCommentMessage(ConsumerRecord record){ if(record==null||record.value()==null){ log.error("消息的内容为空!"); return; } //接收到的消息事件 Event event= JSONObject.parseObject(record.value().toString(),Event.class); if(event==null){ log.error("消息格式错误!"); return; } //发送站内通知,主要是构造message对象 Message message = new Message(); //User表中id为1代表系统用户 message.setFromId(SYSTEM_USER_ID); message.setToId(event.getEntityUserId()); message.setConversationId(event.getTopic()); message.setCreateTime(new Date()); //消息内容 Map<String,Object> content = new HashMap<>(); content.put("userId",event.getUserId()); content.put("entityType",event.getEntityType()); content.put("entityId",event.getEntityId()); if(!event.getData().isEmpty()){ for(Map.Entry<String,Object> entry:event.getData().entrySet()){ content.put(entry.getKey(),entry.getValue()); } } //消息入库 message.setContent(JSONObject.toJSONString(content)); messageService.addMessage(message); } }
(4)处理事件
评论事件:commentController类,在添加评论后添加以下代码
//触发评论事件 Event event=new Event(); event.setTopic(TOPIC_COMMENT); event.setUserId(hostHolder.getUser().getId()); event.setEntityId(comment.getEntityId()); event.setEntityType(comment.getEntityType()); event.setData("postId",discussPostId); if (comment.getEntityType() == ENTITY_TYPE_POST) {//帖子的评论 DiscussPost target = discussPostService.findDiscussPostById(comment.getEntityId()); event.setEntityUserId(target.getUserId()); } else if (comment.getEntityType() == ENTITY_TYPE_COMMENT) {//评论的评论 Comment target = commentService.findCommentById(comment.getEntityId()); event.setEntityUserId(target.getUserId()); } eventProducer.fireEvent(event);//向生产者发送消息
点赞事件:likeController类,在点赞后添加以下代码
//触发点赞事件 if(likeStatus==1){//取消点赞不用通知 Event event=new Event(); event.setTopic(TOPIC_LIKE); event.setUserId(hostHolder.getUser().getId()); event.setEntityType(entityType); event.setEntityId(entityId); event.setEntityUserId(entityUserId); event.setData("postId", postId); eventProducer.fireEvent(event); }
关注事件:followController类,在关注后添加以下代码
//触发关注事件 Event event=new Event(); event.setTopic(TOPIC_FOLLOW); event.setUserId(hostHolder.getUser().getId()); event.setEntityId(entityId); event.setEntityType(entityType); event.setEntityUserId(entityId); eventProducer.fireEvent(event);
(5)针对方法新增的postId参数,需要调整一些静态资源
dicuss-detail页面:
<li class="d-inline ml-2"> <a href="javascript:;" th:onclick="|like(this,1,${post.id},${post.userId},${post.id});|" class="text-primary"> <b th:text="${likeStatus==1?'已赞':'赞'}">赞</b> <i th:text="${likeCount}">11</i> </a> </li>
dicuss.js:
message成功入库:
入库的message的content格式是经过html格式转化的
5.显示系统通知
消息通知页面- 通知列表
- 显示评论、点赞、关注三种类型的通知
- 通知详情
- 分页显示某一类主题所包含的通知
- 未读消息
- 在页面头部显示所有的未读消息数量
(1)MessageMapper
//查询某个主题下的最新通知 Message selectLatestNotice(int userId,String topic);//userId:被通知的人 //查询某个主题所包含的通知数量 int selectNoticeCount(int usrId,String topic); //查询未读的通知数量 int selectNoticeUnreadCount(int userId,String topic); <!--查询某个主题下的最新通知--> <select id="selectLatestNotice" resultType="Message"> select <include refid="selectFields"></include> from message where id in ( select max(id) from message where status != 2 and from_id = 1 and to_id = #{userId} and conversation_id = #{topic} ) </select> <!--查询某个主题所包含的通知数量--> <select id="selectNoticeCount" resultType="int"> select count(id) from message where `status` != 2 and from_id=1 and to_id=#{userId} and conversation_id=#{topic} </select> <!--查询未读的通知数量--> <select id="selectNoticeUnreadCount" resultType="int"> select count(id) from message where `status` = 0 and from_id=1 and to_id=#{userId} <if test="topic!=null"> and conversation_id=#{topic} </if> </select>
(2)MessageService
//每个类型的最新通知 public Message findLatestNotice(int userId,String topic){ return messageMapper.selectLatestNotice(userId,topic); } //每个类型的通知数量 public int findNoticeCount(int userId,String topic){ return messageMapper.selectNoticeCount(userId,topic); } //未读通知数量 public int findNoticeUnreadCount(int userId,String topic){ return messageMapper.selectNoticeUnreadCount(userId,topic); }
(3)MessageController
增加获取通知列表页面方法最下方系统通知数量noticeUnreadCount两行代码需要加到getLetterList()中
//系统通知页面 @RequestMapping(path = "/notice/list",method = RequestMethod.GET) public String getNoticeList(Model model){ User user=hostHolder.getUser(); //评论类通知 Message message=messageService.findLatestNotice(user.getId(),TOPIC_COMMENT); if(message!=null){ Map<String, Object> messageVO = new HashMap<>(); messageVO.put("message", message); String content = HtmlUtils.htmlUnescape(message.getContent()); Map<String, Object> data = JSONObject.parseObject(content, HashMap.class); //content中是事件内容 messageVO.put("user", userService.findUserById((Integer) data.get("userId"))); messageVO.put("entityType", data.get("entityType")); messageVO.put("entityId", data.get("entityId")); messageVO.put("postId", data.get("postId")); int count = messageService.findNoticeCount(user.getId(), TOPIC_COMMENT); messageVO.put("count", count); int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_COMMENT); messageVO.put("unread", unread); model.addAttribute("commentNotice", messageVO); } //点赞类通知 message=messageService.findLatestNotice(user.getId(),TOPIC_LIKE); if(message!=null){ Map<String, Object> messageVO = new HashMap<>(); messageVO.put("message", message); String content = HtmlUtils.htmlUnescape(message.getContent()); Map<String, Object> data = JSONObject.parseObject(content, HashMap.class); //content中是事件内容 messageVO.put("user", userService.findUserById((Integer) data.get("userId"))); messageVO.put("entityType", data.get("entityType")); messageVO.put("entityId", data.get("entityId")); messageVO.put("postId", data.get("postId")); int count = messageService.findNoticeCount(user.getId(), TOPIC_LIKE); messageVO.put("count", count); int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_LIKE); messageVO.put("unread", unread); model.addAttribute("likeNotice", messageVO); } //关注类通知 message=messageService.findLatestNotice(user.getId(),TOPIC_FOLLOW); if(message!=null){ Map<String, Object> messageVO = new HashMap<>(); messageVO.put("message", message); String content = HtmlUtils.htmlUnescape(message.getContent()); Map<String, Object> data = JSONObject.parseObject(content, HashMap.class); //content中是事件内容 messageVO.put("user", userService.findUserById((Integer) data.get("userId"))); messageVO.put("entityType", data.get("entityType")); messageVO.put("entityId", data.get("entityId")); messageVO.put("postId", data.get("postId")); int count = messageService.findNoticeCount(user.getId(), TOPIC_FOLLOW); messageVO.put("count", count); int unread = messageService.findNoticeUnreadCount(user.getId(), TOPIC_FOLLOW); messageVO.put("unread", unread); model.addAttribute("followNotice", messageVO); } //查询未读消息数量(letter和notice页面都需要有) //私信列表 int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null); model.addAttribute("letterUnreadCount", letterUnreadCount); //系统通知 int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);//只要status为0 model.addAttribute("noticeUnreadCount", noticeUnreadCount); return "/site/notice"; }
(4)改写notice.html
<!-- 内容 --> <div class="main"> <div class="container"> <div class="position-relative"> <!-- 选项 --> <ul class="nav nav-tabs mb-3"> <li class="nav-item"> <a class="nav-link position-relative" th:href="@{/letter/list}">朋友私信 <span class="badge badge-danger" th:text="${letterUnreadCount}" th:if="${letterUnreadCount!=0}">3</span></a> </li> <li class="nav-item"> <a class="nav-link position-relative active" th:href="@{/notice/list}">系统通知 <span class="badge badge-danger" th:text="${noticeUnreadCount}" th:if="${noticeUnreadCount!=0}">27</span></a> </li> </ul> </div> <!-- 通知列表 --> <ul class="list-unstyled"> <!--评论类通知--> <li class="media pb-3 pt-3 mb-3 border-bottom position-relative" th:if="${commentNotice.message!=null}"> <span class="badge badge-danger" th:text="${commentNotice.unread!=0?commentNotice.unread:''}">3</span> <img src="http://static.nowcoder.com/images/head/reply.png" class="mr-4 user-header" alt="通知图标"> <div class="media-body"> <h6 class="mt-0 mb-3"> <span>评论</span> <span class="float-right text-muted font-size-12" th:text="${#dates.format(commentNotice.message.createTime,'yyyy-MM-dd HH:mm:ss')}">2019-04-28 14:13:25</span> </h6> <div> <a th:href="@{/notice/detail/comment}"> 用户 <i th:utext="${commentNotice.user.username}">nowcoder</i> 评论了你的<b th:text="${commentNotice.entityType==1?'帖子':'回复'}">帖子</b> ... </a> <ul class="d-inline font-size-12 float-right"> <li class="d-inline ml-2"><span class="text-primary">共 <i th:text="${commentNotice.count}">3</i> 条会话</span></li> </ul> </div> </div> </li> <!--点赞类通知--> <li class="media pb-3 pt-3 mb-3 border-bottom position-relative" th:if="${likeNotice.message!=null}"> <span class="badge badge-danger" th:text="${likeNotice.unread!=0?likeNotice.unread:''}">3</span> <img src="http://static.nowcoder.com/images/head/like.png" class="mr-4 user-header" alt="通知图标"> <div class="media-body"> <h6 class="mt-0 mb-3"> <span>赞</span> <span class="float-right text-muted font-size-12" th:text="${#dates.format(likeNotice.message.createTime,'yyyy-MM-dd HH:mm:ss')}">2019-04-28 14:13:25</span> </h6> <div> <a th:href="@{/notice/detail/like}"> 用户 <i th:utext="${likeNotice.user.username}">nowcoder</i> 点赞了你的<b th:text="${likeNotice.entityType==1?'帖子':'回复'}">帖子</b> ... </a> <ul class="d-inline font-size-12 float-right"> <li class="d-inline ml-2"><span class="text-primary">共 <i th:text="${likeNotice.count}">3</i> 条会话</span></li> </ul> </div> </div> </li> <!--关注类通知--> <li class="media pb-3 pt-3 mb-3 border-bottom position-relative" th:if="${followNotice.message!=null}"> <span class="badge badge-danger" th:text="${followNotice.unread!=0?followNotice.unread:''}">3</span> <img src="http://static.nowcoder.com/images/head/follow.png" class="mr-4 user-header" alt="通知图标"> <div class="media-body"> <h6 class="mt-0 mb-3"> <span>关注</span> <span class="float-right text-muted font-size-12" th:text="${#dates.format(followNotice.message.createTime,'yyyy-MM-dd HH:mm:ss')}">2019-04-28 14:13:25</span> </h6> <div> <a th:href="@{/notice/detail/follow}"> 用户 <i th:utext="${followNotice.user.username}">nowcoder</i> 关注了你 ... </a> <ul class="d-inline font-size-12 float-right"> <li class="d-inline ml-2"><span class="text-primary">共 <i th:text="${followNotice.count}">3</i> 条会话</span></li> </ul> </div> </div> </li> </ul> </div> </div>
系统通知详情
每个类型的通知详情:即某个主题下的通知列表
(1)MessageMapper
// 查询某个主题所包含的通知列表 List<Message> selectNotices(int userId, String topic, int offset, int limit); <!--查询某类型通知列表--> <select id="selectNotices" resultType="Message"> select <include refid="selectFields"></include> from message where `status` !=2 and from_id = 1 and conversation_id = #{topic} order by create_time desc limit #{offset},#{limit} </select>
(2)MessageService
//某类型通知列表 public List<Message> findNotices(int userId,String topic,int offset,int limit){ return messageMapper.selectNotices(userId,topic,offset,limit); }
(3)MessageController
新增getNoticeDetail()方法
//查询某类型详细 通知列表 @RequestMapping(path = "/notice/detail/{topic}",method = RequestMethod.GET) public String getNoticeDetail(@PathVariable("topic")String topic,Page page,Model model){ User user=hostHolder.getUser(); //分页 page.setLimit(5); page.setRows(messageService.findNoticeCount(user.getId(), topic));//通知数量 page.setPath("/notice/detail/"+topic); //查询通知列表 List<Message> noticeList=messageService.findNotices(user.getId(), topic,page.getOffset(),page.getLimit()); List<Map<String,Object>> noticeVoList = new ArrayList<>(); if(noticeList!=null){ for(Message notice:noticeList){ Map<String,Object> map = new HashMap<>(); //通知 map.put("notice",notice); //内容 String content = notice.getContent(); content = HtmlUtils.htmlUnescape(content); HashMap<String,Object> data = JSONObject.parseObject(content, HashMap.class); map.put("user",userService.findUserById((Integer)data.get("userId"))); map.put("entityType",data.get("entityType")); map.put("entityId",data.get("entityId")); map.put("postId",data.get("postId")); map.put("fromUser",userService.findUserById(notice.getFromId())); //系统名 noticeVoList.add(map); } } model.addAttribute("notices",noticeVoList); //设置已读 List<Integer> ids = getLetterIds(noticeList); if(!ids.isEmpty()){ messageService.readMessage(ids); } return "/site/notice-detail"; }
(4)改写notice-detail.html
<!-- 内容 --> <div class="main"> <div class="container"> <div class="row"> <div class="col-8"> <h6><b class="square"></b> 系统通知</h6> </div> <div class="col-4 text-right"> <button type="button" class="btn btn-secondary btn-sm" onclick="back();">返回</button> </div> </div> <!-- 通知列表 --> <ul class="list-unstyled mt-4"> <li class="media pb-3 pt-3 mb-2" th:each="map:${notices}"> <img th:src="${map.fromUser.headerUrl}" class="mr-4 rounded-circle user-header" alt="系统图标"> <div class="toast show d-lg-block" role="alert" aria-live="assertive" aria-atomic="true"> <div class="toast-header"> <strong class="mr-auto" th:utext="${map.fromUser.username}">落基山脉下的闲人</strong> <small th:text="${#dates.format(map.notice.createTime,'yyyy-MM-dd HH:mm:ss')}">2019-04-25 15:49:32</small> <button type="button" class="ml-2 mb-1 close" data-dismiss="toast" aria-label="Close"> <span aria-hidden="true">×</span> </button> </div> <div class="toast-body"> <span th:if="${topic.equals('comment')}"> 用户 <i th:utext="${map.user.username}">nowcoder</i> 评论了你的<b th:text="${map.entityType==1?'帖子':'回复'}">帖子</b>, <a class="text-primary" th:href="@{|/discuss/detail/${map.postId}|}">点击查看</a> ! </span> <span th:if="${topic.equals('like')}"> 用户 <i th:utext="${map.user.username}">nowcoder</i> 点赞了你的<b th:text="${map.entityType==1?'帖子':'回复'}">帖子</b>, <a class="text-primary" th:href="@{|/discuss/detail/${map.postId}|}">点击查看</a> ! </span> <span th:if="${topic.equals('follow')}"> 用户 <i th:utext="${map.user.username}">nowcoder</i> 关注了你, <a class="text-primary" th:href="@{|/user/profile/${map.user.id}|}">点击查看</a> ! </span> </div> </div> </li> </ul> <!-- 分页 --> <nav class="mt-5" th:replace="index::pagination"> <ul class="pagination justify-content-center"> <li class="page-item"><a class="page-link" href="#">首页</a></li> <li class="page-item disabled"><a class="page-link" href="#">上一页</a></li> <li class="page-item active"><a class="page-link" href="#">1</a></li> <li class="page-item"><a class="page-link" href="#">2</a></li> <li class="page-item"><a class="page-link" href="#">3</a></li> <li class="page-item"><a class="page-link" href="#">4</a></li> <li class="page-item"><a class="page-link" href="#">5</a></li> <li class="page-item"><a class="page-link" href="#">下一页</a></li> <li class="page-item"><a class="page-link" href="#">末页</a></li> </ul> </nav> </div> </div> <script> function back() { location.href = CONTEXT_PATH + "/notice/list"; } </script>
(5)首页头部 消息数量
每个请求处理完都要查看,所以用拦截器处理
@Component
public class MessageInterceptor implements HandlerInterceptor {
@Autowired
private HostHolder hostHolder;
@Autowired
private MessageService messageService;
@Override
public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
User user = hostHolder.getUser();
if (user != null && modelAndView != null) {
int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);//私信未读
int noticeUnreadCount = messageService.findNoticeUnreadCount(user.getId(), null);//通知未读
modelAndView.addObject("allUnreadCount", letterUnreadCount + noticeUnreadCount);//未读总数量
}
}
}
将该拦截器添加到config
@Configuration public class WebMvcConfig implements WebMvcConfigurer { @Autowired private AlphaInterceptor alphaInterceptor; @Autowired private LoginTicketInterceptor loginTicketInterceptor; @Autowired private LoginRequiredInterceptor loginRequiredInterceptor; @Autowired private MessageInterceptor messageInterceptor; @Override public void addInterceptors(InterceptorRegistry registry) { registry.addInterceptor(alphaInterceptor) .excludePathPatterns("/css/*","/js/*","/img/*") .addPathPatterns("/register","/login"); //.order(0); 可以设置拦截器顺序 registry.addInterceptor(loginTicketInterceptor) .excludePathPatterns("/css/*","/js/*","/img/*"); registry.addInterceptor(loginRequiredInterceptor) .excludePathPatterns("/**/*.css", "/**/*.js", "/**/*.png", "/**/*.jpg", "/**/*.jpeg"); registry.addInterceptor(messageInterceptor) .excludePathPatterns("/**/*.css", "/**/*.js", "/**/*.png", "/**/*.jpg", "/**/*.jpeg"); } }
最后改下index.html
<li class="nav-item ml-3 btn-group-vertical" th:if="${loginUser!=null}"> <a class="nav-link position-relative" th:href="@{/letter/list}">消息 <span class="badge badge-danger" th:text="${allUnreadCount!=0?allUnreadCount:''}">12</span> </a> </li>