Kafka Queuing: Apache Kafka as a Messaging System
1. Objective
In this Apache Kafka tutorial, we will learn the concept of Apache Kafka Queuing. Basically, Queuing in Kafka is one of the models for messaging traditionally. So, let’s begin with the brief introduction to Kafka as a Messaging System, that will help us to understand the Kafka Queuing well.
Moreover, we will see some of the applications of Kafka Queue to clear the concept better.So, let’s start with Kafka Queuing tutorial.
2. Kafka as a Messaging System
There are two models for messaging traditionally, such as Kafka queuing and publish-subscribe in Kafka.
Let’s discuss both Kafka models in detail:
Test how much you know about Kafka
i. Kafka Queue
In this Kafka messaging system, a pool of Kafka consumers may read from a server. Also, each record goes to one of them here. It has some strengths as well as some weakness.
Its strength is that it permits us to divide up the processing of data over multiple consumer instances, that help us scale our processing.
But its weakness is, it is not multi-subscriber, as soon as one process reads the data it’s gone.
ii. Kafka Publish-Subscribe
Whereas in this Kafka Publish-Subscribe system, the record is broadcast to all the Kafka consumers. It permits us to broadcast data to multiple processes.
However, it also has some limitations, like there is no way of scaling processing because here every message goes to every subscriber.
In Kafka, these two concepts are generalized by the Kafka consumer group. However, the consumer group in Kafka permits us to divide up processing over a collection of processes, with a Kafka queue. Here the collection of processes refers to the members of the consumer group. Moreover, Kafka permits us to broadcast messages to multiple consumer groups, with Kafka publish-subscribe.
Learn Apache Kafka Workflow | Kafka Pub-Sub Messaging
So, the main benefit of Kafka’s model is both these properties are available in every Kafka topic —it can scale processing as well as it is multi-subscriber. Hence, that implies we do not have to select one or the other.
Kafka has stronger ordering guarantees than a traditional messaging system, too.
Since there is no parallelism in the processing in the traditional system, Kafka performs it well with the notion of parallelism. Because Kafka can offer both ordering guarantees and load balancing over a pool of consumer processes.
3. Need for Kafka Queuing
One of Kafka Queuing application is Microservice architecture. It essentially demands some sort of message queuing system.
Let’s understand microservice architecture first. It is a concept of decoupling an interconnected monolithic application into different independent modules as well as external data sources along with APIs. Thus, for the purpose of handling microservices-external-source and inter-microservice and communications, Message queuing comes into the picture.
In addition, while we divide a big monolithic application into smaller, microservices (loosely-coupled), at that time the REST API calls increases which are amongst those microservices, and the number of connections to external data sources also increases.
Read Kafka Performance Tuning – Ways for Kafka Optimization
However, keeping this huge system synchronous is not desirable, because it can render the entire application unresponsive. Also, it can defeat the whole purpose of dividing into microservices in the first place.
Hence, having Kafka at that time makes the whole data flow easier. Because it is distributed, highly fault-tolerant and also it has constant monitoring of broker nodes through services like Zookeeper. So, it makes it efficient to work.
4. Message Queuing in ML Solutions Pipeline
Apart from that, we can also use Kafka queuing for the various ML solution pipelines.
However, ML solutions are built as:
A user interface (on the client side, mobile/web) — →An API server and the database — →Machine learning (blackbox).
However, ML black box is very compute-heavy and also it’s not practically easy to have those requests on a blocking synchronous mode. Moreover, all the requests can be in a queue and configure consumer API to take those requests one by one and feed them into the ML black box, in this scenario. So, while it comes to compute-intensive tasks, this pipeline can easily handle such as recognizing objects from thousands of images, that might take considerable time even without missing any requests.
Let’s revise Apache Kafka Producer For Beginners 2018
Basically, microservices deployed into containers, such as by fault-tolerant distributed clusters of Kafka broker nodes it is mediated and using a Zookeeper it monitors it, which seems like a new way of enterprising software development.
So, this was all about Kafka Queuing: Kafka as a messaging system. Hope you like our explanation.
5. Conclusion
Hence, we have seen the complete concept of Kafka Queuing. Moreover, we discussed why Kafka as a message queuing. Along with this, we learned the two models- Kafka queue and Kafka publish-subscribe. Also, we saw why we need Kafka Queuing. Lastly, we discussed message queuing in the ML solution pipeline. However, if any query occurs regarding Queuing in Kafka, feel free to ask through the comment section.
See also-
Best Apache Kafka Quiz – Solve in 3 Minute
For reference
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 没有源码,如何修改代码逻辑?
· 一个奇形怪状的面试题:Bean中的CHM要不要加volatile?
· [.NET]调用本地 Deepseek 模型
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· .NET Core 托管堆内存泄露/CPU异常的常见思路
· 微软正式发布.NET 10 Preview 1:开启下一代开发框架新篇章
· DeepSeek R1 简明指南:架构、训练、本地部署及硬件要求
· 没有源码,如何修改代码逻辑?
· NetPad:一个.NET开源、跨平台的C#编辑器
· 面试官:你是如何进行SQL调优的?
2019-08-20 zookeeper——官网介绍2