Kafka之 vm.max_map_count

Maximum number of memory map areas a process may have (aka vm.max_map_count).
进程可以具有的最大内存映射区域数（也称为vm.max_map_count）。

See the Linux kernel documentation.
You should keep an eye at this OS-level property when considering the maximum number of partitions a broker may have.
在考虑broker可能具有的最大分区数时，您应该关注此OS级属性

By default, on a number of Linux systems, the value of vm.max_map_count is somewhere around 65535.
Linux 系统 vm.max_map_count 的默认值是 65535

Each log segment, allocated per partition, requires a pair of index/timeindex files, and each of these files consumes 1 map area.
每个分区分配的每个日志段需要一对index / timeindex文件，每个文件占用1个映射区域。

In other words, each log segment uses 2 map areas.
换句话说，每个日志段使用2个映射区域

Thus, each partition requires minimum 2 map areas, as long as it hosts a single log segment.
因此，每个分区至少需要2个映射区域，只要它承载单个日志段即可。

That is to say, creating 50000 partitions on a broker will result allocation of 100000 map areas
and likely cause broker crash with OutOfMemoryError (Map failed) on a system with default vm.max_map_count.
举个例子，如果你创建 50000 分区在一个 broker，将会创建 100000 个map areas 映射取；
在具有默认vm.max_map_count的系统上可能导致代理崩溃并出现OutOfMemoryError（映射失败）

Keep in mind that the number of log segments per partition varies depending on the segment size,
load intensity, retention policy and, generally, tends to be more than one.
请记住，每个分区的日志段数取决于段大小，负载强度，保留策略，通常往往不止一个

 

对于这个问题我是这么考虑的，Kafka使用page cache技术，也就是用户态内存映射内核态内存，这样可以让刷盘快速。前提就是这种映射关系也是有数量限制的，所以一般都会把这个数值配成大于10万