1、ES与kibana部署---rpm包方式

es可以使用二进制、docker、k8s、rpm方式部署，此处以rpm方式为例。相较于二进制部署，省去了繁琐的用户创建、证书生成、密码设置、启动脚本配置等操作，简化部署流程，可以将更多的精力用于es的使用而不是部署上面。

基础环境设置

基础环境的设置主要包含以下几个方面：

• 修改主机名
• 关闭防火墙和SELinux
• 配置时间同步
• 关闭swap分区
• 内核参数优化

其中重点说一下[系统内核参数优化]：

修改文件描述符数目：

修改原因

原因1： Elasticsearch 在节点和 HTTP 客户端之间进行通信也使用了大量的套接字（sockets）。 所有这一切都需要足够的文件描述符。
原因2：linux系统对每个用户、进程、或整个系统的可打开文件描述符数量都有一个限制，一般默认为1024。这对一个小的 Elasticsearch 节点来说实在是太低了，更不用说一个处理数以百计索引的节点。

修改limits.conf配置文件

# 修改limits.conf配置
vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

验证

# ulimit -n
65535

---

修改虚拟内存数大小：

修改原因

Elasticsearch 对各种文件混合使用了 NioFs（ 非阻塞文件系统）和 MMapFs （ 内存映射文件系统）。

请确保配置的最大映射数量，以便有足够的虚拟内存可用于 mmapped 文件

临时设置

# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144

永久设置

cat >> /etc/sysctl.conf << EOF
vm.max_map_count=262144
EOF
# sysctl -p 
vm.max_map_count = 262144

---

ES集群部署

下载rpm包

官网下载地址：

https://www.elastic.co/cn/downloads/past-releases#elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.9.2-x86_64.rpm

#验证下载包是否正确
rpm -K elasticsearch-8.9.2-x86_64.rpm

节点安装es（所有节点）

以下操作在所有节点都需要执行，执行部署相同

安装es的rpm包

安装 Elasticsearch 时，默认情况下会启用和配置安全功能。 安装 Elasticsearch 时，会自动进行以下配置：

• 启用身份验证和授权，并生成超级用户elastic的密码。
• 启用TLS 的证书和密钥，为传输层和 HTTP 层加密生成的。

密码、证书和密钥将会显示在终端。

rpm -ivh elasticsearch-8.9.2-x86_64.rpm

#安装过程会输出如下信息
warning: elasticsearch-8.9.2-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:elasticsearch-0:8.9.2-1          warning: /usr/lib/systemd/system/elasticsearch.service created as /usr/lib/systemd/system/elasticsearch.service.rpmnew
################################# [100%]
warning: ignoring JAVA_HOME=/usr/local/java; using bundled JDK
warning: ignoring JAVA_HOME=/usr/local/java; using bundled JDK
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : C0OSPnzZ1k+Ag+rqRGr4

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with 
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with 
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with 
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
warning: ignoring JAVA_HOME=/usr/local/java; using bundled JDK
warning: ignoring JAVA_HOME=/usr/local/java; using bundled JDK

设置开机自动动

systemctl daemon-reload
systemctl enable elasticsearch.service

创建数据和日志目录

mkdir /data/es-log /data/es-data
chown -R elasticsearch:elasticsearch /data

修改jvm配置

实际生产环境中计算公式：min（机器内存的一半，32GB内存）。也就是说：取机器环境内存的一半和32GB内存之间的小值。jvm配置如下

cat > /etc/elasticsearch/jvm.options.d/es.options << EOF
-Xms32g
-Xmx32g
EOF

修改master节点配置

配置文件说明

配置项	说明
cluster.name	集群名称，唯一确定一个集群。
cluster.initial_master_nodes	用于初始化第一次时选举 master 节点，填写 node name或者node ip
node.name	节点名称，一个集群中的节点名称是唯一固定的，不同节点不能同名。
node.master	主节点属性值
node.data	数据节点属性值
network.host	本节点的ip，或0.0.0.0
http.port	本节点的http端口
transport.port	集群之间通信的端口，若不指定默认：9300
path.logs	修改日志目录
path.data	修改数据目录
discovery.seed_hosts	节点发现需要配置一些种子节点，与7.X之前老版本：disvoery.zen.ping.unicast.hosts类似，一般配置集群中的全部节点

master节点配置

[root@es-master ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: es-cluster
node.name: es-master
# node.roles: [ master, ingest ] # 先不设置集群角色，待集群全部加入后再设置
path.data: /data/es-data
path.logs: /data/es-log
network.host: 192.168.10.100 # master节点IP,或0.0.0.0
# 填写集群ip或主机名列表
discovery.seed_hosts: ["es-master", "es-hot1", "es-hot2", "es-hot2", "es-warm1", "es-warm2", "es-cold"]

xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

cluster.initial_master_nodes: ["es-master"]  # master节点IP或主机名
http.host: 0.0.0.0

启动es服务

systemctl start elasticsearch

其他节点加入集群

节点加入集群

执行命令将节点加入集群，下面操作一定要使用es默认配置文件执行，待reconfigure-node完成后，再修改节点yml配置文件

生成集群注册token

[root@es-master ~]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

eyJ2ZXIiOiI4LjkuMiIsImFkciI6WyIxOTIuMTY4LjIuNDM6OTIwMCJdLCJmZ3IiOiJkYjllZGQwNmU3ZGRlYzk3OWYxZTA5MjI0NDdiODBkM2ViMzBjZjNkNmIzZDAxZDdiODk1NDY0OTBlNzI4MzM4Iiwia2V5IjoienV3SmFKUUI3X2dxMGJPY2pRQUM6Q3c4d3BpVDNUOWVrR3o4YUpZLUJ3ZyJ9

执行节点加入操作

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjkuMiIsImFkciI6WyIxOTIuMTY4LjIuNDM6OTIwMCJdLCJmZ3IiOiJkYjllZGQwNmU3ZGRlYzk3OWYxZTA5MjI0NDdiODBkM2ViMzBjZjNkNmIzZDAxZDdiODk1NDY0OTBlNzI4MzM4Iiwia2V5IjoiME95eWJKUUI3X2dxMGJPY3JRQWE6TFdhWDVwYVRRVVdYajN5bXFOSk1wQSJ9

修改es配置，重启服务

es的node节点配置参考如下

cluster.name: es-cluster
node.name: rss-data-dev-suz-kjc-worker18
node.roles: [ data_content, data_hot ]
path.data: /data1/es/es-data 
path.logs: /data1/es/es-log
network.host: 192.168.2.44
discovery.seed_hosts: ["rss-data-dev-suz-kjc-worker17", "rss-data-dev-suz-kjc-worker18","rss-data-dev-suz-kjc-worker19"]
cluster.initial_master_nodes: ["rss-data-dev-suz-kjc-worker17"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
http.host: 0.0.0.0
transport.host: 0.0.0.0

systemcl restart elasticsearch

重置elastic用户密码

重置密码操作是在master节点执行

[root@rss-data-dev-suz-kjc-worker17 es]# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
warning: ignoring JAVA_HOME=/usr/local/java; using bundled JDK
This tool will reset the password of the [elastic] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [elastic] user successfully reset.
New value: 5_yNxi62Goa87I*OiGwJ

修改master节点属性

由于在集群初始化过程中，master节点写入了集群元数据索引，需要先将元数据迁移至其他数据节点。我们可以使用cluster.routing.allocation.exclude._ip，将指定节点ip上面的分片分配到其他节点上。

[root@es-master ~]# curl -X PUT -H 'content-type:application/json' -d '{"transient":{"cluster.routing.allocation.exclude._ip":"192.168.2.43"}}' --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:5_yNxi62Goa87I*OiGwJ@127.0.0.1:9200/_cluster/settings

{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"exclude":{"_ip":"192.168.2.43"}}}}}}

等待分片迁移完成后，查看分片信息验证，保证master节点无分片数据

[root@rss-data-dev-suz-kjc-worker17 es]# curl --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:5_yNxi62Goa87I*OiGwJ@127.0.0.1:9200/_cat/shards?

.security-7 0 p STARTED 4 29.4kb 192.168.2.45 rss-data-dev-suz-kjc-worker19
.security-7 0 r STARTED 4 29.4kb 192.168.2.44 rss-data-dev-suz-kjc-worker18

修改es配置，重启es服务

[root@es-master ~]# vim /etc/elasticsearch/elasticsearch.yml 
# 指定节点角色
node.roles: [ master, ingest ]
[root@es-master ~]# systemctl restart elasticsearch

访问验证

查看集群状态

curl --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:5_yNxi62Goa87I*OiGwJ@127.0.0.1:9200/_cat/health?

1736995465 02:44:25 es-cluster green 3 2 2 1 0 0 0 0 - 100.0%

查看节点状态

curl --cacert /etc/elasticsearch/certs/http_ca.crt https://elastic:5_yNxi62Goa87I*OiGwJ@127.0.0.1:9200/_cat/nodes?v


ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.2.43           24          73   4    4.78    2.94     2.75 im        *      rss-data-dev-suz-kjc-worker17
192.168.2.45           54          88  12    4.67    3.89     5.43 hs        -      rss-data-dev-suz-kjc-worker19
192.168.2.44           24          77   4    1.25    1.33     1.70 hs        -      rss-data-dev-suz-kjc-worker18

kibana部署

下载安装rpm包

wget https://artifacts.elastic.co/downloads/kibana/kibana-8.9.2-x86_64.rpm

rpm -ivh kibana-8.9.2-x86_64.rpm

kibana注册集群

生成token，用于kibana注册集群

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

eyJ2ZXIiOiI4LjkuMiIsImFkciI6WyIxOTIuMTY4LjIuNDM6OTIwMCJdLCJmZ3IiOiJkYjllZGQwNmU3ZGRlYzk3OWYxZTA5MjI0NDdiODBkM2ViMzBjZjNkNmIzZDAxZDdiODk1NDY0OTBlNzI4MzM4Iiwia2V5IjoiaEZzcmJaUUIwazljZWFDbWlnUmI6Q1dkMno1S25SelN4YzJzaFJBcUx2USJ9

kibana注册集群

/usr/share/kibana/bin/kibana-setup --enrollment-token eyJ2ZXIiOiI4LjkuMiIsImFkciI6WyIxOTIuMTY4LjIuNDM6OTIwMCJdLCJmZ3IiOiJkYjllZGQwNmU3ZGRlYzk3OWYxZTA5MjI0NDdiODBkM2ViMzBjZjNkNmIzZDAxZDdiODk1NDY0OTBlNzI4MzM4Iiwia2V5IjoiaEZzcmJaUUIwazljZWFDbWlnUmI6Q1dkMno1S25SelN4YzJzaFJBcUx2USJ9


✔ Kibana configured successfully.

To start Kibana run:
  bin/kibana

修改配置，启动服务

从es8开始，kibana使用token注册连接es，kibana通过解析token可以获取到es的地址用户名和密码信息，因此在kibana配置文件中无需设置es地址与账号密码信息。

vim /etc/kibana/kibana.yml

server.port: 5601
server.host: "0.0.0.0"
# kibana中文界面
i18n.locale: "zh-CN"
monitoring.ui.ccs.enabled: false

访问验证

访问 192.168.2.43:5601

用户是elastic，用户密码是5_yNxi62Goa87I*OiGwJ

如果stack monitor页面访问正常，说明部署算是正常了