kubernetes中通过static pod部署elasticsearch生产集群

系统：centos 7.6
环境：x.x.x.x x.x.x. x.x.x.x

资源配置：

内存：16G

CPU：8核

硬盘：500G

目前部署的es磁盘/内存比在30:1，如果预算充足这个比例越小越好

1.创建和配置保存目录,因为es官方镜像启动用户为uid为1000的elasticsearch，所以需要在宿主机上创建一个uid为1000的用户

groupadd -g 1000 elasticsearch && useradd -g 1000 -u 1000 -s /sbin/nologin && mkdir -pv /data/k8s/volumn_data/{es_config,es_data} && chown -R 777 /data/k8s/volumn_data/{es_config,es_data}

2.调整系统配置

echo "* soft memlock unlimited" >>/etc/security/limits.conf 

echo "* hard memlock unlimited" >>/etc/security/limits.conf

echo "vm.max_map_count=655360" >> /etc/sysctl.conf    

 sysctl ‐p

            

3.创建configmap

apiVersion: v1
data:
    cluster.name: 'cdp-prd-cluster'
    node.name: 'prd-cdp-es-147'
    path.data: '/data'
    bootstrap.memory_lock: 'true'
    discovery.seed_hosts: '["x.x.x.x", "x.x.x.x","x.x.x.x"]'
    cluster.initial_master_nodes: '["x.x.x.x", "x.x.x.x","x.x.x.x"]'
    ELASTIC_PASSWORD: 'Aa111111'
kind: ConfigMap
metadata:
    name: es-config
    namespace: default

4.部署elasticsearch

apiVersion: v1
kind: Pod
metadata:
    labels:
    app: cdp-elasticsearch
    name: cdp-elasticsearch
    namespace: default
spec:
    containers:
    - image: hub.docker.cn/es:v7.1.1
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
        path: /
        port: 9200
        scheme: HTTP
        httpHeaders:
        - name: Authorization
          value: "xxx"
        initialDelaySeconds: 30
        periodSeconds: 15
        successThreshold: 1
        timeoutSeconds: 5
    name: cdp-elasticsearch
    ports:
    - containerPort: 9200
      name: db
      protocol: TCP
      hostPort: 9200
    - containerPort: 9300   
      name: transport
      protocol: TCP
      hostPort: 9300
    volumeMounts:
    - mountPath: /data
      name: elasticsearch-data
    - mountPath: /usr/share/elasticsearch/config
      name: elasticsearch-config
  hostNetwork: true
  volumes:
  - name: elasticsearch-data
    hostPath:
        path: /data/k8s/volumn_data/es_data
  - name : elasticsearch-config
    hostPath:
        path: /data/k8s/volumn_data/es_config/config
  initContainers:
  - image: alpine:3.6
    command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
    name: elasticsearch-logging-init
    securityContext:
      privileged: true

以上httpHeaders中的 value是更加之后生成的es账号密码经过base64加密而来，所在在集群部署完成之前先不要httpHeaders字段，待完全部署完成之后再加上认证

5.配置TLS
elasticsearch集群正常部署之后，进入到其中一个es节点，执行一下命令生成证书

./bin/elasticsearch‐certutil ca ‐‐days 3660
# 两次回车
./bin/elasticsearch‐certutil cert ‐‐ca elastic‐stack‐ca.p12
#三次回车
mkdir config/certs
mv elastic‐*.p12 config/certs/  

再把证书文件 elastic-certificates.p12 复制到其他master节点并赋予权限(/data/k8s/volumn_data/es_config/config)

6.修改所有节点配置文件

vim /data/k8s/volumn_data/es_config/config/elasticsearch.yml


cluster.name: "cdp-prd-cluster"
network.host: 0.0.0.0
node.name: xxx
path.data: /data
#bootstrap.memory_lock: true
discovery.seed_hosts: ["x.x.x.x","x.x.x.x","x.x.x.x"]
cluster.initial_master_nodes: ["x.x.x.x","x.x.x.x","x.x.x.x"]
#-----
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

7.重启所有节点es

8.生成客户端证书
进入任意一个es节点

./bin/elasticsearch‐certutil cert ‐‐ca config/certs/elastic‐stack‐ca.p12 ‐name "CN=esuser,OU=prd,DC=ddd,DC=com"
# 回车
client.p12
回车

9.拆分证书

mv client.p12 config/certs/

cd config/certs/ 

openssl pkcs12 ‐in client.p12 ‐nocerts ‐nodes > client‐key.pem

openssl pkcs12 ‐in client.p12 ‐clcerts ‐nokeys >11.集群验证 client.crt

openssl pkcs12 ‐in client.p12 ‐cacerts ‐nokeys ‐chain > client‐ca.crt

 chown -R elasticsearch.elasticsearch config/

10.配置密码

./bin/elasticsearch‐setup‐passwords interactive #手动设置各个账号的密码
./bin/elasticsearch‐setup‐passwords auto #随机密码

11.集群验证

curl ‐‐user elastic:xxxxx ‐XGET '127.0.0.1:9200/_cat/health?v&pretty'

12.elasticsearch用户权限创建

#创建所有index读写权限：
curl -XPOST --user XXX:XXX'http://127.0.0.1:9200/_xpack/security/role/readwriterole' -H "Content-Type: application/json" -d '{"indices":[{"names":["*"],"privileges":["read","write"]}]}'

#查询权限：
curl -XPOST --user XXX:XXX'http://127.0.0.1:9200/_xpack/security/role?pretty
#创建用户并授权：
curl -XPOST --user elastic:xxx 'http://127.0.0.1:9200/_xpack/security/user/rwuser' -H "Content-Type: application/json" -d '{
           "password" : "xxx",
           "full_name" : "read write user",
           "email" : "",
           "roles" : [ "readwriterole" ]
            }'
#查询用户：
curl -XPOST --user XXX:XXX'http://127.0.0.1:9200/_xpack/security/user?pretty