二进制部署高可用的etcd集群

此文档已过时，仅作参考使用

使用cfssl生成证书

下载cfssl

?
1
2
3
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssl_1.6.2_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssljson_1.6.2_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssl-certinfo_1.6.2_linux_amd64
  356  mv cfssl_1.6.2_linux_amd64 /usr/local/bin/cfssl
  357  mv cfssl-certinfo_1.6.2_linux_amd64 /usr/sbin/certinfo
  358  mv cfssljson_1.6.2_linux_amd64 /usr/sbin/cfssljson

可以用以下命令生成默认格式

?
1
2
$ cfssl print-defaults config > ca-config.json
$ cfssl print-defaults csr > ca-csr.json

创建ca

  • 制作证书模板配置文件
  • 证书颁发机构，颁发证书时，需要参考不同模板的要求进行证书创建
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@controller CA]# cat  ca-config.json
{
    "signing": {
        "default": {
            "expiry": "87600h"   # 默认过期时间
        },
        "profiles": {  # 设置了多个场景，创建时可以-profile=www/client指定
            "www": {
                "expiry": "87600h",  # 过期时间
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"  # server auth指定此证书只能用在服务器端
                ]
            },
            "client": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"  # 只能用在客户端
                ]
            },
            "all": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth",   # 既可以客户端，也可以服务器
                    "server auth"
                ]
            }
        }
    }
}

证书申请文件

  • 申请证书配置文件
  • 证书申请者申请证书时，要填写的基本信息
  • 我们去办理身份证时向机关提供的基本信息
复制代码
cat ca-csr.json

{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
           "OU": "System"
        }
    ]
}
复制代码

生成CA证书与私钥

?
1
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

 ca：生成文件名称前缀

制作kubernetes证书

使用自签名证书生成子证书方式
?
1
2
3
4
5
6
gencert: 生成新的key(密钥)和签名证书
-initca：初始化一个新ca
-ca：指明ca的证书
-ca-key：指明ca的私钥文件
-config：指明请求证书的json文件
-profile：与-config中的profile对应，是指根据config中的profile段来生成证书的相关信息
etcd证书生成
创建Etcd证书签名请求文件
复制代码
cat etcd-csr.json

{
    "CN":"etcd",
    "hosts": [
        "127.0.0.1",
        "192.168.50.220",
        "192.168.50.221",
        "192.168.50.222",
        "*.kubernetes.master",
        "localhost",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo":"rsa",
        "size": 2048
    },
    "names": [
        {
            "C":"CN",
            "ST":"BeiJing",
            "L":"BeiJing",
            "O":"k8s",
            "OU":"System"
        }
    ]
}
复制代码

 知识点：

  • 这个证书目前专属于 apiserver，加了一个 *.kubernetes.master域名以便内部私有 DNS 解析使用(可删除)；kubernetes 这几个能不能删掉，答案是不可以的；因为当集群创建好后，default namespace 下会创建一个叫 kubenretes 的svc，有一些组件会直接连接这个 svc 来跟 api 通信的，证书如果不包含可能会出现无法连接的情况；其他几个 kubernetes 开头的域名作用相同

  • hosts包含的是授权范围，不在此范围的的节点或者服务使用此证书就会报证书不匹配错误。10.254.0.1是指kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP。

 签发证书

?
1
384  cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem  -config=ca-config.json  -profile=all  etcd-csr.json | cfssljson -bare etcd
?
1
2
3
4
5
6
[root@controller etcd]# ll
total 16
-rw-r--r-- 1 root root 1281 Jan 16 09:50 etcd.csr
-rw-r--r-- 1 root root  595 Jan 16 09:32 etcd-csr.json
-rw------- 1 root root 1675 Jan 16 09:50 etcd-key.pem
-rw-r--r-- 1 root root 1643 Jan 16 09:50 etcd.pem

使用openssl来查看证书信息

openssl x509 -in server.pem -noout -text

使用cfssl-certinfo来查看证书信息

 可以通过命令行来动态设置SAN模块，如设置域名，IP

?
1
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem  -config=ca-conf.json -profile=client -hostname=www.golang-node1.com,www.golang-node2.com,127.0.0.1,10.211.55.10 node-csr.json | cfssljson -bare node -

 

 

 部署高可用的etcd集群

 参考文档：https://www.cnblogs.com/hahaha111122222/p/16016386.html

下载并分发

复制代码
#!/bin/bash
cd /data/k8s/work
source /data/k8s/bin/env.sh
wget https://github.com/etcd-io/etcd/releases/download/v3.3.18/etcd-v3.3.18-linux-amd64.tar.gz
tar -zxf etcd-v3.3.18-linux-amd64.tar.gz

for node_ip in ${ETCD_IPS[@]}
do
  echo ">>> ${node_ip}"
  scp etcd-v3.3.18-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
  ssh root@${node_ip} "chmod +x /data/k8s/bin/*"
done
复制代码
 创建启动脚本
etcd-1
复制代码
]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/root/cert/etcd.pem \
--key-file=/root/cert/etcd-key.pem \
--peer-cert-file=/root/cert/etcd.pem \
--peer-key-file=/root/cert/etcd-key.pem \
--trusted-ca-file=/root/cert/ca.pem \
--peer-trusted-ca-file=/root/cert/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
复制代码

 etcd-2

复制代码
]#  cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/root/cert/etcd.pem \
--key-file=/root/cert/etcd-key.pem \
--peer-cert-file=/root/cert/etcd.pem \
--peer-key-file=/root/cert/etcd-key.pem \
--trusted-ca-file=/root/cert/ca.pem \
--peer-trusted-ca-file=/root/cert/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
复制代码
etcd-3
复制代码
]#  cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/root/cert/etcd.pem \
--key-file=/root/cert/etcd-key.pem \
--peer-cert-file=/root/cert/etcd.pem \
--peer-key-file=/root/cert/etcd-key.pem \
--trusted-ca-file=/root/cert/ca.pem \
--peer-trusted-ca-file=/root/cert/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
复制代码
参数解释

创建配置文件

etcd-1

复制代码
]# cat /etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.50.220:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.220:2379"
ETCD_NAME="etcd-cluster1"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.220:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.220:2379"
ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380"
ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
复制代码

etcd-2

复制代码
]# cat /etc/etcd/etcd.conf
#[Member]
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.50.221:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.221:2379"
ETCD_NAME="etcd-cluster2"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.221:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.221:2379"
ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380"
ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
复制代码

etcd-3

复制代码
]# cat /etc/etcd/etcd.conf
#[Member]
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.50.222:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://192.168.50.222:2379"
ETCD_NAME="etcd-cluster3"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.50.222:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.50.222:2379"
ETCD_INITIAL_CLUSTER="etcd-cluster1=https://192.168.50.220:2380,etcd-cluster2=https://192.168.50.221:2380,etcd-cluster3=https://192.168.50.222:2380"
ETCD_INITIAL_CLUSTER_TOKEN="myk8s-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
复制代码

 启动

?
1
2
444  systemctl  daemon-reload
445  systemctl  start etcd
报错
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: newRaft 4a48efb1bd381d9c [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: 4a48efb1bd381d9c became follower at term 1
Jan 16 11:09:38 controller-2 etcd[77741]: raft2023/01/16 11:09:38 INFO: 4a48efb1bd381d9c switched to configuration voters=(5352791703792655772)
Jan 16 11:09:38 controller-2 etcd[77741]: simple token is not cryptographically signed
Jan 16 11:09:38 controller-2 etcd[77741]: starting server... [version: 3.4.9, cluster version: to_be_decided]
Jan 16 11:09:38 controller-2 etcd[77741]: 4a48efb1bd381d9c as single-node; fast-forwarding 7 ticks (election ticks 8)
Jan 16 11:09:38 controller-2 etcd[77741]: failed to purge snap db file open /data/etcd-data/member/snap: no such file or directory
Jan 16 11:09:38 controller-2 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Jan 16 11:09:38 controller-2 systemd[1]: Failed to start Etcd Server.
-- Subject: Unit etcd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit etcd.service has failed.
--
-- The result is failed.
Jan 16 11:09:38 controller-2 systemd[1]: Unit etcd.service entered failed state.
Jan 16 11:09:38 controller-2 systemd[1]: etcd.service failed.
Jan 16 11:09:38 controller-2 polkitd[786]: Unregistered Authentication Agent for unix-process:77736:49916625 (system bus name :1.315, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_U
 
[root@controller-2 CA]# rm -rf  /data/etcd-data/member/
 
[root@controller-2 CA]# systemctl  start etcd
 
[root@controller-2 CA]# systemctl  status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-01-16 11:09:46 CST; 12s ago
     Docs: https://github.com/coreos
 Main PID: 77757 (etcd)
   CGroup: /system.slice/etcd.service
           └─77757 /data/etcd/bin/etcd --data-dir=/data/etcd-data --wal-dir=/data/etcd-data --name=etcd2 --cert-file=/data/pki/etcd/etcd.pem --key-file=/data/pki/etcd/etcd-key.pem --trusted-ca-file=/data...
 
Jan 16 11:09:46 controller-2 etcd[77757]: raft2023/01/16 11:09:46 INFO: raft.node: 4a48efb1bd381d9c elected leader 4a48efb1bd381d9c at term 2
Jan 16 11:09:46 controller-2 etcd[77757]: published {Name:etcd2 ClientURLs:[https://192.168.50.225:2379]} to cluster 7a42d6c524509625
Jan 16 11:09:46 controller-2 etcd[77757]: ready to serve client requests
Jan 16 11:09:46 controller-2 etcd[77757]: setting up the initial cluster version to 3.4
Jan 16 11:09:46 controller-2 etcd[77757]: ready to serve client requests
Jan 16 11:09:46 controller-2 systemd[1]: Started Etcd Server.
Jan 16 11:09:46 controller-2 etcd[77757]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Jan 16 11:09:46 controller-2 etcd[77757]: serving client requests on 192.168.50.225:2379
Jan 16 11:09:46 controller-2 etcd[77757]: set the initial cluster version to 3.4
Jan 16 11:09:46 controller-2 etcd[77757]: enabled capabilities for version 3.4

集群健康监测

1 /usr/bin/etcdctl --ca-file=/root/cert/ca.pem --cert-file=/root/cert/etcd.pem --key-file=/root/cert/etcd-key.pem --endpoints="https://192.168.50.220:2379,https://192.168.50.221:2379,https://192.168.50.222:2379" cluster-health
2 member 1d0f59292d7002a6 is healthy: got healthy result from https://192.168.50.221:2379
3 member c3b43c4124d987ee is healthy: got healthy result from https://192.168.50.220:2379
4 member fd26658e8648d5e7 is healthy: got healthy result from https://192.168.50.222:2379
5 cluster is healthy

列出集群内所有节点

]# /usr/bin/etcdctl --ca-file=/root/cert/ca.pem --cert-file=/root/cert/etcd.pem --key-file=/root/cert/etcd-key.pem --endpoints="https://192.168.50.220:2379,https://192.168.50.221:2379,https://192.168.50.222:2379"  member list
1d0f59292d7002a6: name=etcd-cluster2 peerURLs=https://192.168.50.221:2380 clientURLs=https://192.168.50.221:2379 isLeader=false
c3b43c4124d987ee: name=etcd-cluster1 peerURLs=https://192.168.50.220:2380 clientURLs=https://192.168.50.220:2379 isLeader=false
fd26658e8648d5e7: name=etcd-cluster3 peerURLs=https://192.168.50.222:2380 clientURLs=https://192.168.50.222:2379 isLeader=true

 

posted @   不会跳舞的胖子  阅读(4)  评论(0)  编辑  收藏  举报
相关博文：
· k8s etcd定时备份与恢复
· k8s StatefulSet + Headless Service 部署reids集群
· 二进制部署etcd集群（企业级）
· etcd集群创建+ssl证书
· 二进制部署etcd-三个集群方案
阅读排行：
· 被坑几百块钱后，我竟然真的恢复了删除的微信聊天记录！
· 没有Manus邀请码？试试免邀请码的MGX或者开源的OpenManus吧
· 【自荐】一款简洁、开源的在线白板工具 Drawnix
· 园子的第一款AI主题卫衣上架——"HELLO! HOW CAN I ASSIST YOU TODAY
· Docker 太简单，K8s 太复杂？w7panel 让容器管理更轻松！
点击右上角即可分享
微信分享提示