二进制部署k8s集群v1.23.9版本-4-安装etcd集群

4.1、etcd集群规划

主机名	IP	角色
hfqg1-201	192.168.1.201	leader
hfqg1-202	192.168.1.202	flower
hfqg1-203	192.168.1.203	flower

4.2、创建自签证书

192.168.1.200服务器操作
创建etcd证书，客户端访问与节点互相访问使用同一套证书。

[root@hfqg1-200 certs]# cat etcd-csr.json
{
"CN": "k8s-etcd",
"hosts": [
"127.0.0.1",
"192.168.1.196",
"192.168.1.197",
"192.168.1.198",
"192.168.1.199",
"192.168.1.200",
"192.168.1.201",
"192.168.1.202",
"192.168.1.203"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "k8s",
"OU": "system"
}
]
}

说明：

CN：本处CN可随便定义

hosts：etcd安装的主机IP地址，必须是IP地址，不能是网段，可能的主机都列出来。
这里列出的主机和--advertise-client-urls定义有关
如果有新增主机不在列表中，需要重新签发证书

names中的配置：

C：国家

ST：州/省

L：市

O：组织，二进制部署随便定义，使用kubeadm时，要求值为system:masters

OU：部门

4.3、生成etcd证书和私钥

192.168.1.200服务器操作
cd /opt/certs

cfssl gencert
-ca=ca.pem
-ca-key=ca-key.pem
-config=ca-config.json
-profile=kubernetes
etcd-csr.json | cfssl-json -bare etcd

4.4、检查生成的证书私钥

192.168.1.200服务器操作

[root@hfqg1-200 certs]# ll etcd*
-rw-r--r-- 1 root root 1110 7月 28 10:37 etcd.csr
-rw-r--r-- 1 root root 507 7月 28 10:37 etcd-csr.json
-rw------- 1 root root 1679 7月 28 10:37 etcd-key.pem
-rw-r--r-- 1 root root 1480 7月 28 10:37 etcd.pem

4.5、创建etcd用户

192.168.1.201、192.168.1.202、192.168.1.203服务器操作
useradd -s /sbin/nologin -M etcd

4.6、下载软件、解压做软链接

192.168.1.201服务器操作
mkdir /opt/src
cd /opt/src
etcd下载地址
wget https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gz
tar xf etcd-v3.5.4-linux-amd64.tar.gz -C /opt
cd /opt
mv etcd-v3.5.4-linux-amd64 etcd-v3.5.4
ln -s etcd-v3.5.4 etcd
添加环境变量
vim /etc/profile
export PATH=$PATH:/opt/etcd
source /etc/profile
查看版本
etcd --verion

4.7、创建目录，拷贝证书私钥

192.168.1.201、192.168.1.202、192.168.1.203服务器操作
mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
201、202、203服务器上将200机器的etcd证书拷贝到/opt/etcd/certs目录下
如下图所示

注意：私钥的权限 400

4.8、创建etcd启动脚本

192.168.1.201服务器操作

cat /opt/etcd/etcd-server-startup.sh
#!/bin/sh
# listen-peer-urls etcd节点之间通信端口
# listen-client-urls 客户端与etcd通信端口
# quota-backend-bytes 配额大小
# 需要修改的参数：name,listen-peer-urls,listen-client-urls,initial-advertise-peer-urls
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/etcd/etcd --name etcd-server-1-201
--enable-v2=true
--data-dir /data/etcd/etcd-server
--listen-peer-urls https://192.168.1.201:2380
--listen-client-urls https://192.168.1.201:2379,http://127.0.0.1:2379
--quota-backend-bytes 8000000000
--initial-advertise-peer-urls https://192.168.1.201:2380
--advertise-client-urls https://192.168.1.201:2379,http://127.0.0.1:2379
--initial-cluster etcd-server-1-201=https://192.168.1.201:2380,etcd-server-1-202=https://192.168.1.202:2380,etcd-server-1-203=https://192.168.1.203:2380
--initial-cluster-token etcd-cluster-k8s
--initial-cluster-state new
--trusted-ca-file ./certs/ca.pem
--cert-file ./certs/etcd.pem
--key-file ./certs/etcd-key.pem
--client-cert-auth
--peer-trusted-ca-file ./certs/ca.pem
--peer-cert-file ./certs/etcd.pem
--peer-key-file ./certs/etcd-key.pem
--peer-client-cert-auth
--listen-metrics-urls=https://192.168.1.201:2381
--enable-pprof=false
--log-outputs stdout

etcd成员之间通信，2380端口
外部访问etcd，2379端口
参数说明:

name：etcd节点成员名称，在一个etcd集群中必须唯一性，可使用Hostname或者machine-id
data-dir：etcd数据保存目录
listen-peer-urls：和其它成员节点间通信地址，每个节点不同，必须使用IP，使用域名无效。
listen-client-urls：对外提供服务的地址，127.0.0.1允许非安全方式访问，使用域名无效。
initial-advertise-peer-urls：节点监听地址，集群成员使用该地址访问本节点，并会通告集群其它节点
initial-cluster：集群中所有节点信息，格式为：节点名称+监听的本地端口，多个节点用逗号隔开，即：name=https://initial-advertise-peer-urls
initial-cluster-state：加入集群的当前状态，new是新集群，existing表示加入已有集群
initial-cluster-token：集群引导创建期间所使用的TOKEN。
advertise-client-urls：节点成员客户端url列表，对外公告此节点客户端监听地址，可以使用域名
client-cert-auth：客户端访问本节点时，是否需要证书认证
trusted-ca-file：本节点2379使用的CA证书
cert-file：本节点2379所使用的证书
key-file：本节点2379所使用的密钥
peer-client-cert-auth：集群成员访问本节点时，是否需要证书认证
peer-trusted-ca-file：本节点2380所使用的CA证书
peer-cert-file：本节点2380所使用的证书
peer-key-file：本节点2380所使用的密钥
log-outputs：日志输出方式
listen-metrics-urls：metrics数据的获取地址

4.9、调整权限

192.168.1.201服务器
chmod +x etcd-server-startup.sh
chown -R etcd.etcd /opt/etcd-v3.5.4/ /data/etcd /data/logs/etcd-server/
chown -R etcd.etcd /opt/etcd

4.10、安装supervisor

201、202、203三台服务器
yum install supervisor -y
systemctl enable supervisord
systemctl start supervisord
systemctl status supervisord

4.11、创建etcd-server启动配置文件

201、202、203三台服务器操作

192.168.1.201服务器

[root@hfqg1-201 etcd]# cat /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-201]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=5 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)

192.168.0.202服务器
[root@hfqg1-202 etcd]# cat /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-202]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=5 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)

192.168.1.203服务器
[root@hfqg1-203 etcd]# cat /etc/supervisord.d/etcd-server.ini
[program:etcd-server-1-203]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=5 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)

4.12、启动服务并检查

192.168.1.201服务器操作
supervisorctl update
supervisorctl status

netstat -luntp | grep etcd

确保监听了2379、2380和2381三个端口。

4.13、安装部署集群其余节点

把201节点上已经安装好的etcd拷贝到202和203节点上
192.168.1.201服务器操作
cd /opt
scp -r etcd/ hfqg1-202:/opt/
scp -r etcd/ hfqg1-203:/opt/
192.168.1.202和192.168.1.203操作
chown -R etcd.etcd /opt/etcd/ /data/etcd /data/logs/etcd-server/
修改启动脚本中IP地址，如下所示：

202服务器
[root@hfqg1-202 ~]# cat /opt/etcd/etcd-server-startup.sh
#!/bin/sh
# listen-peer-urls etcd节点之间通信端口
# listen-client-urls 客户端与etcd通信端口
# quota-backend-bytes 配额大小
\ # 需要修改的参数：name,listen-peer-urls,listen-client-urls,initial-advertise-peer-urls
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/etcd/etcd --name etcd-server-1-202
--enable-v2=true
--data-dir /data/etcd/etcd-server
--listen-peer-urls https://192.168.1.202:2380
--listen-client-urls https://192.168.1.202:2379,http://127.0.0.1:2379
--quota-backend-bytes 8000000000
--initial-advertise-peer-urls https://192.168.1.202:2380
--advertise-client-urls https://192.168.1.202:2379,http://127.0.0.1:2379
--initial-cluster etcd-server-1-201=https://192.168.1.201:2380,etcd-server-1-202=https://192.168.1.202:2380,etcd-server-1-203=https://192.168.1.203:2380
--initial-cluster-token etcd-cluster-k8s
--initial-cluster-state new
--trusted-ca-file ./certs/ca.pem
--cert-file ./certs/etcd.pem
--key-file ./certs/etcd-key.pem
--client-cert-auth
--peer-trusted-ca-file ./certs/ca.pem
--peer-cert-file ./certs/etcd.pem
--peer-key-file ./certs/etcd-key.pem
--peer-client-cert-auth
--listen-metrics-urls=https://192.168.1.202:2381
--enable-pprof=false
--log-outputs stdout

203服务器
[root@hfqg1-203 ~]# cat /opt/etcd/etcd-server-startup.sh
#!/bin/sh
# listen-peer-urls etcd节点之间通信端口
# listen-client-urls 客户端与etcd通信端口
# quota-backend-bytes 配额大小
# 需要修改的参数：name,listen-peer-urls,listen-client-urls,initial-advertise-peer-urls
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/etcd/etcd --name etcd-server-1-203
--enable-v2=true
--data-dir /data/etcd/etcd-server
--listen-peer-urls https://192.168.1.203:2380
--listen-client-urls https://192.168.1.203:2379,http://127.0.0.1:2379
--quota-backend-bytes 8000000000
--initial-advertise-peer-urls https://192.168.1.203:2380
--advertise-client-urls https://192.168.1.203:2379,http://127.0.0.1:2379
--initial-cluster etcd-server-1-201=https://192.168.1.201:2380,etcd-server-1-202=https://192.168.1.202:2380,etcd-server-1-203=https://192.168.1.203:2380
--initial-cluster-token etcd-cluster-k8s
--initial-cluster-state new
--trusted-ca-file ./certs/ca.pem
--cert-file ./certs/etcd.pem
--key-file ./certs/etcd-key.pem
--client-cert-auth
--peer-trusted-ca-file ./certs/ca.pem
--peer-cert-file ./certs/etcd.pem
--peer-key-file ./certs/etcd-key.pem
--peer-client-cert-auth
--listen-metrics-urls=https://192.168.1.203:2381
--enable-pprof=false
--log-outputs stdout

分别启动etcd
supervisorctl update
supervisorctl status
202

203

4.14、检查集群状态

cd /opt/etcd
./etcdctl --cacert=/opt/etcd/certs/ca.pem --cert=/opt/etcd/certs/etcd.pem --key=/opt/etcd/certs/etcd-key.pem --endpoints=https://192.168.1.201:2379,https://192.168.1.202:2379,https://192.168.1.203:2379 endpoint health -w table

./etcdctl --cacert=/opt/etcd/certs/ca.pem --cert=/opt/etcd/certs/etcd.pem --key=/opt/etcd/certs/etcd-key.pem --endpoints=https://192.168.1.201:2379,https://192.168.1.202:2379,https://192.168.1.203:2379 endpoint status -w table

./etcdctl --endpoints="192.168.1.201:2379,192.168.1.202:2379,192.168.1.203:2379" --cacert=./certs/ca.pem --cert=./certs/etcd.pem --key=./certs/etcd-key.pem endpoint status --write-out=table

至此，etcd集群部署完成。

posted @ 2022-08-05 10:59 霸都运维阅读(313) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

夜深勿梦少年事

努力成为更好的自己！！！