二进制部署K8S-2集群部署

感谢老男孩教育王导的公开视频，文档整理自https://www.yuque.com/duduniao/k8s。

因为在后期运行容器需要有大量的物理硬件资源使用的环境是用的Wmware EXSI如果使用笔记本可适当调整虚拟机配置。

角色	主机名	IP	CPU	Mem	OS
(主)负载均衡、内网DNS	hdss7-11.host.com	10.4.7.11	4C	4G	CentOS Linux release 7.8
(备)负载均衡、内网DNS	hdss7-12.host.com	10.4.7.12	4C	4G	CentOS Linux release 7.8
k8s master、k8s node、etcd	hdss7-21.host.com	10.4.7.21	4C	8G	CentOS Linux release 7.8
k8s master、k8s node、etcd	hdss7-22.host.com	10.4.7.22	4C	8G	CentOS Linux release 7.8
harbor私有仓库、nfs	hdss7-200.host.com	10.4.7.200	4C	4G	CentOS Linux release 7.8

2. 安装前准备

2.1. 环境准备

所有机器都需要执行

[root@hdss7-11 ~]# systemctl stop firewalld
[root@hdss7-11 ~]# systemctl disable firewalld
[root@hdss7-11 ~]# setenforce 0
[root@hdss7-11 ~]# sed -ir '/^SELINUX=/s/=.+/=disabled/' /etc/selinux/config

[root@hdss7-11 ~]# yum install -y epel-release
[root@hdss7-11 ~]# yum install -y wget net-tools telnet tree nmap sysstat lrzsz dos2unix bind-utils vim less ntpdate

# 所有机器时间同步，生产环境需要有时间同步服务器。
ntpdate ntp1.aliyun.com

2.2. bind安装

2.2.1. hdss7-11 安装bind

[root@hdss7-11 ~]# yum install -y bind

2.2.2. hdss7-11 配置bind

主配置文件

[root@hdss7-11 ~]# vim /etc/named.conf  # 确保以下配置正确
  listen-on port 53 { 10.4.7.11; };
    directory   "/var/named";
    allow-query     { any; };
  forwarders      { 10.4.7.254; };      # 上级DNS，需要手动添加
  recursion yes;
  dnssec-enable no;
  dnssec-validation no;

在 hdss7-11.host.com 配置区域文件

# 增加两个zone配置，od.com为业务域，host.com.zone为主机域

[root@hdss7-11 ~]# cat>>/etc/named.rfc1912.zones<<EOF  
zone "host.com" IN {
        type  master;
        file  "host.com.zone";
        allow-update { 10.4.7.11; };
};

zone "od.com" IN {
        type  master;
        file  "od.com.zone";
        allow-update { 10.4.7.11; };
};
EOF

在 hdss7-11.host.com 配置主机域文件

# line6中时间需要修改
[root@hdss7-11 ~]# vim /var/named/host.com.zone
$ORIGIN host.com.
$TTL 600    ; 10 minutes
@       IN SOA  dns.host.com. dnsadmin.host.com. (
                2021043001 ; serial
                10800      ; refresh (3 hours)
                900        ; retry (15 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
            NS   dns.host.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11
HDSS7-11           A    10.4.7.11
HDSS7-12           A    10.4.7.12
HDSS7-21           A    10.4.7.21
HDSS7-22           A    10.4.7.22
HDSS7-200          A    10.4.7.200

在 hdss7-11.host.com 配置业务域文件

[root@hdss7-11 ~]# vim /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600    ; 10 minutes
@           IN SOA  dns.od.com. dnsadmin.od.com. (
                2021043001 ; serial
                10800      ; refresh (3 hours)
                900        ; retry (15 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
                NS   dns.od.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11

在 hdss7-11.host.com 启动bind服务，并测试

[root@hdss7-11 ~]# named-checkconf  # 检查配置文件
[root@hdss7-11 ~]# systemctl start named ; systemctl enable named
[root@hdss7-11 ~]# host HDSS7-200 10.4.7.11
Using domain server:
Name: 10.4.7.11
Address: 10.4.7.11#53
Aliases: 

HDSS7-200.host.com has address 10.4.7.200

2.2.3. 修改宿主机DNS

修改所有主机的dns服务器地址

[root@hdss7-11 ~]# sed -i '/DNS1/s/10.4.7.254/10.4.7.11/' /etc/sysconfig/network-scripts/ifcfg-ens32
[root@hdss7-11 ~]# systemctl restart network
[root@hdss7-11 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search host.com
nameserver 10.4.7.11

这里的客户机是windows，直接更改的hosts

10.4.7.11              HDSS7-11.host.com
10.4.7.12              HDSS7-12.host.com
10.4.7.21              HDSS7-21.host.com
10.4.7.22              HDSS7-22.host.com
10.4.7.200             HDSS7-200.host.com harbor.od.com k8s-yaml.od.com wlw.od.com
10.4.7.10              traefik.od.com dashboard.od.com jenkins.od.com dubbo-monitor.od.com demo.od.com

2.3. 根证书准备

在 hdss7-200 下载工具

[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssl-json
[root@hdss7-200 ~]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfo
[root@hdss7-200 ~]# chmod u+x /usr/local/bin/cfssl*

在 hdss7-200 签发根证书

[root@hdss7-200 ~]# mkdir /opt/certs/ ; cd /opt/certs/
# 根证书配置：
# CN 一般写域名，浏览器会校验
# names 为地区和公司信息
# expiry 为过期时间
[root@hdss7-200 certs]# cat>>/opt/certs/ca-csr.json<<EOF
{
    "CN": "OldboyEdu",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ],
    "ca": {
        "expiry": "175200h"
    }
}
EOF

[root@hdss7-200 certs]# cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
2020/01/05 10:42:07 [INFO] generating a new CA key and certificate from CSR
2020/01/05 10:42:07 [INFO] generate received request
2020/01/05 10:42:07 [INFO] received CSR
2020/01/05 10:42:07 [INFO] generating key: rsa-2048
2020/01/05 10:42:08 [INFO] encoded CSR
2020/01/05 10:42:08 [INFO] signed certificate with serial number 451005524427475354617025362003367427117323539780

[root@hdss7-200 certs]# ls -l ca*
-rw-r--r-- 1 root root  993 Jan  5 10:42 ca.csr
-rw-r--r-- 1 root root  328 Jan  5 10:39 ca-csr.json
-rw------- 1 root root 1675 Jan  5 10:42 ca-key.pem
-rw-r--r-- 1 root root 1346 Jan  5 10:42 ca.pem

2.4. docker环境准备

需要安装docker的机器：hdss7-21 hdss7-22 hdss7-200，以hdss7-21为例

[root@hdss7-21 ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@hdss7-21 ~]# yum install -y docker-ce

# 不安全的registry中增加了harbor地址
# 各个机器上bip网段不一致，bip中间两段与宿主机最后两段相同，目的是方便定位问题 
[root@hdss7-21 ~]# mkdir /etc/docker/
[root@hdss7-21 ~]# vim /etc/docker/daemon.json
{
  "graph": "/data/docker",
  "storage-driver": "overlay2",
  "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.od.com"],
  "registry-mirrors": ["https://registry.docker-cn.com"],
  "bip": "172.7.21.1/24",
  "exec-opts": ["native.cgroupdriver=systemd"],
  "live-restore": true
}
[root@hdss7-21 ~]# mkdir /data/docker
[root@hdss7-21 ~]# systemctl start docker ; systemctl enable docker

2.5. harbor安装

参考地址：https://www.yuque.com/duduniao/trp3ic/ohrxds#9Zpxx

官方地址：https://goharbor.io/

下载地址：https://github.com/goharbor/harbor/releases

2.5.1. hdss7-200 安装harbor

# 目录说明：
# /opt/src : 源码、文件下载目录
# /opt/release : 各个版本软件存放位置
# /opt/apps : 各个软件当前版本的软链接
[root@hdss7-200 ~]# mkdir /opt/{src,apps,release}
[root@hdss7-200 ~]# cd /opt/src
[root@hdss7-200 src]# wget https://github.com/goharbor/harbor/releases/download/v1.9.4/harbor-offline-installer-v1.9.4.tgz
[root@hdss7-200 src]# tar -xvf harbor-offline-installer-v1.9.4.tgz
[root@hdss7-200 src]# mv harbor /opt/release/harbor-v1.9.4
[root@hdss7-200 src]# ln -s /opt/release/harbor-v1.9.4 /opt/apps/harbor
[root@hdss7-200 src]# ll /opt/apps/
total 0
lrwxrwxrwx 1 root root 26 Jan  5 11:13 harbor -> /opt/release/harbor-v1.9.4

# 实验环境仅修改以下配置项，生产环境还得修改密码
[root@hdss7-200 src]# vim /opt/apps/harbor/harbor.yml
hostname: harbor.od.com
http:
  port: 180
data_volume: /data/harbor
location: /data/harbor/logs
[root@hdss7-200 src]# yum install -y docker-compose
[root@hdss7-200 src]# cd /opt/apps/harbor/
[root@hdss7-200 harbor]# ./install.sh 
......
✔ ----Harbor has been installed and started successfully.----
[root@hdss7-200 harbor]# docker-compose ps 
      Name                     Command               State             Ports          
--------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up                               
harbor-db           /docker-entrypoint.sh            Up      5432/tcp                 
harbor-jobservice   /harbor/harbor_jobservice  ...   Up                               
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up      127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up      8080/tcp                 
nginx               nginx -g daemon off;             Up      0.0.0.0:180->8080/tcp    
redis               redis-server /etc/redis.conf     Up      6379/tcp                 
registry            /entrypoint.sh /etc/regist ...   Up      5000/tcp                 
registryctl         /harbor/start.sh                 Up

设置harbor开机启动

 # 增加以下内容
[root@hdss7-200 harbor]# cat>>/etc/rc.d/rc.local<<EOF 
# start harbor
cd /opt/apps/harbor
/usr/bin/docker-compose stop
/usr/bin/docker-compose start
EOF

# 注意要给执行权限
chmod +x /etc/rc.d/rc.local

2.5.2. hdss7-200 安装nginx

安装Nginx反向代理harbor

# 当前机器中Nginx功能较少，使用yum安装即可。如有多个harbor考虑源码编译且配置健康检查
# nginx配置此处忽略，仅仅使用最简单的配置。
[root@hdss7-200 ~]# yum -y install nginx

[root@hdss7-200 harbor]# cat>>/etc/nginx/conf.d/harbor.conf<<EOF
server {
    listen       80;
    server_name  harbor.od.com;
    # 避免出现上传失败的情况
    client_max_body_size 1000m;

    location / {
        proxy_pass http://127.0.0.1:180;
    }
}
EOF

[root@hdss7-200 harbor]# systemctl start nginx ; systemctl enable nginx

hdss7-11 配置DNS解析

# serial序列号需要滚动一个,根据日期编写的。只是一个数字而已
[root@hdss7-11 ~]# vim /var/named/od.com.zone 
$ORIGIN od.com.
$TTL 600    ; 10 minutes
@           IN SOA  dns.od.com. dnsadmin.od.com. (
                2021043002 ; serial    # 更改这个数字
                10800      ; refresh (3 hours)
                900        ; retry (15 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
                NS   dns.od.com.
$TTL 60 ; 1 minute
dns                A    10.4.7.11
harbor             A    10.4.7.200
[root@hdss7-11 ~]# systemctl restart named.service  # reload 无法使得配置生效
[root@hdss7-11 ~]# host harbor.od.com
harbor.od.com has address 10.4.7.200

新建项目: public

测试harbor

[root@hdss7-200 ~]# docker pull nginx:1.17.9
[root@hdss7-200 ~]# docker tag nginx:1.17.9 harbor.od.com/public/nginx:v1.17.9
[root@hdss7-200 ~]# docker login -u admin harbor.od.com
                    # 默认密码: Harbor12345
[root@hdss7-200 ~]# docker image push harbor.od.com/public/nginx:v1.17.9

3. 主控节点安装

3.1. etcd安装

etcd 的leader选举机制，要求至少为3台或以上的奇数台。本次安装涉及：hdss7-12，hdss7-21，hdss7-22

3.1.1. 签发etcd证书

证书签发服务器 hdss7-200:

创建ca的json配置: /opt/certs/ca-config.json
- server 表示服务端连接客户端时携带的证书，用于客户端验证服务端身份
- client 表示客户端连接服务端时携带的证书，用于服务端验证客户端身份
- peer 表示相互之间连接时使用的证书，如etcd节点之间验证

cat>>/opt/certs/ca-config.json<<EOF
{
    "signing": {
        "default": {
            "expiry": "175200h"
        },
        "profiles": {
            "server": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF

创建etcd证书配置：/opt/certs/etcd-peer-csr.json

重点在hosts上，将所有可能的etcd服务器添加到host列表，不能使用网段，新增etcd服务器需要重新签发证书

cat>>/opt/certs/etcd-peer-csr.json<<EOF
{
    "CN": "k8s-etcd",
    "hosts": [
        "10.4.7.11",
        "10.4.7.12",
        "10.4.7.21",
        "10.4.7.22"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}
EOF

签发证书

[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
[root@hdss7-200 certs]# ll etcd-peer*
-rw-r--r-- 1 root root 1062 Jan  5 17:01 etcd-peer.csr
-rw-r--r-- 1 root root  363 Jan  5 16:59 etcd-peer-csr.json
-rw------- 1 root root 1675 Jan  5 17:01 etcd-peer-key.pem
-rw-r--r-- 1 root root 1428 Jan  5 17:01 etcd-peer.pem

3.1.2. 安装etcd

etcd地址：https://github.com/etcd-io/etcd/

实验使用版本: etcd-v3.1.20-linux-amd64.tar.gz

本次安装涉及：hdss7-12，hdss7-21，hdss7-22

下载etcd

[root@hdss7-12 ~]# useradd -s /sbin/nologin -M etcd
[root@hdss7-12 ~]# cd /opt/src/
[root@hdss7-12 src]# wget https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz
[root@hdss7-12 src]# tar -xf etcd-v3.1.20-linux-amd64.tar.gz 
[root@hdss7-12 src]# mv etcd-v3.1.20-linux-amd64 /opt/release/etcd-v3.1.20
[root@hdss7-12 src]# ln -s /opt/release/etcd-v3.1.20 /opt/apps/etcd
[root@hdss7-12 src]# ll /opt/apps/etcd
lrwxrwxrwx 1 root root 25 Jan  5 17:56 /opt/apps/etcd -> /opt/release/etcd-v3.1.20
[root@hdss7-12 src]# mkdir -p /opt/apps/etcd/certs /data/etcd /data/logs/etcd-server

下发证书到各个etcd上

[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# for i in 12 21 22;do scp ca.pem etcd-peer.pem etcd-peer-key.pem hdss7-${i}:/opt/apps/etcd/certs/ ;done

[root@hdss7-12 src]# md5sum /opt/apps/etcd/certs/*
8778d0c3411891af61a287e49a70c89a  /opt/apps/etcd/certs/ca.pem
7918783c2f6bf69e96edf03e67d04983  /opt/apps/etcd/certs/etcd-peer-key.pem
d4d849751a834c7727d42324fdedf92d  /opt/apps/etcd/certs/etcd-peer.pem

创建启动脚本(部分参数每台机器不同)

[root@hdss7-12 ~]# vim /opt/apps/etcd/etcd-server-startup.sh
#!/bin/sh
# listen-peer-urls etcd节点之间通信端口
# listen-client-urls 客户端与etcd通信端口
# quota-backend-bytes 配额大小
# 需要修改的参数：name,listen-peer-urls,listen-client-urls,initial-advertise-peer-urls

WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/etcd/etcd --name etcd-server-7-12 \
    --data-dir /data/etcd/etcd-server \
    --listen-peer-urls https://10.4.7.12:2380 \
    --listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
    --quota-backend-bytes 8000000000 \
    --initial-advertise-peer-urls https://10.4.7.12:2380 \
    --advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
    --initial-cluster  etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
    --ca-file ./certs/ca.pem \
    --cert-file ./certs/etcd-peer.pem \
    --key-file ./certs/etcd-peer-key.pem \
    --client-cert-auth  \
    --trusted-ca-file ./certs/ca.pem \
    --peer-ca-file ./certs/ca.pem \
    --peer-cert-file ./certs/etcd-peer.pem \
    --peer-key-file ./certs/etcd-peer-key.pem \
    --peer-client-cert-auth \
    --peer-trusted-ca-file ./certs/ca.pem \
    --log-output stdout

[root@hdss7-12 ~]# chmod u+x /opt/apps/etcd/etcd-server-startup.sh
[root@hdss7-12 ~]# chown -R etcd.etcd /opt/apps/etcd/ /data/etcd /data/logs/etcd-server

3.1.3. 启动etcd

因为这些进程都是要启动为后台进程，要么手动启动，要么采用后台进程管理工具，实验中使用后台管理工具

[root@hdss7-12 ~]# yum install -y supervisor
[root@hdss7-12 ~]# systemctl start supervisord ; systemctl enable supervisord
[root@hdss7-12 ~]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-12]
command=/opt/apps/etcd/etcd-server-startup.sh         
numprocs=1                                            
directory=/opt/apps/etcd                              
autostart=true                                        
autorestart=true                                      
startsecs=30                                         
startretries=3                                        
exitcodes=0,2                                         
stopsignal=QUIT                                       
stopwaitsecs=10                                      
user=etcd                                            
redirect_stderr=true                                 
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log 
stdout_logfile_maxbytes=64MB                          
stdout_logfile_backups=5                             
stdout_capture_maxbytes=1MB                           
stdout_events_enabled=false   
killasgroup=true
stopasgroup=true

[root@hdss7-12 ~]# mkdir /data/logs/etcd-server/ -p

[root@hdss7-12 ~]# supervisorctl update
etcd-server-7-12: added process group

etcd 进程状态查看

[root@hdss7-12 ~]# supervisorctl status  # supervisorctl 状态
etcd-server-7-12                 RUNNING   pid 22375, uptime 0:00:39

[root@hdss7-12 ~]# netstat -lntp|grep etcd
tcp        0      0 10.4.7.12:2379          0.0.0.0:*               LISTEN      22379/etcd          
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      22379/etcd          
tcp        0      0 10.4.7.12:2380          0.0.0.0:*               LISTEN      22379/etcd

[root@hdss7-12 ~]# /opt/apps/etcd/etcdctl member list # 随着etcd重启，leader会变化
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=true
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false

[root@hdss7-22 /opt/apps/etcd]# ./etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy

etcd 启停方式

[root@hdss7-12 ~]# supervisorctl start etcd-server-7-12
[root@hdss7-12 ~]# supervisorctl stop etcd-server-7-12
[root@hdss7-12 ~]# supervisorctl restart etcd-server-7-12
[root@hdss7-12 ~]# supervisorctl status etcd-server-7-12

3.2. apiserver 安装

3.2.1. 下载kubernetes服务端

aipserver 涉及的服务器：hdss7-21，hdss7-22

下载 kubernetes 二进制版本包需要FQ工具

进入kubernetes的github页面: https://github.com/kubernetes/kubernetes
进入tags页签: https://github.com/kubernetes/kubernetes/tags
选择要下载的版本: https://github.com/kubernetes/kubernetes/releases/tag/v1.15.2
点击 CHANGELOG-${version}.md 进入说明页面: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#downloads-for-v1152
下载Server Binaries: https://dl.k8s.io/v1.15.2/kubernetes-server-linux-amd64.tar.gz

[root@hdss7-21 ~]# cd /opt/src
[root@hdss7-21 src]# wget https://dl.k8s.io/v1.15.2/kubernetes-server-linux-amd64.tar.gz

[root@hdss7-21 src]# tar -xf kubernetes-server-linux-amd64.tar.gz 
[root@hdss7-21 src]# mv kubernetes /opt/release/kubernetes-v1.15.2
[root@hdss7-21 src]# ln -s /opt/release/kubernetes-v1.15.2 /opt/apps/kubernetes
[root@hdss7-21 src]# ll /opt/apps/kubernetes
lrwxrwxrwx 1 root root 31 Jan  6 12:59 /opt/apps/kubernetes -> /opt/release/kubernetes-v1.15.2

[root@hdss7-21 src]# cd /opt/apps/kubernetes
[root@hdss7-21 kubernetes]# rm -f kubernetes-src.tar.gz 
[root@hdss7-21 kubernetes]# cd server/bin/
[root@hdss7-21 bin]# rm -f *.tar *_tag  # *.tar *_tag 镜像文件
[root@hdss7-21 bin]# ll
total 884636
-rwxr-xr-x 1 root root  43534816 Aug  5 18:01 apiextensions-apiserver
-rwxr-xr-x 1 root root 100548640 Aug  5 18:01 cloud-controller-manager
-rwxr-xr-x 1 root root 200648416 Aug  5 18:01 hyperkube
-rwxr-xr-x 1 root root  40182208 Aug  5 18:01 kubeadm
-rwxr-xr-x 1 root root 164501920 Aug  5 18:01 kube-apiserver
-rwxr-xr-x 1 root root 116397088 Aug  5 18:01 kube-controller-manager
-rwxr-xr-x 1 root root  42985504 Aug  5 18:01 kubectl
-rwxr-xr-x 1 root root 119616640 Aug  5 18:01 kubelet
-rwxr-xr-x 1 root root  36987488 Aug  5 18:01 kube-proxy
-rwxr-xr-x 1 root root  38786144 Aug  5 18:01 kube-scheduler
-rwxr-xr-x 1 root root   1648224 Aug  5 18:01 mounter

3.2.2. 签发证书

签发证书涉及的服务器：hdss7-200

签发client证书（apiserver和etcd通信证书）

[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# cat>>/opt/certs/client-csr.json<<EOF
{
    "CN": "k8s-node",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}
EOF

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssl-json -bare client
2020/01/06 13:42:47 [INFO] generate received request
2020/01/06 13:42:47 [INFO] received CSR
2020/01/06 13:42:47 [INFO] generating key: rsa-2048
2020/01/06 13:42:47 [INFO] encoded CSR
2020/01/06 13:42:47 [INFO] signed certificate with serial number 268276380983442021656020268926931973684313260543
2020/01/06 13:42:47 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@hdss7-200 certs]# ls client* -l
-rw-r--r-- 1 root root  993 Jan  6 13:42 client.csr
-rw-r--r-- 1 root root  280 Jan  6 13:42 client-csr.json
-rw------- 1 root root 1679 Jan  6 13:42 client-key.pem
-rw-r--r-- 1 root root 1363 Jan  6 13:42 client.pem

签发server证书（apiserver和其它k8s组件通信使用）

# hosts中将所有可能作为apiserver的ip添加进去，VIP 10.4.7.10 也要加入
[root@hdss7-200 certs]# cat>>/opt/certs/apiserver-csr.json<<EOF
{
    "CN": "k8s-apiserver",
    "hosts": [
        "127.0.0.1",
        "192.168.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "10.4.7.10",
        "10.4.7.21",
        "10.4.7.22",
        "10.4.7.23"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}
EOF

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json |cfssl-json -bare apiserver
2020/01/06 13:46:56 [INFO] generate received request
2020/01/06 13:46:56 [INFO] received CSR
2020/01/06 13:46:56 [INFO] generating key: rsa-2048
2020/01/06 13:46:56 [INFO] encoded CSR
2020/01/06 13:46:56 [INFO] signed certificate with serial number 573076691386375893093727554861295529219004473872
2020/01/06 13:46:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@hdss7-200 certs]# ls apiserver* -l
-rw-r--r-- 1 root root 1249 Jan  6 13:46 apiserver.csr
-rw-r--r-- 1 root root  566 Jan  6 13:45 apiserver-csr.json
-rw------- 1 root root 1675 Jan  6 13:46 apiserver-key.pem
-rw-r--r-- 1 root root 1598 Jan  6 13:46 apiserver.pem

证书下发

[root@hdss7-200 certs]# for i in 21 22;do echo hdss7-$i;ssh hdss7-$i "mkdir /opt/apps/kubernetes/server/bin/certs";scp apiserver-key.pem apiserver.pem ca-key.pem ca.pem client-key.pem client.pem hdss7-$i:/opt/apps/kubernetes/server/bin/certs/;done

3.2.3. 配置apiserver日志审计

aipserver 涉及的服务器：hdss7-21，hdss7-22

[root@hdss7-21 bin]# mkdir /opt/apps/kubernetes/conf

# 打开文件后，设置 :set paste，避免自动缩进
[root@hdss7-21 bin]# \vi /opt/apps/kubernetes/conf/audit.yaml
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

3.2.4. 配置启动脚本

aipserver 涉及的服务器：hdss7-21，hdss7-22

创建启动脚本

[root@hdss7-21 bin]# vim /opt/apps/kubernetes/server/bin/kube-apiserver-startup.sh
#!/bin/bash

WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-apiserver \
    --apiserver-count 2 \
    --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \
    --audit-policy-file ../../conf/audit.yaml \
    --authorization-mode RBAC \
    --client-ca-file ./certs/ca.pem \
    --requestheader-client-ca-file ./certs/ca.pem \
    --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
    --etcd-cafile ./certs/ca.pem \
    --etcd-certfile ./certs/client.pem \
    --etcd-keyfile ./certs/client-key.pem \
    --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
    --service-account-key-file ./certs/ca-key.pem \
    --service-cluster-ip-range 192.168.0.0/16 \
    --service-node-port-range 3000-29999 \
    --target-ram-mb=1024 \
    --kubelet-client-certificate ./certs/client.pem \
    --kubelet-client-key ./certs/client-key.pem \
    --log-dir  /data/logs/kubernetes/kube-apiserver \
    --tls-cert-file ./certs/apiserver.pem \
    --tls-private-key-file ./certs/apiserver-key.pem \
    --v 2
    
    # 添加可执行权限
    chmod +x /opt/release/kubernetes-v1.15.2/server/bin/kube-apiserver-startup.sh

配置supervisor启动配置

[root@hdss7-21 bin]# vim /etc/supervisord.d/kube-apiserver.ini
[program:kube-apiserver-7-21]
command=/opt/apps/kubernetes/server/bin/kube-apiserver-startup.sh
numprocs=1
directory=/opt/apps/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
killasgroup=true
stopasgroup=true

[root@hdss7-21 bin]# mkdir /data/logs/kubernetes/kube-apiserver/ -p

[root@hdss7-21 bin]# supervisorctl update
[root@hdss7-21 bin]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 23637, uptime 22:26:08
kube-apiserver-7-21              RUNNING   pid 32591, uptime 0:05:37

启停apiserver

[root@hdss7-12 ~]# supervisorctl start kube-apiserver-7-21
[root@hdss7-12 ~]# supervisorctl stop kube-apiserver-7-21
[root@hdss7-12 ~]# supervisorctl restart kube-apiserver-7-21
[root@hdss7-12 ~]# supervisorctl status kube-apiserver-7-21

查看进程

[root@hdss7-21 bin]# netstat -lntp|grep api
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      32595/kube-apiserve 
tcp6       0      0 :::6443                 :::*                    LISTEN      32595/kube-apiserve 
[root@hdss7-21 bin]# ps uax|grep kube-apiserver|grep -v grep
root      32591  0.0  0.0 115296  1476 ?        S    20:17   0:00 /bin/bash /opt/apps/kubernetes/server/bin/kube-apiserver-startup.sh
root      32595  3.0  2.3 402720 184892 ?       Sl   20:17   0:16 /opt/apps/kubernetes/server/bin/kube-apiserver --apiserver-count 2 --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log --audit-policy-file ../../conf/audit.yaml --authorization-mode RBAC --client-ca-file ./certs/ca.pem --requestheader-client-ca-file ./certs/ca.pem --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --etcd-cafile ./certs/ca.pem --etcd-certfile ./certs/client.pem --etcd-keyfile ./certs/client-key.pem --etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 --service-account-key-file ./certs/ca-key.pem --service-cluster-ip-range 192.168.0.0/16 --service-node-port-range 3000-29999 --target-ram-mb=1024 --kubelet-client-certificate ./certs/client.pem --kubelet-client-key ./certs/client-key.pem --log-dir /data/logs/kubernetes/kube-apiserver --tls-cert-file ./certs/apiserver.pem --tls-private-key-file ./certs/apiserver-key.pem --v 2

3.3. 配置apiserver L4代理

3.3.1. nginx配置

L4 代理涉及的服务器：hdss7-11，hdss7-12

[root@hdss7-11 ~]# yum install -y nginx nginx-mod-stream
[root@hdss7-11 ~]# vim /etc/nginx/nginx.conf  
# 末尾加上以下内容，stream 只能加在 main 中
# 此处只是简单配置下nginx，实际生产中，建议进行更合理的配置
stream {
    log_format proxy '$time_local|$remote_addr|$upstream_addr|$protocol|$status|'
                     '$session_time|$upstream_connect_time|$bytes_sent|$bytes_received|'
                     '$upstream_bytes_sent|$upstream_bytes_received' ;

    upstream kube-apiserver {
        server 10.4.7.21:6443     max_fails=3 fail_timeout=30s;
        server 10.4.7.22:6443     max_fails=3 fail_timeout=30s;
    }
    server {
        listen 7443;
        proxy_connect_timeout 2s;
        proxy_timeout 900s;
        proxy_pass kube-apiserver;
        access_log /var/log/nginx/proxy.log proxy;
    }
}

[root@hdss7-11 ~]# systemctl start nginx; systemctl enable nginx
[root@hdss7-11 ~]# curl 127.0.0.1:7443  # 测试几次
Client sent an HTTP request to an HTTPS server.
[root@hdss7-11 ~]# cat /var/log/nginx/proxy.log 
06/Jan/2020:21:00:27 +0800|127.0.0.1|10.4.7.21:6443|TCP|200|0.001|0.000|76|78|78|76
06/Jan/2020:21:05:03 +0800|127.0.0.1|10.4.7.22:6443|TCP|200|0.020|0.019|76|78|78|76
06/Jan/2020:21:05:04 +0800|127.0.0.1|10.4.7.21:6443|TCP|200|0.001|0.001|76|78|78|76

3.3.2. keepalived配置

aipserver L4 代理涉及的服务器：hdss7-11，hdss7-12

安装keepalive

[root@hdss7-11 ~]# yum install -y keepalived
[root@hdss7-11 ~]# vim /etc/keepalived/check_port.sh # 配置检查脚本
#!/bin/bash
if [ $# -eq 1 ] && [[ $1 =~ ^[0-9]+ ]];then
    [ $(netstat -lntp|grep ":$1 " |wc -l) -eq 0 ] && echo "[ERROR] nginx may be not running!" && exit 1 || exit 0
else
    echo "[ERROR] need one port!"
    exit 1
fi
[root@hdss7-11 ~]# chmod +x /etc/keepalived/check_port.sh

配置主节点：/etc/keepalived/keepalived.conf

主节点中，必须加上 nopreempt

因为一旦因为网络抖动导致VIP漂移，不能让它自动飘回来，必须要分析原因后手动迁移VIP到主节点！如主节点确认正常后，重启备节点的keepalive，让VIP飘到主节点.

keepalived 的日志输出配置此处省略，生产中需要进行处理。

vi /etc/keepalived/keepalived.conf 

! Configuration File for keepalived
global_defs {
   router_id 10.4.7.11
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 7443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 10.4.7.11
    nopreempt

    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        10.4.7.10
    }
}

配置备节点：/etc/keepalived/keepalived.conf

注意根据实际的网卡名称更改网卡名配置。

vi /etc/keepalived/keepalived.conf 

! Configuration File for keepalived
global_defs {
    router_id 10.4.7.12
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 7443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    virtual_router_id 251
    mcast_src_ip 10.4.7.12
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
        10.4.7.10
    }
}

启动keepalived

[root@hdss7-11 ~]# systemctl start keepalived ; systemctl enable keepalived
[root@hdss7-11 ~]# ip addr show ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6d:b8:82 brd ff:ff:ff:ff:ff:ff
    inet 10.4.7.11/24 brd 10.4.7.255 scope global noprefixroute ens32
       valid_lft forever preferred_lft forever
    inet 10.4.7.10/32 scope global ens32
       valid_lft forever preferred_lft forever
......

3.4. controller-manager 安装

controller-manager 涉及的服务器：hdss7-21，hdss7-22

controller-manager 设置为只调用当前机器的 apiserver，走127.0.0.1网卡，因此不配制SSL证书

[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kube-controller-manager-startup.sh
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-controller-manager \
    --cluster-cidr 172.7.0.0/16 \
    --leader-elect true \
    --log-dir /data/logs/kubernetes/kube-controller-manager \
    --master http://127.0.0.1:8080 \
    --service-account-private-key-file ./certs/ca-key.pem \
    --service-cluster-ip-range 192.168.0.0/16 \
    --root-ca-file ./certs/ca.pem \
    --v 2
[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kube-controller-manager-startup.sh

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-controller-manager.ini
[program:kube-controller-manager-7-21]
command=/opt/apps/kubernetes/server/bin/kube-controller-manager-startup.sh                     
numprocs=1                                                                        
directory=/opt/apps/kubernetes/server/bin                                             
autostart=true                                                                    
autorestart=true                                                                  
startsecs=30                                                                      
startretries=3                                                                    
exitcodes=0,2                                                                     
stopsignal=QUIT                                                                   
stopwaitsecs=10                                                                  
user=root                                                                        
redirect_stderr=true                                                             
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log  
stdout_logfile_maxbytes=64MB                                                      
stdout_logfile_backups=4                                                        
stdout_capture_maxbytes=1MB                                                     
stdout_events_enabled=false   
killasgroup=true
stopasgroup=true

[root@hdss7-21 ~]# mkdir /data/logs/kubernetes/kube-controller-manager
[root@hdss7-21 ~]# supervisorctl update
kube-controller-manager-7-21: stopped
kube-controller-manager-7-21: updated process group
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 23637, uptime 1 day, 0:16:54
kube-apiserver-7-21              RUNNING   pid 32591, uptime 1:56:23
kube-controller-manager-7-21     RUNNING   pid 33357, uptime 0:00:38

3.5. kube-scheduler安装

kube-scheduler 涉及的服务器：hdss7-21，hdss7-22

kube-scheduler 设置为只调用当前机器的 apiserver，走127.0.0.1网卡，因此不配制SSL证书

[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kube-scheduler-startup.sh
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-scheduler \
    --leader-elect  \
    --log-dir /data/logs/kubernetes/kube-scheduler \
    --master http://127.0.0.1:8080 \
    --v 2
[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kube-scheduler-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-scheduler

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-scheduler.ini
[program:kube-scheduler-7-21]
command=/opt/apps/kubernetes/server/bin/kube-scheduler-startup.sh                     
numprocs=1                                                               
directory=/opt/apps/kubernetes/server/bin                                     
autostart=true                                                           
autorestart=true                                                         
startsecs=30                                                             
startretries=3                                                           
exitcodes=0,2                                                            
stopsignal=QUIT                                                          
stopwaitsecs=10                                                          
user=root                                                                
redirect_stderr=true                                                     
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log 
stdout_logfile_maxbytes=64MB                                             
stdout_logfile_backups=4                                                 
stdout_capture_maxbytes=1MB                                              
stdout_events_enabled=false
killasgroup=true
stopasgroup=true

[root@hdss7-21 ~]# supervisorctl update
kube-scheduler-7-21: stopped
kube-scheduler-7-21: updated process group
[root@hdss7-21 ~]# supervisorctl status 
etcd-server-7-21                 RUNNING   pid 23637, uptime 1 day, 0:26:53
kube-apiserver-7-21              RUNNING   pid 32591, uptime 2:06:22
kube-controller-manager-7-21     RUNNING   pid 33357, uptime 0:10:37
kube-scheduler-7-21              RUNNING   pid 33450, uptime 0:01:18

3.6. 检查主控节点状态

[root@hdss7-21 ~]# ln -s /opt/apps/kubernetes/server/bin/kubectl /usr/local/bin/
[root@hdss7-21 ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-1               Healthy   {"health": "true"}   
etcd-0               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}

[root@hdss7-22 ~]# ln -s /opt/apps/kubernetes/server/bin/kubectl /usr/local/bin/
[root@hdss7-22 ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-2               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-0               Healthy   {"health": "true"}

4. 运算节点部署

4.1. kubelet 部署

4.1.1. 签发证书

证书签发在 hdss7-200 操作

[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# vim kubelet-csr.json # 将所有可能的kubelet机器IP添加到hosts中
{
    "CN": "k8s-kubelet",
    "hosts": [
    "127.0.0.1",
    "10.4.7.10",
    "10.4.7.21",
    "10.4.7.22",
    "10.4.7.23",
    "10.4.7.24",
    "10.4.7.25",
    "10.4.7.26",
    "10.4.7.27",
    "10.4.7.28"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
2020/01/06 23:10:56 [INFO] generate received request
2020/01/06 23:10:56 [INFO] received CSR
2020/01/06 23:10:56 [INFO] generating key: rsa-2048
2020/01/06 23:10:56 [INFO] encoded CSR
2020/01/06 23:10:56 [INFO] signed certificate with serial number 61221942784856969738771370531559555767101820379
2020/01/06 23:10:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@hdss7-200 certs]# ls kubelet* -l
-rw-r--r-- 1 root root 1115 Jan  6 23:10 kubelet.csr
-rw-r--r-- 1 root root  452 Jan  6 23:10 kubelet-csr.json
-rw------- 1 root root 1675 Jan  6 23:10 kubelet-key.pem
-rw-r--r-- 1 root root 1468 Jan  6 23:10 kubelet.pem

[root@hdss7-200 certs]# for i in 21 22;do scp kubelet.pem kubelet-key.pem hdss7-$i:/opt/apps/kubernetes/server/bin/certs/;done

4.1.2. 创建kubelet配置

kubelet配置在 hdss7-21 hdss7-22 操作

set-cluster # 创建需要连接的集群信息，可以创建多个k8s集群信息

[root@hdss7-21 ~]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/apps/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig

set-credentials # 创建用户账号，即用户登陆使用的客户端私有和证书，可以创建多个证书

[root@hdss7-21 ~]# kubectl config set-credentials k8s-node \
--client-certificate=/opt/apps/kubernetes/server/bin/certs/client.pem \
--client-key=/opt/apps/kubernetes/server/bin/certs/client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig

set-context # 设置context，即确定账号和集群对应关系

[root@hdss7-21 ~]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=k8s-node \
--kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig

use-context # 设置当前使用哪个context

[root@hdss7-21 ~]# kubectl config use-context myk8s-context --kubeconfig=/opt/apps/kubernetes/conf/kubelet.kubeconfig

4.1.3. 授权k8s-node用户

此步骤只需要在一台master节点执行

授权 k8s-node 用户绑定集群角色 system:node ，让 k8s-node 成为具备运算节点的权限。

[root@hdss7-21 ~]# vim k8s-node.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: k8s-node
[root@hdss7-21 ~]# kubectl create -f k8s-node.yaml 
clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
[root@hdss7-21 ~]# kubectl get clusterrolebinding k8s-node
NAME       AGE
k8s-node   36s

4.1.4. 装备pause镜像

将pause镜像放入到harbor私有仓库中，仅在 hdss7-200 操作：

[root@hdss7-200 ~]# docker image pull kubernetes/pause
[root@hdss7-200 ~]# docker image tag kubernetes/pause:latest harbor.od.com/public/pause:latest
[root@hdss7-200 ~]# docker login -u admin harbor.od.com
[root@hdss7-200 ~]# docker image push harbor.od.com/public/pause:latest

4.1.5. 创建启动脚本

在node节点创建脚本并启动kubelet，涉及服务器： hdss7-21 hdss7-22

[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kubelet-startup.sh
#!/bin/sh
# 注意修改主机名
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kubelet \
    --anonymous-auth=false \
    --cgroup-driver systemd \
    --cluster-dns 192.168.0.2 \
    --cluster-domain cluster.local \
    --runtime-cgroups=/systemd/system.slice \
    --kubelet-cgroups=/systemd/system.slice \
    --fail-swap-on="false" \
    --client-ca-file ./certs/ca.pem \
    --tls-cert-file ./certs/kubelet.pem \
    --tls-private-key-file ./certs/kubelet-key.pem \
    --hostname-override hdss7-21.host.com \
    --image-gc-high-threshold 20 \
    --image-gc-low-threshold 10 \
    --kubeconfig ../../conf/kubelet.kubeconfig \
    --log-dir /data/logs/kubernetes/kube-kubelet \
    --pod-infra-container-image harbor.od.com/public/pause:latest \
    --root-dir /data/kubelet
[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kubelet-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-21]
command=/opt/apps/kubernetes/server/bin/kubelet-startup.sh
numprocs=1
directory=/opt/apps/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
killasgroup=true
stopasgroup=true

[root@hdss7-21 ~]# supervisorctl update
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 23637, uptime 1 day, 14:56:25
kube-apiserver-7-21              RUNNING   pid 32591, uptime 16:35:54
kube-controller-manager-7-21     RUNNING   pid 33357, uptime 14:40:09
kube-kubelet-7-21                RUNNING   pid 37232, uptime 0:01:08
kube-scheduler-7-21              RUNNING   pid 33450, uptime 14:30:50
[root@hdss7-21 ~]# kubectl get node
NAME                STATUS   ROLES    AGE     VERSION
hdss7-21.host.com   Ready    <none>   3m13s   v1.15.2
hdss7-22.host.com   Ready    <none>   3m13s   v1.15.2

4.1.6. 修改节点角色

使用 kubectl get nodes 获取的Node节点角色为空，可以按照以下方式修改,这个操作只是在节点上面打了个标签。

[root@hdss7-21 ~]# kubectl get node
NAME                STATUS   ROLES    AGE     VERSION
hdss7-21.host.com   Ready    <none>   3m13s   v1.15.2
hdss7-22.host.com   Ready    <none>   3m13s   v1.15.2
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/node=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl get node
NAME                STATUS   ROLES         AGE     VERSION
hdss7-21.host.com   Ready    master,node   7m44s   v1.15.2
hdss7-22.host.com   Ready    master,node   7m44s   v1.15.2

4.2. kube-proxy部署

4.2.1. 签发证书

证书签发在 hdss7-200 操作

[root@hdss7-200 ~]# cd /opt/certs/
# CN 其实是k8s中的角色
[root@hdss7-200 certs]# cat>>kube-proxy-csr.json<<EOF 
{
    "CN": "system:kube-proxy",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}
EOF

[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client
2020/01/07 21:45:53 [INFO] generate received request
2020/01/07 21:45:53 [INFO] received CSR
2020/01/07 21:45:53 [INFO] generating key: rsa-2048
2020/01/07 21:45:53 [INFO] encoded CSR
2020/01/07 21:45:53 [INFO] signed certificate with serial number 620191685968917036075463174423999296907693104226
2020/01/07 21:45:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

# 因为kube-proxy使用的用户是kube-proxy，不能使用client证书，必须要重新签发自己的证书
[root@hdss7-200 certs]# ls kube-proxy-c* -l  
-rw-r--r-- 1 root root 1005 Jan  7 21:45 kube-proxy-client.csr
-rw------- 1 root root 1675 Jan  7 21:45 kube-proxy-client-key.pem
-rw-r--r-- 1 root root 1375 Jan  7 21:45 kube-proxy-client.pem
-rw-r--r-- 1 root root  267 Jan  7 21:45 kube-proxy-csr.json

[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-21:/opt/apps/kubernetes/server/bin/certs/                                                                         100% 1375   870.6KB/s   00:00    
[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-22:/opt/apps/kubernetes/server/bin/certs/

for i in 21 22;do scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-$i:/opt/apps/kubernetes/server/bin/certs/;done

4.2.2. 创建kube-proxy配置

在所有node节点创建，涉及服务器：hdss7-21 ，hdss7-22

[root@hdss7-21 ~]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/apps/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig
  
[root@hdss7-21 ~]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/apps/kubernetes/server/bin/certs/kube-proxy-client.pem \
--client-key=/opt/apps/kubernetes/server/bin/certs/kube-proxy-client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig
  
[root@hdss7-21 ~]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=kube-proxy \
--kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig
  
[root@hdss7-21 ~]# kubectl config use-context myk8s-context --kubeconfig=/opt/apps/kubernetes/conf/kube-proxy.kubeconfig

4.2.3. 加载ipvs模块

kube-proxy 共有3种流量调度模式，分别是 namespace，iptables，ipvs，其中ipvs性能最好。

[root@hdss7-21 ~]# for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; /sbin/modinfo -F filename $i >/dev/null 2>&1 && /sbin/modprobe $i;done
[root@hdss7-21 ~]# lsmod | grep ip_vs  # 查看ipvs模块

4.2.4. 创建启动脚本

# 注意更改主机名
[root@hdss7-21 ~]# vim /opt/apps/kubernetes/server/bin/kube-proxy-startup.sh
#!/bin/sh

WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/apps/kubernetes/server/bin/kube-proxy \
  --cluster-cidr 172.7.0.0/16 \
  --hostname-override hdss7-21.host.com \
  --proxy-mode=ipvs \
  --ipvs-scheduler=nq \
  --kubeconfig ../../conf/kube-proxy.kubeconfig
[root@hdss7-21 ~]# chmod u+x /opt/apps/kubernetes/server/bin/kube-proxy-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-proxy
[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-proxy.ini
[program:kube-proxy-7-21]
command=/opt/apps/kubernetes/server/bin/kube-proxy-startup.sh                
numprocs=1                                                      
directory=/opt/apps/kubernetes/server/bin                            
autostart=true                                                  
autorestart=true                                                
startsecs=30                                                    
startretries=3                                                  
exitcodes=0,2                                                   
stopsignal=QUIT                                                 
stopwaitsecs=10                                                 
user=root                                                       
redirect_stderr=true                                            
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log
stdout_logfile_maxbytes=64MB                                    
stdout_logfile_backups=5                                       
stdout_capture_maxbytes=1MB                                     
stdout_events_enabled=false
killasgroup=true
stopasgroup=true

[root@hdss7-21 ~]# supervisorctl update

4.2.5. 验证集群

[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 23637, uptime 2 days, 0:27:18
kube-apiserver-7-21              RUNNING   pid 32591, uptime 1 day, 2:06:47
kube-controller-manager-7-21     RUNNING   pid 33357, uptime 1 day, 0:11:02
kube-kubelet-7-21                RUNNING   pid 37232, uptime 9:32:01
kube-proxy-7-21                  RUNNING   pid 47088, uptime 0:06:19
kube-scheduler-7-21              RUNNING   pid 33450, uptime 1 day, 0:01:43

[root@hdss7-21 ~]# yum install -y ipvsadm
[root@hdss7-21 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.1:443 nq
  -> 10.4.7.21:6443               Masq    1      0          0         
  -> 10.4.7.22:6443               Masq    1      0          0

cat >/root/nginx-ds.yaml <<'EOF'
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: harbor.od.com/public/nginx:v1.17.9
        ports:
        - containerPort: 80
EOF

[root@hdss7-21 ~]# kubectl apply -f nginx-ds.yaml
[root@hdss7-21 ~]# kubectl get pods -o wide
NAME             READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
nginx-ds-cfc8d   1/1     Running   0          64s   172.7.22.2   hdss7-22.host.com   <none>           <none>
nginx-ds-vd4mk   1/1     Running   0          64s   172.7.21.2   hdss7-21.host.com   <none>           <none>

[root@hdss7-21 ~]# curl -I 172.7.21.2
HTTP/1.1 200 OK
Server: nginx/1.17.6
Date: Tue, 07 Jan 2020 14:28:46 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 19 Nov 2019 12:50:08 GMT
Connection: keep-alive
ETag: "5dd3e500-264"
Accept-Ranges: bytes

[root@hdss7-21 ~]# curl -I 172.7.22.2  # 缺少网络插件，无法跨节点通信

posted @ 2021-05-06 12:27 Gshelldon 阅读(461) 评论(0) 编辑收藏举报

刷新页面返回顶部

黄褐色的菊花怪

大大怪将军！！！

二进制部署K8S-2集群部署

二进制部署K8S-2集群部署

2. 安装前准备

2.1. 环境准备

2.2. bind安装

2.2.1. hdss7-11 安装bind

2.2.2. hdss7-11 配置bind

2.2.3. 修改宿主机DNS

2.3. 根证书准备

2.4. docker环境准备

2.5. harbor安装

2.5.1. hdss7-200 安装harbor

2.5.2. hdss7-200 安装nginx

3. 主控节点安装

3.1. etcd安装

3.1.1. 签发etcd证书

3.1.2. 安装etcd

3.1.3. 启动etcd

3.2. apiserver 安装

3.2.1. 下载kubernetes服务端

3.2.2. 签发证书

3.2.3. 配置apiserver日志审计

3.2.4. 配置启动脚本

3.3. 配置apiserver L4代理

3.3.1. nginx配置

3.3.2. keepalived配置

3.4. controller-manager 安装

3.5. kube-scheduler安装

3.6. 检查主控节点状态

4. 运算节点部署

4.1. kubelet 部署

4.1.1. 签发证书

4.1.2. 创建kubelet配置

4.1.3. 授权k8s-node用户

4.1.4. 装备pause镜像

4.1.5. 创建启动脚本

4.1.6. 修改节点角色

4.2. kube-proxy部署

4.2.1. 签发证书

4.2.2. 创建kube-proxy配置

4.2.3. 加载ipvs模块

4.2.4. 创建启动脚本

4.2.5. 验证集群

公告