基于Flannel的二进制安装k8s-1.26.1

Rocky Linux 9.1安装和基本设置#

下载镜像并最小化安装#

从Rocky Linux官网下载新版安装镜像ISO，传送门，具体安装过程省略，9.1的root账号默认锁定，并且不允许root用户使用密码进行SSH登录，在安装过程中需要

1. 取消锁定root账号
2. 允许root用户使用密码进行ssh登录

dnf update -y
reboot

$ uname -r
5.14.0-162.12.1.el9_1.0.2.x86_64

$ cat /etc/redhat-release 
Rocky Linux release 9.1 (Blue Onyx)

主机安全设置#

## 关闭swap分区
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab

## 禁用防火墙
systemctl stop firewalld
systemctl disable firewalld

## 禁用selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0

安装需要的软件和工具#

dnf install -y wget tree curl bash-completion jq vim net-tools telnet git lrzsz bridge-utils \
    chrony

设置NTP服务#

RL9.1默认安装chrony服务并且服务已经启动，如果需要修改NTP服务器可以修改配置文件/etc/chrony.conf

## 使用客户端进行验证
$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^+ tick.ntp.infomaniak.ch        1   7   373    98   +414us[ +414us] +/-  107ms
^* time.cloudflare.com           3   7   377   100  +1955us[+2064us] +/-  107ms
^+ time.cloudflare.com           3   7   367    99  -9897us[-9897us] +/-  117ms
^+ electrode.felixc.at           3   7   377    98  +1295us[+1295us] +/-  148ms

配置unlimit#

## 文件句柄
ulimit -SHn 65535

$ cat > /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

安装ipvsadmin#

dnf install -y ipvsadm ipset sysstat conntrack libseccomp

$ cat > /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

systemctl restart systemd-modules-load.service

$ lsmod | grep ip_vs
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 188416  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          176128  1 ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
libcrc32c              16384  3 nf_conntrack,xfs,ip_vs

修改内核参数#

$ cat > /etc/sysctl.d/95-k8s-sysctl.conf <<EOF 
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-arptables = 1

fs.may_detach_mounts = 1
vm.swappiness = 0
vm.overcommit_memory=1
vm.panic_on_oom=0
vm.max_map_count=655360
fs.inotify.max_user_watches=89100
fs.file-max=52706963				# 打开最大文件数
fs.nr_open=52706963
fs.aio-max-nr=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

$ sysctl --system

设置网卡IP#

使用nmtui图形化工具修改网卡ip地址为静态。

设置主机名称#

hostnamectl set-hostname m01 && bash
hostnamectl set-hostname w01 && bash
hostnamectl set-hostname w02 && bash

设定hosts文件，用于主机名解析#

$ cat >> /etc/hosts <<EOF
192.168.1.10 m01
192.168.1.11 w01
192.168.5.10 w02
EOF

环境规划#

对于学习来说，最主要的是使用k8s，熟悉原理以及操作，所以搭建最简单环境进行学习，学会后可以进行扩展到多master，多etcd以及前端添加lb等；2台worker节点之间跨子网，用于CNI跨子网的配置和验证。

本次部署默认是在m01节点上下载生成配置，然后将配置分发到2台worker节点。

部署架构#

m01和w01在相同的网段（192.168.1.0/24），w02在其他网段（192.168.5.0/24）,2个网段使用VyOS路由器互相连接起来

网络规划#

• 节点网络：192.168.1.0/24，192.168.5.0/24，跨不同子网的node节点
• service网络：10.96.0.0/16
• pod网络：172.16.0.0/16

主机规划#

主机名	IP地址	角色	安装组件
m01	192.168.1.10	master	apiserver，controller-manager，scheduler，etcd，kubectl，kubelet，kube-proxy，containerd
w01	192.168.1.11	worker	kubelet，kube-proxy，containerd
w02	192.168.5.10	worker	kubelet，kube-proxy，containerd

软件版本#

• kubernets：1.26.1
• etcd：3.5.7
• cfssl：1.6.3
• coredns：1.10.0
• metrics-server：0.6.1
• containerd：1.6.16
• runc：1.1.4

基本设定#

配置免密认证#

## 安装sshpass
dnf install -y sshpass

## 生成密钥文件
ssh-keygen -f /root/.ssh/id_rsa -P ''

## 分发密钥文件
export HOSTS="m01 w01 w02 192.168.1.10 192.168.1.11 192.168.5.10"
export SSHPASS=tony

$ for host in $HOSTS; do \
sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $host; \
done

## 测试免密登陆
$ ssh w01
Last login: Fri Feb  3 20:37:46 2023 from 192.168.1.10

安装PKI管理工具-cfssl#

检查下载新版本的工具，传送门

## 下载cfssl二进制程序
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssl_1.6.3_linux_amd64 -O /usr/local/bin/cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssljson_1.6.3_linux_amd64 -O /usr/local/bin/cfssljson

## 添加执行权限
chmod +x /usr/local/bin/cfssl*

安装配置containerd#

下载二进制文件#

检查软件的版本，传送门，下载带cri-containerd-cni开头的文件，这个tar包里面包含了containerd以及crictl管理工具和cni网络插件，下载后可以使用tar -tf <包名>，查看tar包的内容。

## 创建配置生成目录
mkdir -p /root/containerd/{app,bin,cnibin,config,service}
cd /root/containerd

## 下载二进制文件
wget https://github.com/containerd/containerd/releases/download/v1.6.16/cri-containerd-cni-1.6.16-linux-amd64.tar.gz -O app/containerd.tar.gz

## 解压
tar -xf app/containerd.tar.gz --strip-components=3 -C bin usr/local/bin/{containerd*,crictl,ctr} 
tar -xf app/containerd.tar.gz --strip-components=3 -C cnibin opt/cni/bin/

## 下载runc
## 查看版本https://github.com/opencontainers/runc/releases/
wget https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64 -O bin/runc

## 添加执行权限
chmod +x bin/runc

服务启动service文件#

生成脚本containerd_config.sh

$ cat <<'EOF'> containerd_config.sh 
## 创建service文件
cat > service/containerd.service <<EOF1
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
## Having non-zero Limit*s causes performance problems due to accounting overhead
## in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
## Comment TasksMax if your systemd version does not supports it.
## Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF1

## containerd需要加载2个内核模块
cat > config/containerd.conf <<EOF2
overlay
br_netfilter
EOF2
EOF

执行

bash -x containerd_config.sh

## 在service目录下生成
├── containerd.service

## 在config目录下生成
├── containerd.conf

生成配置文件，并按需修改#

## 创建配置文件
./bin/containerd config default > config/config.toml

## 在config目录下生成
├── config.toml

## bin_dir：containerd二进制文件
## conf_dir: cni 插件存储路径

$ vim config/config.toml
## root：容器存储路径，默认/var/lib/containerd，也可以修改成磁盘空间充足的路径
root = "/data/containerd"

## sandbox_image：pause镜像名称以及镜像tag，国内无法访问，可以更换成阿里云的
	#sandbox_image = "registry.k8s.io/pause:3.6"
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7"
    
## 配置SystemdCgroup = true，使用systemd启动的发行版，使用systemd管理cgroup
       [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
          # SystemdCgroup = false
          SystemdCgroup = true

## 配置镜像加速，参考https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration
    [plugins."io.containerd.grpc.v1.cri".registry]
      # config_path = ""
      config_path = "/etc/containerd/certs.d"

## 阿里云的加速地址可以更换成自己的地址 
$ cat <<EOF> config/hosts.toml
server = "https://docker.io"
[host."https://en2560w3.mirror.aliyuncs.com",host."https://registry.docker-cn.com"]
  capabilityes = ["pull", "resolve"]
EOF

## 在config目录下生成
├── hosts.toml

分发二进制文件、配置及创建相关路径#

$ for i in m01 w01 w02; do \
echo ">>> $i"; \
ssh $i "mkdir -p /etc/containerd/certs.d/docker.io"; \
ssh $i "mkdir -p /etc/cni/net.d"; \
ssh $i "mkdir -p /opt/cni/bin"; \
ssh $i "mkdir -p /opt/containerd"; \
scp bin/* $i:/usr/local/bin/; \
scp cnibin/* $i:/opt/cni/bin/; \
scp service/containerd.service $i:/usr/lib/systemd/system/; \
scp config/config.toml $i:/etc/containerd/; \
scp config/containerd.conf $i:/etc/modules-load.d/; \
scp config/hosts.toml $i:/etc/containerd/certs.d/docker.io/; \
done

启动containerd服务#

$ for i in m01 w01 w02; do \
echo ">>> $i"; \
ssh $i "systemctl restart systemd-modules-load.service"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable containerd"; \
ssh $i "systemctl restart containerd --no-block"; \
ssh $i "systemctl is-active containerd"; \
done

测试containerd#

Containerd有namespaces的概念，不同namespaces之间进行隔离，镜像和容器均不可见

## 查看名称空间
ctr ns list

## 创建ns
ctr ns create test          

## 测试拉取busybox镜像，使用ctr拉取，镜像的路径要写全，没有指明ns，默认保存在default命名空间
ctr images pull docker.io/library/busybox:latest

## 查看镜像，test命名空间没有镜像
ctr -n test images list

## 查看镜像
ctr images list

## 启动一个容器
ctr run -d docker.io/library/busybox:latest busybox

## 查看容器
$ ctr container ls
TASK     PID      STATUS
busybox    20808    RUNNING

$ ctr task ls

## TIP:
## 在containerd中，container和task是分离的，container描述的是容器分配和附加资源的元数据对象，是静态内容，task
## 是任务是系统上一个活动的、正在运行的进程。 task应该在每次运行后删除，而container可以被多次使用、更新和查询。这
## 点和docker中container定义是不一样的。

配置crictl#

使用containerd命令管理镜像较底层，对人类不友好，k8s内部提供了crictl来管理镜像，相当于docker命令行工具,crictl是遵循CRI接口规范的一个命令行工具，通常用它来检查和管理kubelet节点上的容器运行时和镜像,使用cricti工具之前，需要先创建crictl的配置文件。

生成脚本gen_crictl_config.sh

$ cat <<'EOF'> gen_crictl_config.sh
## 注意runtime-endpoint和image-endpoint必须与/etc/containerd/config.toml中配置保持一致。
cat > config/crictl.yaml <<EOF1
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF1
EOF

执行

bash -x gen_crictl_config.sh

## 在config目录下生成
├── crictl.yaml

分发crictl.yaml

$ for i in m01 w01 w02; do \
echo ">>> $i"; \
scp config/crictl.yaml $i:/etc/
done

测试crictl#

crictl的使用方法基本和docker用法相同

## 拉取镜像
crictl pull busybox:1.28
## 列出所有cri容器镜像
$ crictl images

部署etcd集群#

下载etcd#

## 创建保存配置的文件夹
mkdir -p /root/etcd/{bin,config,service,ssl,app}
cd /root/etcd

## 下载etcd二进制文件
## github二进制包下载地址：https://github.com/etcd-io/etcd/releases
wget https://github.com/etcd-io/etcd/releases/download/v3.5.7/etcd-v3.5.7-linux-amd64.tar.gz -O app/etcd.tar.gz

tar -xf app/etcd.tar.gz --strip-components=1 -C bin/ etcd-v3.5.7-linux-amd64/etcd{,ctl}

生成etcd使用的证书#

生成脚本gen_etcd_cert.sh

$ cat <<'EOF'> gen_etcd_cert.sh
## example: bash gen_etcd_cert.sh 127.0.0.1,m01,m02,m03,192.168.1.51,192.168.1.52,192.168.1.53

HOSTNAME=$1

## etcd ca的配置文件
cat > ca-config.json <<EOF1
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "peer": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF1

## etcd的ca证书签名请求文件
cat > etcd-ca-csr.json <<EOF2
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangZhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF2

## 生成etcd集群使用的ca根证书
cfssl gencert \
	-initca etcd-ca-csr.json | cfssljson -bare ssl/etcd-ca

## 生成etcd集群使用的证书申请签名文件
cat > etcd-csr.json <<EOF3
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangZhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF3

## 生产部署etcd集群可以使用3张证书用于不同认证。
## 1.etcd server持有的服务端证书
## 2.peer集群中节点互相通信使用的客户端证书
## 3.配置在kube-apiserver中用来与etcd-server做双向认证的客户端证书
## 学习环境使用一张peer类型的证书进行认证
cfssl gencert \
	-ca=ssl/etcd-ca.pem \
	-ca-key=ssl/etcd-ca-key.pem \
	-config=ca-config.json \
	-hostname=${HOSTNAME} \
	-profile=peer etcd-csr.json | cfssljson -bare ssl/etcd
EOF

执行

## example：bash gen_etcd_cert.sh <etcd相关主机ip及主机名>
bash -x gen_etcd_cert.sh 127.0.0.1,m01,192.168.1.10

## 在ssl目录下生成
├── etcd-ca.csr
├── etcd-ca-key.pem
├── etcd-ca.pem
├── etcd.csr
├── etcd-key.pem
├── etcd.pem

生成参数文件及启动service文件#

生成脚本etcd_config.sh

$ cat <<'EOF'> etcd_config.sh
## example: bash etcd_config.sh m01 192.168.1.51 m02=https://192.168.1.52:2380,m03=https://192.168.1.53:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd
ETCD_CONF_DIR=/opt/etcd/config
ETCD_CA_CERT=etcd-ca.pem
ETCD_SERVER_CERT_PREFIX=etcd

cat > config/etcd.config.yaml.$1 <<EOF1
name: '${ETCD_NAME}'
data-dir: ${WORK_DIR}/data
wal-dir: ${WORK_DIR}/data/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://${ETCD_IP}:2380'
listen-client-urls: 'https://${ETCD_IP}:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://${ETCD_IP}:2380'
advertise-client-urls: 'https://${ETCD_IP}:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: '${ETCD_NAME}=https://${ETCD_IP}:2380,${ETCD_CLUSTER}'
initial-cluster-token: 'etcd-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '${WORK_DIR}/ssl/${ETCD_SERVER_CERT_PREFIX}.pem'
  key-file: '${WORK_DIR}/ssl/${ETCD_SERVER_CERT_PREFIX}-key.pem'
  client-cert-auth: true
  trusted-ca-file: '${WORK_DIR}/ssl/${ETCD_CA_CERT}'
  auto-tls: true
peer-transport-security:
  cert-file: '${WORK_DIR}/ssl/${ETCD_SERVER_CERT_PREFIX}.pem'
  key-file: '${WORK_DIR}/ssl/${ETCD_SERVER_CERT_PREFIX}-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '${WORK_DIR}/ssl/${ETCD_CA_CERT}'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF1

cat > service/etcd.service <<EOF2

[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
--config-file=${ETCD_CONF_DIR}/etcd.config.yaml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF2
EOF

执行

## example：./etcd_config.sh <ETCD主机名> <ETCD_IP> <ETCD集群其他的信息>
bash -x etcd_config.sh m01 192.168.1.10

## 在config目录下生成
├── etcd.config.yaml.m01

## 在service目录下生成
├── etcd.service

分发etcd二进制文件、证书、配置及服务文件#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "mkdir -p /opt/etcd/{config,data,ssl}"; \
scp bin/etcd* $i:/usr/local/bin; \
scp ssl/etcd{,-key,-ca}.pem $i:/opt/etcd/ssl/; \
scp config/etcd.config.yaml.$i $i:/opt/etcd/config/etcd.config.yaml; \
scp service/etcd.service $i:/usr/lib/systemd/system/; \
done

启动etcd服务#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable etcd"; \
ssh $i "systemctl restart etcd --no-block"; \
ssh $i "systemctl is-active etcd"; \
done

验证集群#

## 查看集群
export ETCDCTL_API=3
export ENDPOINTS=192.168.1.10:2379
etcdctl \
    --endpoints="$ENDPOINTS" \
    --cacert=/opt/etcd/ssl/etcd-ca.pem \
    --cert=/opt/etcd/ssl/etcd.pem \
    --key=/opt/etcd/ssl/etcd-key.pem endpoint status \
    --write-out=table

etcdctl \
    --endpoints="$ENDPOINTS" \
    --cacert=/opt/etcd/ssl/etcd-ca.pem \
    --cert=/opt/etcd/ssl/etcd.pem \
    --key=/opt/etcd/ssl/etcd-key.pem member list \
    --write-out=table
    
etcdctl \
    --endpoints="$ENDPOINTS" \
    --cacert=/opt/etcd/ssl/etcd-ca.pem \
    --cert=/opt/etcd/ssl/etcd.pem \
    --key=/opt/etcd/ssl/etcd-key.pem endpoint health \
    --write-out=table

部署k8s组件#

在master节点-m01部署3个服务

• kube-apiserver
• kube-controller-manager
• kube-scheduler

下载kubernets二进制文件#

下载二进制文件，传送门

## 创建配置目录
mkdir -p /root/k8s/{app,ssl,config,service,bin,kubeconfig}
cd /root/k8s

## 下载kubernets二进制包，按版本直接修改v1.26.1 -> v1.xx.x
wget https://dl.k8s.io/v1.26.1/kubernetes-server-linux-amd64.tar.gz -O app/kubernetes-server.tar.gz

## 解压
tar -xf app/kubernetes-server.tar.gz  --strip-components=3 -C bin/ \
	kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

生成k8s使用的ca证书#

生成脚本gen_k8s_ca_cert.sh

$ cat <<'EOF'> gen_k8s_ca_cert.sh
cat > ca-config.json <<EOF1
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "peer": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF1

## 生成CA证书签名请求的配置文件
cat > ca-csr.json <<EOF2
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "GuangDong",
            "ST": "GuangZhou",
            "O": "Kubernetes",
            "OU": "System"
        }
    ],
  "ca": {
    "expiry": "876000h"
  }    
}
EOF2

## 生成ca证书和ca的私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ssl/ca
EOF

执行

bash -x gen_k8s_ca_cert.sh

## 在ssl目录下生成
├── ca-key.pem
├── ca.pem

部署apiserver#

生成apiserver所需证书#

生成gen_apiserver_cert.sh

$ cat <<'EOF'> gen_apiserver_cert.sh
## 生成apiserver的证书和私钥（apiserver和其它k8s组件通信使用）
APISERVER_NAME=$1

cat > kube-apiserver-csr.json <<EOF1
{
    "CN": "kube-apiserver",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "GuangDong",
            "ST": "GuangZhou",
            "O": "Kubernetes",
            "OU": "System"
        }
    ]
}
EOF1

cfssl gencert -ca=ssl/ca.pem -ca-key=ssl/ca-key.pem -config=ca-config.json \
	-hostname=${APISERVER_NAME} \
	-profile=peer kube-apiserver-csr.json | cfssljson -bare ssl/kube-apiserver

## apiserver聚合证书
## 访问kube-apiserver的另一种方式就是使用kube-proxy来代理访问, 而该证书就是用来支持SSL代理访问的. 在该种访问模
## 式下，我们是以http的方式发起请求到代理服务的, 此时, 代理服务会将该请求发送给kube-apiserver, 在此之前, 代理会
## 将发送给kube-apiserver的请求头里加入证书信息。

## 客户端 -- 发起请求 ---> 代理 -- Add Header信息:发起请求 --> kube-apiserver
## 如果apiserver所在的主机上没有运行kube-proxy，既无法通过服务的ClusterIP进行访问，需要
## --enable-aggregator-routing=true

## 生成ca签名请求文件
cat > front-proxy-ca-csr.json <<EOF2
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  },
  "ca": {
    "expiry": "876000h"
  }
}
EOF2

## 此根证书用在requestheader-client-ca-file配置选项中, kube-apiserver使用该证书来验证客户端证书是否为自己所
## 签发
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare ssl/front-proxy-ca

## 生成front-proxy-client证书请求文件
## 这里的CN名称要和apiserver启动参数--requestheader-allowed-names=front-proxy-client相同
cat > front-proxy-client-csr.json <<EOF3
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF3

## 生成代理层证书，代理端使用此证书，用来代用户向kube-apiserver认证
cfssl gencert -ca=ssl/front-proxy-ca.pem -ca-key=ssl/front-proxy-ca-key.pem -config=ca-config.json \
	-profile=peer front-proxy-client-csr.json | cfssljson -bare ssl/front-proxy-client

## 创建ServiceAccount Key —— secret
## serviceaccount账号的一种认证方式，创建serviceaccount的时候会创建一个与之绑定的secret，这个secret会生成
## token，这组密钥对仅提供给controller-manager使用，controller-manager通过sa.key对token进行签名, master
## 节点通过公钥sa.pub进行签名的验证
openssl genrsa -out ssl/sa.key 2048
openssl rsa -in ssl/sa.key -pubout -out ssl/sa.pub
EOF

执行

## 10.96.0.1是server_cluseter_IP网段的第一个ip地址
bash -x gen_apiserver_cert.sh  127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.96.0.1,192.168.1.10,m01

生成配置文件及启动service文件#

生成脚本apiserver_config.sh

## --service-cluster-ip-range，该网段不能和宿主机的网段、pod网段重复
$ cat <<'EOF'> apiserver_config.sh
## 创建 kube-apiserver 启动参数配置文件
MASTER_ADDRESS=$1
ETCD_SERVERS=$2
ETCD_CERT_DIR=/opt/etcd/ssl
K8S_CERT_DIR=/opt/k8s/ssl
K8S_CONF_DIR=/opt/k8s/config
API_CERT_PRIFIX=kube-apiserver

cat > service/kube-apiserver.service.${MASTER_ADDRESS} <<EOF1
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
--v=2 \\
--allow-privileged=true \\
--bind-address=${MASTER_ADDRESS} \\
--advertise-address=${MASTER_ADDRESS} \\
--secure-port=6443 \\
--service-cluster-ip-range=10.96.0.0/16 \
--service-node-port-range=30000-39999 \\
--etcd-servers=${ETCD_SERVERS} \\
--etcd-cafile=${ETCD_CERT_DIR}/etcd-ca.pem \\
--etcd-certfile=${ETCD_CERT_DIR}/etcd.pem \\
--etcd-keyfile=${ETCD_CERT_DIR}/etcd-key.pem \\
--client-ca-file=${K8S_CERT_DIR}/ca.pem \\
--tls-cert-file=${K8S_CERT_DIR}/${API_CERT_PRIFIX}.pem \\
--tls-private-key-file=${K8S_CERT_DIR}/${API_CERT_PRIFIX}-key.pem \\
--kubelet-client-certificate=${K8S_CERT_DIR}/${API_CERT_PRIFIX}.pem \\
--kubelet-client-key=${K8S_CERT_DIR}/${API_CERT_PRIFIX}-key.pem \\
--service-account-key-file=${K8S_CERT_DIR}/sa.pub \\
--service-account-signing-key-file=${K8S_CERT_DIR}/sa.key \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \\
--authorization-mode=Node,RBAC \\
--enable-bootstrap-token-auth=true \\
--enable-aggregator-routing=true \\
--proxy-client-cert-file=${K8S_CERT_DIR}/front-proxy-client.pem \\
--proxy-client-key-file=${K8S_CERT_DIR}/front-proxy-client-key.pem \\
--requestheader-client-ca-file=${K8S_CERT_DIR}/front-proxy-ca.pem \\
--requestheader-allowed-names=front-proxy-client \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-username-headers=X-Remote-User
#--token-auth-file=\${K8S_CONF_DIR}/token.csv 这里禁用token文件进行认证
    
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF1
EOF

执行

## bash apiserver_config.sh <master_IP> <etcd_cluster>
bash -x apiserver_config.sh 192.168.1.10 https://192.168.1.10:2379

## 在service目录下生成
├── kube-apiserver.service.192.168.1.10

分发二进制文件、证书及service文件#

$ for i in 192.168.1.10; do \
echo ">>> $i"; \
ssh $i "mkdir -p /opt/k8s/{ssl,config,log}"; \
scp bin/kube-apiserver  $i:/usr/local/bin/; \
scp ssl/{kube*.pem,ca{,-key}.pem,front-proxy-client*.pem,front-proxy-ca.pem,sa.*} $i:/opt/k8s/ssl/; \
scp service/kube-apiserver.service.$i $i:/usr/lib/systemd/system/kube-apiserver.service; \
done

启动kube-apiserver服务#

for i in m01; do \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable kube-apiserver"; \
ssh $i "systemctl restart kube-apiserver --no-block"; \
ssh $i "systemctl is-active kube-apiserver"; \
done

部署kubectl#

先部署kubectl客户端工具，部署后可以使用命令kubectl查看集群的信息

生成kubectl所需证书#

生成脚本gen_kubectl_cert.sh

$ cat <<EOF> gen_kubectl_cert.sh
## 生成kubectl的证书和私钥
## k8s安装时会创建一个集群角色(clusterrole)，名字为cluster-admin，对集群具有最高管理权限同时会创建一个集群角色
## 绑定(clusterrolebingding)，名字也叫做cluster-admin，这个绑定将集群角色(cluster-admin)和用户组
## (system:masters)关联起来，于是属于system:masters组内的用户，都会有这个集群角色赋予的权限；
## 生成证书时会定义用户clusteradmin，所属组为system:masters，所以clusteradmin拥有集群角色(cluster-admin)赋
## 予的权限。
cat > kubectl-csr.json <<EOF1
{
  "CN": "clusteradmin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangZhou",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF1

cfssl gencert -ca=ssl/ca.pem -ca-key=ssl/ca-key.pem -config=ca-config.json -profile=peer kubectl-csr.json | cfssljson -bare ssl/kubectl
EOF

执行

bash -x gen_kubectl_cert.sh

## 在ssl目录先生成
├── kubectl-key.pem
├── kubectl.pem

生成kubeconfig文件#

生成脚本kubeconfig_kubectl_config.sh

## 参数根据自己部署需要进行修改
## USERNAME必须和证书申请的CN名字相同，这里的用户名是clusteradmin
$ cat <<'EOF' > kubeconfig_kubectl_config.sh
APISERVER_IP=$1
K8S_CERT_DIR=$2
PORT=6443
KUBE_APISERVER=https://${APISERVER_IP}:${PORT}
CLUSTER_NAME=kubernetes
USERNAME=clusteradmin
KUBECONFIG_FILE=kubeconfig/kubectl.kubeconfig
CONTEXT_NAME=${USERNAME}@${CLUSTER_NAME}
CERT_PRFIX=kubectl

## 设置集群参数
./bin/kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=${K8S_CERT_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBECONFIG_FILE}

## 设置客户端认证参数
./bin/kubectl config set-credentials ${USERNAME} \
	--client-certificate=${K8S_CERT_DIR}/${CERT_PRFIX}.pem \
	--client-key=${K8S_CERT_DIR}/${CERT_PRFIX}-key.pem \
	--embed-certs=true \
	--kubeconfig=${KUBECONFIG_FILE}

## 设置context---将用户和集群关联起来
./bin/kubectl config set-context ${CONTEXT_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${USERNAME} \
    --kubeconfig=${KUBECONFIG_FILE}

## 设置默认contexts
./bin/kubectl config use-context ${CONTEXT_NAME} \
    --kubeconfig=${KUBECONFIG_FILE}
EOF

执行

## example kubeconfig_kubectl_config.sh <API_SERVER> <CERTPATH>
bash -x kubeconfig_kubectl_config.sh 192.168.1.10 ssl

## 在kubeconfig目录下生成
├── kubectl.kubeconfig

分发kubeconfig文件#

## 分发kubeconfig证书
$ for i in m01; do \
echo ">>> $i"; \
ssh $i "mkdir -p $HOME/.kube/"; \
scp bin/kubectl $i:/usr/local/bin/; \
scp kubeconfig/kubectl.kubeconfig $i:$HOME/.kube/config; \
done

kubectl命令补全功能#

## bash配置
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

查看集群状态#

$ kubectl cluster-info
Kubernetes control plane is running at https://192.168.1.10:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl get componentstatus
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get "https://127.0.0.1:10259/healthz": dial tcp 127.0.0.1:10259: connect: connection refused
controller-manager   Unhealthy   Get "https://127.0.0.1:10257/healthz": dial tcp 127.0.0.1:10257: connect: connection refused
etcd-0               Healthy     {"health":"true","reason":""}

$ kubectl get all -A
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   17m

部署controller-manager组件#

生成controller-manager所需证书#

创建生成证书脚本gen_controller_cert.sh

$ cat <<'EOF'> gen_controller_cert.sh
CONTROLLER_IP=$1

## 生成controller-manager证书签名请求
cat > kube-controller-manager-csr.json <<EOF1
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangZhou",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF1

cfssl gencert -ca=ssl/ca.pem -ca-key=ssl/ca-key.pem \
-config=ca-config.json \
-hostname=${CONTROLLER_IP} \
-profile=peer kube-controller-manager-csr.json | cfssljson -bare ssl/kube-controller-manager
EOF

执行

bash -x gen_controller_cert.sh 127.0.0.1,192.168.1.10

## 在ssl目录下生成
├── kube-controller-manager-key.pem
├── kube-controller-manager.pem

生成kubeconfig文件#

$ cat <<'EOF'> kubeconfig_kube-controller-manager.sh
APISERVER_IP=$1
K8S_CERT_DIR=$2
PORT=6443
KUBE_APISERVER=https://${APISERVER_IP}:${PORT}
KUBECONFIG_FILE=kubeconfig/kube-controller-manager.kubeconfig
CLUSTER_NAME=kubernetes
USERNAME=system:kube-controller-manager
CONTEXT_NAME=${USERNAME}@${CLUSTER_NAME}
CERT_PRFIX=kube-controller-manager

## 设置集群参数
./bin/kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=${K8S_CERT_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBECONFIG_FILE}

## 设置用户认证参数
./bin/kubectl config set-credentials ${USERNAME} \
	--client-certificate=${K8S_CERT_DIR}/${CERT_PRFIX}.pem \
	--client-key=${K8S_CERT_DIR}/${CERT_PRFIX}-key.pem \
	--embed-certs=true \
	--kubeconfig=${KUBECONFIG_FILE}

## 设置context---将用户和集群关联起来
./bin/kubectl config set-context ${CONTEXT_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${USERNAME} \
    --kubeconfig=${KUBECONFIG_FILE}

## 设置默认context
./bin/kubectl config use-context ${CONTEXT_NAME} \
    --kubeconfig=${KUBECONFIG_FILE}
EOF

执行

## example：./kube-controller-manager_config.sh <MASTER_IPADDR> <证书目录>
bash -x kubeconfig_kube-controller-manager.sh 192.168.1.10 ssl

## 在kubeconfig目录下生成
├── kube-controller-manager.kubeconfig

生成kube-controller-manager的service文件#

生成脚本controller_manager_config.sh

## --cluster-cidr为pod网段，不能和宿主机网段，service网段重复
$ cat <<'EOF'> kube-controller-manager.sh
K8S_CERT_DIR=/opt/k8s/ssl
K8S_CONF_DIR=/opt/k8s/config
PODCIDR=172.16.0.0/16

cat > service/kube-controller-manager.service <<EOF1
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
--v=2 \\
--bind-address=127.0.0.1 \\
--root-ca-file=${K8S_CERT_DIR}/ca.pem \\
--cluster-signing-cert-file=${K8S_CERT_DIR}/ca.pem \\
--cluster-signing-key-file=${K8S_CERT_DIR}/ca-key.pem \\
--service-account-private-key-file=${K8S_CERT_DIR}/sa.key \\
--tls-cert-file=${K8S_CERT_DIR}/kube-controller-manager.pem \\
--tls-private-key-file=${K8S_CERT_DIR}/kube-controller-manager-key.pem \\
--kubeconfig=${K8S_CONF_DIR}/kube-controller-manager.kubeconfig \\
--leader-elect=true \\
--use-service-account-credentials=true \\
--node-monitor-grace-period=40s \\
--node-monitor-period=5s \\
--pod-eviction-timeout=2m0s \\
--controllers=*,bootstrapsigner,tokencleaner \\
--allocate-node-cidrs=true \\
--cluster-cidr=${PODCIDR} \\
--requestheader-client-ca-file=${K8S_CERT_DIR}/front-proxy-ca.pem \\
--node-cidr-mask-size=24
      
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF1
EOF

执行

bash -x kube-controller-manager.sh

# 在service目录下生成以下文件
├── kube-controller-manager.service

分发二进制文件、证书、kubeconfig文件及service文件#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "mkdir -p /opt/k8s/{ssl,config}"; \
scp bin/kube-controller-manager  $i:/usr/local/bin/;
scp ssl/kube-controller*.pem $i:/opt/k8s/ssl/; \
scp service/kube-controller-manager.service $i:/usr/lib/systemd/system/; \
scp kubeconfig/kube-controller-manager.kubeconfig $i:/opt/k8s/config/; \
done

启动kube-controller-manager服务#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable kube-controller-manager"; \
ssh $i "systemctl restart kube-controller-manager --no-block"; \
ssh $i "systemctl is-active kube-controller-manager"; \
done

验证#

$ ss -tlp | grep kube-controller
LISTEN 0      16384       127.0.0.1:10257             0.0.0.0:*    users:(("kube-controller",pid=2864,fd=7))

$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get "https://127.0.0.1:10259/healthz": dial tcp 127.0.0.1:10259: connect: connection refused
controller-manager   Healthy     ok
etcd-0               Healthy     {"health":"true","reason":""}

部署kube-scheduler组件#

生成kube-scheduler所需证书#

生成脚本gen_schduler_cert.sh

$ cat <<'EOF'> gen_schduler_cert.sh
## 生成 kube-scheduler 的证书和私钥
SCHEDULER_IP=$1
CSR_NAME_PREFIX=kube-scheduler

cat > ${CSR_NAME_PREFIX}-csr.json <<EOF1
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangZhou",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF1

cfssl gencert -ca=ssl/ca.pem -ca-key=ssl/ca-key.pem \
-config=ca-config.json \
-hostname=${SCHEDULER_IP} \
-profile=peer ${CSR_NAME_PREFIX}-csr.json | cfssljson -bare ssl/${CSR_NAME_PREFIX}
EOF

执行

bash -x gen_schduler_cert.sh 127.0.0.1,192.168.1.10

## 在ssl目录下生成
├── kube-scheduler-key.pem
├── kube-scheduler.pem

生成kubeconfig文件#

生成脚本kubeconfig_kube-scheduler.sh

$ cat <<'EOF' > kubeconfig_kube-scheduler.sh 
APISERVER_IP=$1
K8S_CERT_DIR=$2
PORT=6443
KUBE_APISERVER=https://${APISERVER_IP}:${PORT}
KUBECONFIG_FILE=kubeconfig/kube-scheduler.kubeconfig
CLUSTER_NAME=kubernetes
USERNAME=system:kube-scheduler
CONTEXT_NAME=${USERNAME}@${CLUSTER_NAME}
CERT_PRFIX=kube-scheduler

## 设置集群参数
./bin/kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=${K8S_CERT_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBECONFIG_FILE}

## 设置用户认证参数
./bin/kubectl config set-credentials ${USERNAME} \
	--client-certificate=${K8S_CERT_DIR}/${CERT_PRFIX}.pem \
	--client-key=${K8S_CERT_DIR}/${CERT_PRFIX}-key.pem \
	--embed-certs=true \
	--kubeconfig=${KUBECONFIG_FILE}

## 设置context---将用户和集群关联起来
./bin/kubectl config set-context ${CONTEXT_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${USERNAME} \
    --kubeconfig=${KUBECONFIG_FILE}

## 设置默认context
./bin/kubectl config use-context ${CONTEXT_NAME} \
    --kubeconfig=${KUBECONFIG_FILE}
EOF

添加可执行权限并运行

bash -x kubeconfig_kube-scheduler.sh 192.168.1.10 ssl

# 在kubeconfig目录下生成
├── kube-scheduler.kubeconfig

生成kube-scheduler的service文件#

生成脚本kube-scheduler.sh

$ cat <<'EOF'> kube-scheduler.sh
K8S_CONF_DIR=/opt/k8s/config

cat > service/kube-scheduler.service <<EOF1
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
--v=2 \\
--bind-address=127.0.0.1 \\
--leader-elect=true \\
--kubeconfig=${K8S_CONF_DIR}/kube-scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF1
EOF

执行

bash -x kube-scheduler.sh

## 在service目录下生成
├── kube-scheduler.service

分发二进制文件、证书、kubeconfig文件及service文件#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "mkdir -p /opt/k8s/{ssl,config}"; \
scp bin/kube-scheduler $i:/usr/local/bin/ ;
scp ssl/kube-scheduler*.pem $i:/opt/k8s/ssl/; \
scp service/kube-scheduler.service $i:/usr/lib/systemd/system/; \
scp kubeconfig/kube-scheduler.kubeconfig $i:/opt/k8s/config/; \
done

启动kube-scheduler服务#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable kube-scheduler"; \
ssh $i "systemctl restart kube-scheduler --no-block"; \
ssh $i "systemctl is-active kube-scheduler"; \
done

验证集群#

$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true","reason":""}

至此master节点三大组件部署完毕

部署kubelet#

配置TLS Bootstrap#

为什么这个证书不是手动管理？因为k8s的master节点可能是固定的，创建好之后一直就是那几台，但worker节点可能变化比较多，如果添加，删除，故障维护时手动添加会比较麻烦，证书和主机名是有绑定的，而我们的主机名又是不一样的，所以需要有一种机制自动颁发证书请求。

每个节点的kubelet组件都要使用由apiserver使用的CA签发的有效证书才能与apiserver通讯；此时如果节点多起来，为每个节点单独签署证书将是一件非常繁琐的事情；TLS bootstrapping功能就是让kubelet先使用一个预定的低权限用户连接到apiserver，然后向apiserver申请证书，kubelet的证书由apiserver动态签署。

生成kubeconfig文件#

bootstrap.kubeconfig文件是一个用来向apiserver申请证书的文件

生成脚本kubeconfig_bootstrap_config.sh

$ cat <<'EOF'> kubeconfig_bootstrap_config.sh
APISERVER_IP=$1
K8S_CERT_DIR=$2
K8S_CONF_DIR=/opt/k8s/config
PORT=6443
KUBE_APISERVER=https://${APISERVER_IP}:${PORT}
KUBECONFIG_FILE=kubeconfig/bootstrap.kubeconfig
CLUSTER_NAME=kubernetes

## 生成bootstrap的token
TOKEN_ID=$(openssl rand -hex 3)
TOKEN_SECRET=$(openssl rand -hex 8)
BOOTSTRAP_TOKEN=${TOKEN_ID}.${TOKEN_SECRET}

USERNAME=system:bootstrap:${TOKEN_ID}
CONTEXT_NAME=${USERNAME}@${CLUSTER_NAME}

## 创建bootstrap.kubeconfig
## 设置集群参数
./bin/kubectl config set-cluster ${CLUSTER_NAME} \
--certificate-authority=${K8S_CERT_DIR}/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBECONFIG_FILE}

## 设置客户端认证参数，kubelet 使用bootstrap token认证
./bin/kubectl config set-credentials ${USERNAME} \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=${KUBECONFIG_FILE}

## 设置上下文参数
./bin/kubectl config set-context ${CONTEXT_NAME}  \
--cluster=kubernetes \
--user=${USERNAME} \
--kubeconfig=${KUBECONFIG_FILE}

## 使用上下文参数生成 bootstrap.kubeconfig 文件
./bin/kubectl config use-context ${CONTEXT_NAME} --kubeconfig=${KUBECONFIG_FILE}

## 创建boostrap token secret
cat > config/bootstrap-token-secret.yaml <<EOF1
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-${TOKEN_ID}
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  token-id: ${TOKEN_ID}
  token-secret: ${TOKEN_SECRET}
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
EOF1
EOF

执行

bash -x kubeconfig_bootstrap_config.sh 192.168.1.10 ssl

## 在config目录下生成
├── bootstrap-token-secret.yaml

## 在kubeconfig目录下生成
├── bootstrap.kubeconfig

导入bootstrap-token-secret#

## 创建secret
$ kubectl apply -f config/bootstrap-token-secret.yaml
secret/bootstrap-token-a956ff created

查看bootstrap-token#

$ kubectl get secret -nkube-system
NAME                     TYPE                            DATA   AGE
bootstrap-token-a956ff   bootstrap.kubernetes.io/token   5      12s

生成kubelet配置文件#

生成脚本kubelet.sh

$ cat <<'EOF'> kubelet_config.sh
K8S_CONF_DIR=/opt/k8s/config
K8S_CERT_DIR=/opt/k8s/ssl
CLUSTER_DNS=10.96.0.10

## 生成kubelet参数文件
cat > config/kubelet.conf <<EOF1
KUBELET_OPTS="--v=4 \\
--container-runtime-endpoint=unix:///run/containerd/containerd.sock \\
--runtime-cgroups=/systemd/system.slice \\
--kubeconfig=${K8S_CONF_DIR}/kubelet.kubeconfig \\
--bootstrap-kubeconfig=${K8S_CONF_DIR}/bootstrap.kubeconfig \\
--config=${K8S_CONF_DIR}/kubelet.yaml \\
--cert-dir=${K8S_CERT_DIR} \\
--node-labels=node.kubernetes.io/node="
EOF1

## 生成kubelet配置yaml文件
cat > config/kubelet.yaml <<EOF2
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: ${K8S_CERT_DIR}/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
runtimeRequestTimeout: 15m
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- ${CLUSTER_DNS}
clusterDomain: cluster.local
EOF2

## 生成kubelet.service服务启动文件
cat > service/kubelet.service <<EOF3
[Unit]
Description=Kubernetes Kubelet
After=containerd.service
Requires=containerd.service

[Service]
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/hugetlb/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/blkio/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/cpuset/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/devices/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/net_cls,net_prio/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/perf_event/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/cpu,cpuacct/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/freezer/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/memory/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/pids/systemd/system.slice
ExecStartPre=-/bin/mkdir -p /sys/fs/cgroup/systemd/systemd/system.slice
LimitNOFILE=655350
LimitNPROC=655350
LimitCORE=infinity
LimitMEMLOCK=infinity
## 在centos系统上需要配置CPUAccounting和MemoryAccounting
CPUAccounting=true
MemoryAccounting=true
EnvironmentFile=${K8S_CONF_DIR}/kubelet.conf
ExecStart=/usr/local/bin/kubelet \$KUBELET_OPTS

Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target
EOF3
EOF

执行

bash -x kubelet_config.sh

## 在service目录下
├── kubelet.service

## 在config目录下
├── kubelet.conf 
├── kubelet.yaml

分发二进制文件、配置文件、证书、kuconfig文件及service文件#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "mkdir -p /opt/k8s/{config,ssl,manifests}"; \
scp bin/kubelet $i:/usr/local/bin/; \
scp config/kubelet.{conf,yaml} $i:/opt/k8s/config/; \
scp kubeconfig/bootstrap.kubeconfig $i:/opt/k8s/config/; \
scp ssl/ca.pem $i:/opt/k8s/ssl; \
scp service/kubelet.service $i:/usr/lib/systemd/system/; \
done

启动kubelet服务#

$ for i in m01; do \
echo ">>> $i"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable kubelet"; \
ssh $i "systemctl restart kubelet --no-block"; \
ssh $i "systemctl is-active kubelet"; \
done

授权#

使用systemctl status kubelet会发现kubelet启动失败，进一步使用journalctl -xe -u kubelet.service --no-pager | less会发现如下错误，提示User "system:bootstrap:<tokenid>"不能创建资源certificatesigningrequests

m01 kubelet[1452]: Error: failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:a956ff" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope

在默认情况下，使用Bootstrap Token进行引导时，Kubelet组件使用Token发起的请求其用户名为system:bootstrap:<tokenid>，所属组为system:bootstrappers，然后创建CSR请求，但是此用户没有任何权限；在k8s中已经创建了一个clusterrole（system:node-bootstrapper），此集群角色具有发起CSR请求的权限，我们需要创建一个clusterrolebinding将clusterrole和此token的用户名或者所属组进行关联，然后system:bootstrap:<tokenid>拥有了system:node-bootstrapper的权限，这样任何用户拿着这个token连接apiserver都具有system:node-bootstrapper的权限

这里创建一个clusterrolebinding（名称:create-csrs-for-bootstrapping）将clusterrole和group进行绑定

kubectl create clusterrolebinding create-csrs-for-bootstrapping \
  --clusterrole=system:node-bootstrapper \
  --group=system:bootstrappers:default-node-token 

$ kubectl get clusterrolebinding create-csrs-for-bootstrapping
NAME                            ROLE                                   AGE
create-csrs-for-bootstrapping   ClusterRole/system:node-bootstrapper   22s

手动签发kubelet证书并查看集群#

任何人使用token进行认证通过后进入授权阶段，api-server从该token中获取namespace和name信息,并将该token特殊对待，授予anyone bootstrap权利，将该匿名用户划分到system:bootstraps组，至此anyone使用该token认证的时候都具有了system:node-bootstrapper的权利

## 重新启动kubelet服务
systemctl restart kubelet.service
for i in m01; do \
echo ">>> $i"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable kubelet"; \
ssh $i "systemctl restart kubelet --no-block"; \
ssh $i "systemctl is-active kubelet"; \
done

## 查看节点kubelet启动证书请求状态，这时已经是Pending状态
$ kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
node-csr-swEhM80wF2MM02rWKEdzKL0KYaqb2y8QK0WXKja3RK0   9s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:a956ff   <none>              Pending

## 手动签发kubelet的证书
$ kubectl certificate approve node-csr-swEhM80wF2MM02rWKEdzKL0KYaqb2y8QK0WXKja3RK0
certificatesigningrequest.certificates.k8s.io/node-csr-swEhM80wF2MM02rWKEdzKL0KYaqb2y8QK0WXKja3RK0 approved

## 再次查看证书请求状态，已经变成了Approved,Issued
$ kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
node-csr-swEhM80wF2MM02rWKEdzKL0KYaqb2y8QK0WXKja3RK0   70s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:a956ff   <none>              Approved,Issued

## 查看node，由于网络插件还未安装，状态显示为NotReady
$ kubectl get nodes
NAME   STATUS     ROLES    AGE   VERSION
m01    NotReady   <none>   48s   v1.26.1

自动批准，自动续期，自动颁发#

要是有很多worker节点要安装kubelet，手工去approve证书请求会很繁琐，增加工作量，就有了自动批准，自动续期，自动颁发的方法。

kubelet所发起的CSR请求是由controller manager签署的；如果想要是实现自动续期，就需要让controller manager能够在 kubelet发起证书请求的时候自动帮助其签署证书；那么controller manager不可能对所有的CSR证书申请都自动签署，这时候就需要配置RBAC规则，保证controller manager只对kubelet发起的特定CSR请求自动批准即可；在TLS bootstrapping官方文档中，CSR有三种请求类型:

• nodeclient: kubelet以O=system:nodes和CN=system:node:(node name)形式发起的CSR请求
• selfnodeclient: kubelet client renew自己的证书发起的CSR请求(与上一个证书就有相同的O和CN)
• selfnodeserver: kubelet server renew 自己的证书发起的CSR请求

通俗点讲就是:
nodeclient类型的CSR仅在第一次启动时会产生，

selfnodeclient类型的CSR请求实际上就是kubelet renew自己作为client跟 apiserver通讯时使用的证书产生的，selfnodeserver类型的CSR请求则是kubelet首次申请或后续renew自己的10250 api端口证书时产生的，以下为3中CSR请求分别创建3种对应的Clusterrole

创建3个clusterrolebinding#

• 自动批准kubelet首次用于与 apiserver 通讯证书的 CSR 请求(nodeclient)
• 自动批准kubelet首次用于10250端口鉴权的CSR请求(实际上这个请求走的也是selfnodeserver 类型 CSR)
• 自动批准kubelet后续renew用于与apiserver通讯证书的 CSR 请求(selfnodeclient)
• 自动批准kubelet后续renew用于10250端口鉴权的 CSR 请求(selfnodeserver)

## 自动批准kubelet的首次CSR请求(用于与apiserver通讯的证书)
kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --group=system:bootstrappers

## 自动批准kubelet后续renew用于与apiserver通讯证书的CSR请求
kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes

## 自动批准kubelet发起的用于10250端口鉴权证书的CSR请求(包括后续 renew)
kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes

分发二进制文件、配置文件、证书、kuconfig文件及service文件#

## 分发到其他主机
$ for i in w01 w02; do \
echo ">>> $i"; \
ssh $i "mkdir -p /opt/k8s/{config,ssl,manifests}"; \
scp bin/kubelet $i:/usr/local/bin/; \
scp config/kubelet.{conf,yaml} $i:/opt/k8s/config/; \
scp kubeconfig/bootstrap.kubeconfig $i:/opt/k8s/config/; \
scp ssl/ca.pem $i:/opt/k8s/ssl; \
scp service/kubelet.service $i:/usr/lib/systemd/system/; \
done

启动kubelet服务#

$ for i in w01 w02; do \
echo ">>> $i"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable kubelet"; \
ssh $i "systemctl restart kubelet --no-block"; \
ssh $i "systemctl is-active kubelet"; \
done

查看csr和集群信息#

## 再次查看csr，其他主机已经自动申请证书并自动批准，颁发
$ kubectl get csr
NAME                                                   AGE     SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
node-csr-MWQKyY1rzUIX4dhLZZNeBza2PVAlBg_NAkNoY7LSqCs   1s     kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:a956ff   <none>              Approved,Issued
node-csr-pHNMpmBFv2q34Z8S82yifT2BrKjFA1dVFs3Wk2-wiKI   3s     kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:a956ff   <none>              Approved,Issued
node-csr-swEhM80wF2MM02rWKEdzKL0KYaqb2y8QK0WXKja3RK0   2m8s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:a956ff   <none>              Approved,Issued

$ kubectl get nodes
NAME   STATUS     ROLES    AGE     VERSION
m01    NotReady   <none>   6m38s   v1.26.1
w01    NotReady   <none>   112s    v1.26.1
w02    NotReady   <none>   110s    v1.26.1

## 修改node的role标签
kubectl label nodes m01 node-role.kubernetes.io/master=
kubectl label nodes w01 node-role.kubernetes.io/worker=
kubectl label nodes w02 node-role.kubernetes.io/worker=

$ kubectl get nodes
NAME   STATUS     ROLES    AGE     VERSION
m01    NotReady   master   7m16s   v1.26.1
w01    NotReady   worker   2m30s   v1.26.1
w02    NotReady   worker   2m28s   v1.26.1

部署kube-proxy#

kube-proxy运行在所有worker节点上，它监听apiserver中service和endpoint的变化情况，创建路由规则提供服务IP和负载均衡功能。

生成证书#

kube-proxy提取证书中的CN作为客户端的用户名，即system:kube-proxy。 kube-apiserver预定义的 RBAC使用的ClusterRoleBindings system:node-proxier将用户system:kube-proxy与ClusterRole system:node-proxier绑定，该Role授予节点调用kube-apiserver proxy相关api的权限；

$ kubectl get clusterrole | grep node-proxier
system:node-proxier                                                    2023-02-10T05:53:48Z

$ kubectl describe clusterrolebinding/system:node-proxier
Name:         system:node-proxier
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
Role:
  Kind:  ClusterRole
  Name:  system:node-proxier
Subjects:
  Kind  Name               Namespace
----  ----               ---------
  User  system:kube-proxy  

生成脚本gen_kube_proxy_cert.sh

$ cat <<EOF> gen_kube_proxy_cert.sh
## 生成 kube-proxy 的证书和私钥，
cat > kube-proxy-csr.json <<EOF1
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangZhou",
      "O": "system:kube-proxy",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF1

cfssl gencert -ca=ssl/ca.pem -ca-key=ssl/ca-key.pem -config=ca-config.json -profile=peer kube-proxy-csr.json | cfssljson -bare ssl/kube-proxy
EOF

执行

bash -x gen_kube_proxy_cert.sh

## 在ssl目录下生成
├── kube-proxy-key.pem
├── kube-proxy.pem

生成kubeconfig文件#

认证方式有2种

• 证书认证
• token认证

本次部署采用的是证书认证

生成脚本kube-proxy_kubeconfig.sh

$ cat <<'EOF'> kube-proxy_kubeconfig.sh
K8S_CONF_DIR=/opt/k8s/config

APISERVER_IP=$1
K8S_CERT_DIR=$2
PORT=6443
CLUSTER_NAME=kubernetes
KUBE_APISERVER=https://${APISERVER_IP}:${PORT}
KUBECONFIG_FILE=kubeconfig/kube-proxy.kubeconfig
USERNAME=system:kube-proxy
CONTEXT_NAME=${USERNAME}@${CLUSTER_NAME}
CERT_PRFIX=kube-proxy

./bin/kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=${K8S_CERT_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBECONFIG_FILE}

./bin/kubectl config set-credentials ${USERNAME} \
    --client-certificate=${K8S_CERT_DIR}/${CERT_PRFIX}.pem \
    --client-key=${K8S_CERT_DIR}/${CERT_PRFIX}-key.pem \
    --embed-certs=true \
    --kubeconfig=${KUBECONFIG_FILE}

./bin/kubectl config set-context ${CONTEXT_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${USERNAME} \
    --kubeconfig=${KUBECONFIG_FILE}

./bin/kubectl config use-context ${CONTEXT_NAME} \
    --kubeconfig=${KUBECONFIG_FILE}
EOF

执行

bash -x kube-proxy_kubeconfig.sh 192.168.1.10 ssl

## 在kueconfig目录下生成
├── kube-proxy.kubeconfig

生成配置文件和service文件#

生成脚本kube-proxy_config.sh

## clusterCIDR: 172.16.0.0/16	这个是pod网段
$ cat <<'EOF'> kube-proxy_config.sh
K8S_CONF_DIR=/opt/k8s/config
CLUSER_CIDR=172.16.0.0/16

## 创建 kube-proxy 启动参数配置文件
cat > config/kube-proxy.yaml <<EOF1
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: ${K8S_CONF_DIR}/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: ${CLUSER_CIDR} 
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF1

##-----------------------------------
## 创建 kube-proxy.service 服务管理文件
cat > service/kube-proxy.service <<EOF2
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
    --config=${K8S_CONF_DIR}/kube-proxy.yaml \\
    --v=2
    
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF2
EOF

执行

bash -x kube-proxy_config.sh

## 在config目录下生成
├── kube-proxy.yaml

## 在service目录下生成
├── kube-proxy.service

分发二进制文件、配置文件、kubeconfig文件及service文件#

$　for i in m01 w01 w02; do \
echo ">>> $i"; \
scp bin/kube-proxy $i:/usr/local/bin/; \
scp config/kube-proxy.yaml $i:/opt/k8s/config/; \
scp kubeconfig/kube-proxy.kubeconfig $i:/opt/k8s/config/; \
scp service/kube-proxy.service $i:/usr/lib/systemd/system/; \
scp ssl/front-proxy-ca.pem $i:/opt/k8s/ssl/; \
done

启动kube-proxy服务#

$ for i in m01 w01 w02; do \
echo ">>> $i"; \
ssh $i "systemctl daemon-reload"; \
ssh $i "systemctl enable kube-proxy"; \
ssh $i "systemctl restart kube-proxy --no-block"; \
ssh $i "systemctl is-active kube-proxy"; \
done

查看服务状态#

$ ss -tnlp | grep kube-proxy
LISTEN 0      16384      127.0.0.1:10249      0.0.0.0:*    users:(("kube-proxy",pid=2272,fd=14))
LISTEN 0      16384              *:10256            *:*    users:(("kube-proxy",pid=2272,fd=12))

$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr
  -> 192.168.2.10:6443            Masq    1      0          0

部署Flannel#

下载最新版，传送门

## 创建配置目录
mkdir -p /root/addons/flannel
cd /root/addons/flannel

## 下载yaml文件
wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml \
    -O flannel.yaml
    
vim flannel.yaml
## cni-conf.json：flannel提供给kubelet使用的，kubelet根据这个配置调用flannel cni。
## net-conf.json：给DeamonSet的主容器的flanneld使用的，配置要和k8s的podcidr一致

## 修改pod cidr，Backend使用默认的vxlan
  net-conf.json: |
    {
      # "Network": "10.244.0.0/16",
      "Network": "172.16.0.0/16",
      "Backend": {
        "Type": "vxlan",
        # "VNI": 4000,  ## 可选，默认是1
        # "Port": 4789, ## 可选，默认8472
        "DirectRouting": true  ## 建议使用，在同一个二层网络直接走路由，跨网段才走vxlan隧道
      }

## 修改pod cidr，Backend使用IPIP的配置
  net-conf.json: |
    {
      # "Network": "10.244.0.0/16",
      "Network": "172.16.0.0/16",
      "Backend": {
        "Type": "ipip",
        "DirectRouting": true  ## 建议使用
      }
      
kubectl apply -f flannel.yaml

$ kubectl get nodes
NAME   STATUS   ROLES    AGE     VERSION
m01    Ready    master   3d14h   v1.26.1
w01    Ready    worker   3d14h   v1.26.1
w02    Ready    worker   3d14h   v1.26.1

$ kubectl get pods -A
NAMESPACE      NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-mxj56   1/1     Running   0          2m4s
kube-flannel   kube-flannel-ds-tnrkc   1/1     Running   0          2m4s
kube-flannel   kube-flannel-ds-xsq28   1/1     Running   0          2m4s

$  ip -d add show flannel.1
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether f6:69:ba:e9:d9:6b brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
    vxlan id 1 local 192.168.1.10 dev ens32 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet 172.16.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::f469:baff:fee9:d96b/64 scope link
       valid_lft forever preferred_lft forever

## 运行正常后，fanneld会在node主机的/etc/cni/net.d下生成配置文件，之后kubelet将会调用它       
$ cat /etc/cni/net.d/10-flannel.conflist
{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

部署addons#

部署coredns#

用于集群内部service的解析，可以让pod把service name解析成service ip，然后通过service的ip地址进行连接到对应的应用

，打开传送门，查看coredns的最新版本，目前最新是1.10.0版本。

部署coredns后，每个pod启动后，会在resolv.conf之注入dns信息

$ cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

生成yaml资源文件#

创建脚本文件gen_coredns_config.sh

cat <<'EOF'> gen_coredns_config.sh

DNS_DOMAIN=cluster.local
IMAGE_REGISTRY=registry.cn-hangzhou.aliyuncs.com/ik8ss/coredns:v1.10.0
DNS_MEMORY_LIMIT=170Mi
DNS_SERVER_IP=10.96.0.10

cat > coredns.yaml <<EOF1
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
  labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: Reconcile
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: EnsureExists
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
  labels:
      addonmanager.kubernetes.io/mode: EnsureExists
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes ${DNS_DOMAIN} in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  # replicas: not specified here:
  # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2. Default is 1.
  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: k8s-app
                    operator: In
                    values: ["kube-dns"]
              topologyKey: kubernetes.io/hostname
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - name: coredns
        image: ${IMAGE_REGISTRY}
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: ${DNS_MEMORY_LIMIT}
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: ${DNS_SERVER_IP}
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP
EOF1
EOF

执行脚本

bash -x gen_coredns_config.sh

# 当前目录下生成以下文件
├── coredns.yaml

应用yaml资源文件并查看#

kubectl apply -f coredns.yaml

$ kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS       AGE
kube-system   coredns-7f8b8f7b8-rhnnj                    1/1     Running   0          20s

部署metrics-server#

在k8s中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和pod的内存，磁盘，CPU和网络使用率。

可以传送门下载yaml文件，但是要修改几个参数

IMAGE_REGISTRY：在阿里云的容器服务里面查看对应metrics-server镜像的版本

生成yaml资源文件#

创建脚本文件gen_metrics-server_config.sh

$ cat <<'EOF'> gen_metrics-server_config.sh
CERT_PATH=/opt/k8s/ssl
CLIENT_CA_FILE=${CERT_PATH}/front-proxy-ca.pem
IMAGE_REGISTRY=registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.6.2

cat > metrics-server.yaml <<EOF1
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls   ## kubectl top nodes
        - --requestheader-client-ca-file=${CLIENT_CA_FILE}   ## 聚合CA证书 front-proxy-ca.crt
        - --requestheader-username-headers=X-Remote-User
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-extra-headers-prefix=X-Remote-Extra-
        image: ${IMAGE_REGISTRY}
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
        - name: ca-ssl
          mountPath: ${CERT_PATH}  
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      hostNetwork: true	   # kubectl top pods
      volumes:
      - emptyDir: {}
        name: tmp-dir
      - name: ca-ssl
        hostPath:
          path: ${CERT_PATH}
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
EOF1
EOF

执行脚本

bash -x gen_metrics-server_config.sh

# 在当前目录下生成
└── metrics-server.yaml

应用yaml资源文件并查看#

kubectl apply -f metrics-server.yaml

$ kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
metrics-server-84668c4755-8d4gg   1/1     Running   0          76s

查看资源消耗#

$ kubectl top nodes
NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
m01    179m         8%     1244Mi          15%
w01    78m          3%     601Mi           72%
w02    63m          3%     584Mi           70%

$ kubectl top pods -A
NAMESPACE      NAME                              CPU(cores)   MEMORY(bytes)
kube-flannel   kube-flannel-ds-gz6w4             10m          24Mi
kube-flannel   kube-flannel-ds-kxgtw             12m          22Mi
kube-flannel   kube-flannel-ds-qctql             11m          15Mi
kube-system    coredns-b47bf9f44-5pgxg           3m           20Mi
kube-system    metrics-server-84668c4755-8d4gg   3m           16Mi

部署dashboard#

查看dashbord的最新版本，传送门

安装#

创建脚本文件gen_dashboard_config.sh

$ cat <<'EOF'> gen_dashboard_config.sh
CLIENT_CA_FILE=${CERT_PATH}/front-proxy-ca.pem
IMAGE_DASHBOARD=registry.cn-hangzhou.aliyuncs.com/ik8ss/dashboard:v2.7.0
IMAGE_METRICS=registry.cn-hangzhou.aliyuncs.com/ik8ss/metrics-scraper:v1.0.8

cat > dashboard.yaml <<EOF1
apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
---
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30000
  selector:
    k8s-app: kubernetes-dashboard
  type: NodePort
---
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard
---
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: kubernetes-dashboard
          image: ${IMAGE_DASHBOARD}
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: dashboard-metrics-scraper
          image: ${IMAGE_METRICS}
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}
EOF1
EOF

执行

bash -x gen_dashboard_config.sh

kubectl apply -f dashboard.yaml

$ kubectl get pod -A
kubernetes-dashboard   dashboard-metrics-scraper-657cfb9584-zvpr5   1/1     Running   0          80s
kubernetes-dashboard   kubernetes-dashboard-6d67fd849b-k8bj6        1/1     Running   0          80s

使用token访问#

## 创建ServiceAccount用户，用户名为dashboard-admin
kubectl create serviceaccount dashboard-admin -n kube-system

## 创建一个clusterrolebinding，名称为dashboard-admin，将用户dashboard-admin和cluster-admin角色绑定
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

## 查看用户dashboard-admin
$ kubectl describe secrets dashboard-admin -n kube-system
Name:         dashboard-admin
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: fb9e55a7-3fbe-4637-beb3-b471f26d5e9c

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1342 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IjlQV2Y2eWZfOTd0b0xlN01YeTJRNERwcFV3dmFFVTZoNVRqX2tYWXo0SWcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZmI5ZTU1YTctM2ZiZS00NjM3LWJlYjMtYjQ3MWYyNmQ1ZTljIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.XR7g6JIhF41LhyuRlpjVtEMYTAa-5citH0OX35Yh1FYLyAeTdBdHNwHjHUGcWaf2etq-yhfSSECmNEvgoFvWq9tGG685HqryEsqTDd1O6qQ867pNiCFJH-A919_m85b5_r5w82xOzDdXD98HXvbhW1Am_6eklpzvK7XrX4N_i0h-ClNPt3YxLHrFymwTwSdIGg6ik6mSWmHAi5-MxoawnR7HKEVEMM0z2HAcsU4FnrGcv1HX_7SAi55nE5lbbmHDX1aVEsYsfSilswNeh5hpwHoqbFYXblG0Th_aVMd_w_yibYwcPk1W-Q2Ydw2zLshLmPYSGicuqu2ytYbqBLAdHA

有了令牌就可以使用令牌登陆了

$ kubectl get pod -n kubernetes-dashboard -o wide
NAME                                         READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
dashboard-metrics-scraper-657cfb9584-zvpr5   1/1     Running   0          92m   172.16.0.3   m01    <none>           <none>
kubernetes-dashboard-6d67fd849b-k8bj6        1/1     Running   0          92m   172.16.2.2   w02    <none>           <none>

https://192.168.1.10:3000

使用kubeconfig登陆#

cd /root/k8s

$ cat <<'EOF' > kubeconfig_dashboard.sh
DASHBOARD_TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IjlQV2Y2eWZfOTd0b0xlN01YeTJRNERwcFV3dmFFVTZoNVRqX2tYWXo0SWcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZmI5ZTU1YTctM2ZiZS00NjM3LWJlYjMtYjQ3MWYyNmQ1ZTljIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.XR7g6JIhF41LhyuRlpjVtEMYTAa-5citH0OX35Yh1FYLyAeTdBdHNwHjHUGcWaf2etq-yhfSSECmNEvgoFvWq9tGG685HqryEsqTDd1O6qQ867pNiCFJH-A919_m85b5_r5w82xOzDdXD98HXvbhW1Am_6eklpzvK7XrX4N_i0h-ClNPt3YxLHrFymwTwSdIGg6ik6mSWmHAi5-MxoawnR7HKEVEMM0z2HAcsU4FnrGcv1HX_7SAi55nE5lbbmHDX1aVEsYsfSilswNeh5hpwHoqbFYXblG0Th_aVMd_w_yibYwcPk1W-Q2Ydw2zLshLmPYSGicuqu2ytYbqBLAdHA

APISERVER_IP=$1
K8S_CERT_DIR=$2
PORT=6443
KUBE_APISERVER=https://${APISERVER_IP}:${PORT}
KUBECONFIG_FILE=kubeconfig/dashboard-admin.kubeconfig
CLUSTER_NAME=kubernetes
USERNAME=kube-system:dashboard-admin
CONTEXT_NAME=${USERNAME}@${CLUSTER_NAME}

# 设置集群参数
./bin/kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=${K8S_CERT_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBECONFIG_FILE}

# 设置用户认证参数
./bin/kubectl config set-credentials ${USERNAME} \
	--token=${DASHBOARD_TOKEN} \
	--kubeconfig=${KUBECONFIG_FILE}

# 设置context---将用户和集群关联起来
./bin/kubectl config set-context ${CONTEXT_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${USERNAME} \
    --kubeconfig=${KUBECONFIG_FILE}

# 设置默认context
./bin/kubectl config use-context ${CONTEXT_NAME} \
    --kubeconfig=${KUBECONFIG_FILE}
EOF

bash -x kubeconfig_dashboard.sh 192.168.1.10 ssl

# 在kueconfig目录下生成
├── dashboard-admin.kubeconfig

## 下载dashboard-admin.kubeconfig，web登陆是选kubeconfig文件验证

https://192.168.1.10:30000/

集群验证#

集群可用需要通过以下测试项

• pod必须能解析相同namespace下service
• pod必须能解析跨不同namespace的service
• 每个节点必须要能访问k8s的kubernetes的service端口443和kube-dns的service端口53
• pod与pod之间通通信
  • 相同namespace内的pod之间能通信
  • 跨不同namespace的pod之前能通信
  • 跨不同worker节点的pod之间能通信

创建测试pod#

mkdir -p /root/yaml
cd /root/yaml

$ cat <<EOF> busybox.yaml
apiVersion: apps/v1
kind: Deployment    ## 创建一个deployment对象
metadata:
  name: deployment-busybox
  namespace: default
spec:
  replicas: 4    ## 1个pod副本
  selector:
     matchLabels:
         app: busybox
  template:
     metadata:
       labels:
         app: busybox
     spec:
        containers:
        - image: busybox:1.28
          name: busybox-container
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh","-ce","sleep 3600"]
        restartPolicy: Always
EOF

$ kubectl apply -f busybox.yaml 
deployment.apps/deployment-busybox created

$ kubectl get pod -o wide
kubectl get pod -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
deployment-busybox-7bb56cfc66-65rpj   1/1     Running   0          7m8s   172.16.1.4   w01    <none>           <none>
deployment-busybox-7bb56cfc66-7gt85   1/1     Running   0          7m8s   172.16.2.6   w02    <none>           <none>
deployment-busybox-7bb56cfc66-9xkq8   1/1     Running   0          43m    172.16.0.5   m01    <none>           <none>
deployment-busybox-7bb56cfc66-w4mft   1/1     Running   0          15m    172.16.2.5   w02    <none>           <none>

pod必须能解析相同namespace下service#

## 集群安装完毕后，default名称空间下自动创建了一个service，名称为kubernetes
$ kubectl get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes       ClusterIP   10.96.0.1       <none>        443/TCP        10d

$ kubectl exec pod/deployment-busybox-7bb56cfc66-9xkq8 -- nslookup kubernetes
error: unable to upgrade connection: Forbidden (user=kube-apiserver, verb=create, resource=nodes, subresource=proxy)

## 提示user kube-apiserver被禁止，apiserver访问kubelet也需要授权，创建一个集群角色绑定--apiserver-kubelet
$ kubectl create clusterrolebinding kube-apiserver --clusterrole=system:kubelet-api-admin --user=kube-apiserver

## 再次查询，可以正常解析
$ kubectl exec deployment-busybox-7bb56cfc66-9xkq8 -- nslookup kubernetes
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

跨不同namespace的service#

## busybox在default下，kube-dns在kube-system下，可以正常解析
$ kubectl exec deployment-busybox-7bb56cfc66-9xkq8  -n default -- nslookup kube-dns.kube-system
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

每个节点必须要能访问k8s的kubernetes svc 443和kube-dns的 service的53#

## 用telnet命令测试k8s service的443端口和kube-dns的53端口。
$ telnet 10.96.0.1 443
Trying 10.96.0.1...
Connected to 10.96.0.1.
Escape character is '^]'.

$ telnet 10.96.0.10 53
Trying 10.96.0.10...
Connected to 10.96.0.10.
Escape character is '^]'.
Connection closed by foreign host.
# 2个service的端口可以telnet

pod与pod之间通通信#

$ kubectl get pod -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
deployment-busybox-7bb56cfc66-65rpj   1/1     Running   0          7m8s   172.16.1.4   w01    <none>           <none>
deployment-busybox-7bb56cfc66-7gt85   1/1     Running   0          7m8s   172.16.2.6   w02    <none>           <none>
deployment-busybox-7bb56cfc66-9xkq8   1/1     Running   0          43m    172.16.0.5   m01    <none>           <none>
deployment-busybox-7bb56cfc66-w4mft   1/1     Running   0          15m    172.16.2.5   w02    <none>           <none>

## 相同namespace,相同worker节点之间的通信
$ kubectl exec deployment-busybox-7bb56cfc66-7gt85 -- ping -c 2 172.16.1.4
PING 172.16.1.4 (172.16.1.4): 56 data bytes
64 bytes from 172.16.1.4: seq=0 ttl=62 time=1.908 ms
64 bytes from 172.16.1.4: seq=1 ttl=62 time=2.495 ms

## 相同namespace,不同worker节点之间的通信
$ kubectl exec deployment-busybox-7bb56cfc66-7gt85 -- ping -c 2 172.16.0.5
PING 172.16.0.5 (172.16.0.5): 56 data bytes
64 bytes from 172.16.0.5: seq=0 ttl=62 time=2.739 ms
64 bytes from 172.16.0.5: seq=1 ttl=62 time=2.156 ms

## 不同namespace之间的通信
$ kubectl get pod -n kube-system -owide
NAME                              READY   STATUS    RESTARTS   AGE   IP             NODE   NOMINATED NODE   READINESS GATES
coredns-556b886bc4-24b8s          1/1     Running   0          82m   172.16.0.2     m01    <none>           <none>

$ kubectl exec deployment-busybox-7bb56cfc66-7gt85 -- ping -c 2 172.16.0.2
PING 172.16.0.2 (172.16.0.2): 56 data bytes
64 bytes from 172.16.0.2: seq=0 ttl=62 time=2.924 ms
64 bytes from 172.16.0.2: seq=1 ttl=62 time=3.008 ms

至此一个学习用的k8s集群搭建完毕。

部署一个应用#

$ kubectl create deployment nginx-dpm --image=nginx:1.14-alpine
deployment.apps/nginx created

$ kubectl get pod -o wide
NAME                                  READY   STATUS    RESTARTS          AGE    IP           NODE   NOMINATED NODE   READINESS GATES
nginx-dpm-658767d5f-b4278             1/1     Running   0                 4m2s   172.16.0.6   m01    <none>           <none>

$ kubectl expose deployment nginx-dpm --port=80 --type=NodePort
service/nginx exposed

$ kubectl get svc 
NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
service/nginx-dpm    NodePort    10.96.73.96   <none>        80:37988/TCP   15s

$ curl http://192.168.1.10:37988
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

## 查看pod的ip网段和所在主机,pod被调度在m01主机上
$ kubectl get pod nginx-dpm-658767d5f-b4278 -owide
NAME                        READY   STATUS    RESTARTS   AGE     IP           NODE   NOMINATED NODE   READINESS GATES
nginx-dpm-658767d5f-b4278   1/1     Running   0          7m28s   172.16.0.6   m01    <none>           <none>

## 查看m01的PodCIDR也是在172.16.0.0/24，说明pod的IP从这个cidr进行分配
$ kubectl describe nodes m01
....
PodCIDR:                      172.16.0.0/24
PodCIDRs:                     172.16.0.0/24
....

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 ens32
172.16.0.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
172.16.1.0      192.168.1.11    255.255.255.0   UG    0      0        0 ens32
172.16.2.0      172.16.2.0      255.255.255.0   UG    0      0        0 flannel.1
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens32

$ kubectl exec nginx-dpm-658767d5f-b4278 -- ip a
error: unable to upgrade connection: Forbidden (user=kube-apiserver, verb=create, resource=nodes, subresource=proxy)

## 提示user kube-apiserver被禁止
kubectl create clusterrolebinding apiserver-kubelet --clusterrole=system:kubelet-api-admin --user=kube-apiserver

$ kubectl exec nginx-dpm-658767d5f-b4278 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 46:6b:9a:cb:f7:50 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.6/24 brd 172.16.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::446b:9aff:fecb:f750/64 scope link 
       valid_lft forever preferred_lft forever