二进制方式部署K8S（kubernetes)集群（测试、学习环境）-单主双从
1. 二进制方式部署（一主多从）
1.1 环境准备
|
角色 |
IP |
组件 |
|
master |
10.27.134.250 |
kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubectl 、kubelet 、kube-proxy |
|
node1 |
10.27.134.251 |
docker、etcd、kubelet 、kube-proxy |
|
node2 |
10.27.134.252 |
docker、etcd、kubelet 、kube-proxy |
##版本说明
Linux：CentOS Linux release 7.6.1810 (Core)
Container runtimes : docker-20.10.9
Etcd : v3.5.1
Kubernetes : v1.23.1
签名工具：
cfssl
cfssljson
cfssl-certinfo
1.2 系统初始化（所有节点均需操作）
1.2.1 主机名解析
##修改主机名
[root@jss-k8s-1 ~]# hostnamectl set-hostname master #master节点
[root@jss-k8s-2 ~]# hostnamectl set-hostname node1 #node1节点
[root@jss-k8s-3 ~]# hostnamectl set-hostname node2 #node2节点
##三台服务器的地址和主机名添加到三个节点的/etc/hosts文件中
[root@master ~]# cat >> /etc/hosts << EOF
10.27.134.250 master
10.27.134.251 node1
10.27.134.252 node2
EOF
1.2.2 时间同步
##kubernetes要求集群中的节点时间必须精确一直，这里使用chronyd服务从网络同步时间
# 启动chronyd服务，设置开机启动
# systemctl start chronyd && systemctl enable chronyd
# 核验时间
# date
1.2.3 禁用iptable和firewalld服务
##kubernetes和docker 在运行的中会产生大量的iptables规则，为了不让系统规则跟它们混淆，直接关闭系统的规则
##关闭firewalld服务
# systemctl stop firewalld && systemctl disable firewalld
##关闭iptables服务
# systemctl stop iptables && systemctl disable iptables
1.2.4 禁用selinux
##selinux是linux系统下的一个安全服务，如果不关闭它，在安装集群中会产生各种各样的奇葩问题
# 永久关闭
# sed -i 's/enforcing/disabled/' /etc/selinux/config
# 临时关闭
# setenforce 0
1.2.5 禁用swap分区
##swap分区指的是虚拟内存分区，它的作用是物理内存使用完，之后将磁盘空间虚拟成内存来使用，启用swap设备会对系统的性能产生非常负面的影响，因此kubernetes要求每个节点都要禁用swap设备，但是如果因为某些原因确实不能关闭swap分区，就需要在集群安装过程中通过明确的参数进行配置说明
# 永久关闭
# vim /etc/fstab
# 临时关闭
# swapoff -a
1.2.6 将桥接的 IPv4流量传递到 iptables的链上
##加载网桥过滤模块
# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
##修改内核参数
# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
##配置生效
# sysctl --system
##重启系统，让配置彻底生效
# reboot
1.3 部署二进制包
1.3.1 安装docker
1.3.1.1 下载二进制包
操作对象：所有节点
##在master主机创建目录存放二进制软件包
# mkdir -p /usr/local/k8s
# cd /usr/local/k8s
##下载docker包，
# wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz
##解压
# tar zxvf docker-20.10.9.tgz
##复制到本地运行目录
# mv docker/* /usr/bin
1.3.1.2 systemd管理docker
操作对象：所有节点
##创建docker.service脚本
# cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
1.3.1.3 创建配置文件，配置镜像加速
操作对象：所有节点
#在所有节点主机创建docker配置文件默认的目录
# mkdir /etc/docker
# cat > /etc/docker/daemon.json<< EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
1.3.1.4 启动docker并设置开机启动
# systemctl daemon-reload && systemctl start docker && systemctl enable docker
1.3.2 部署etcd集群
1.3.2.1 下载cfssl
操作对象：master节点
# cd /usr/local/k8s
# cat > cfssl << EOF
https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
EOF
# wget -i cfssl
#赋予可执行的权限
# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
# mv cfssl_linux-amd64 /usr/local/bin/cfssl
# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
# mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
1.3.2.2 创建自签证书颁发机构（CA）的配置文件
##创建证书生成的工作目录（后面相关证书的生成都在此目录下操作）
# mkdir -p ~/TLS/{etcd,k8s}
# cd ~/TLS/etcd
## 创建ca-config配置文件
# cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
##创建ca-csr配置文件
# cat > ca-csr.json << EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
1.3.2.3 生成证书
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca –
##会生成ca.pem和ca-key.pem文件
# ls *pem
1.3.2.4 自签CA签发etcd HTTPS证书
##创建证书申请文件
# cat > server-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"10.27.134.250",
"10.27.134.251",
"10.27.134.252"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
##生成HTTPS证书
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
##会生成server.pem和server-key.pem文件
# ls server*pem
1.3.2.5 下载ETCD二进制包
# cd /usr/local/k8s
# wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz
##创建etcd的工作目录
# mkdir -p /opt/etcd/{bin,cfg,ssl}
#解压
# tar zxvf etcd-v3.5.1-linux-amd64.tar.gz
##将二进制文件复制到工作目录
# cp etcd-v3.5.1-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
1.3.2.6 创建etcd 配置文件
# cat > /opt/etcd/cfg/etcd.conf << EOF
#根据实际情况修改节点地址
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.27.134.250:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.27.134.250:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.27.134.250:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.27.134.250:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.27.134.250:2380,etcd-2=https://10.27.134.251:2380,etcd-3=https://10.27.134.252:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_ENABLE_V2="true"
EOF
##解释：
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIALCLUSTER_TOKEN：集群Token
ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
1.3.2.7 systemd 管理etcd
##将之前生成的证书复制到etcd的工作目录
# cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl
# cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
1.3.2.8 将配置文件、证书和etcd.service文件复制到其他节点
##node1
# scp -r /opt/etcd/ root@node1:/opt/
# scp /usr/lib/systemd/system/etcd.service root@node1:/usr/lib/systemd/system/
##node2
# scp -r /opt/etcd/ root@node2:/opt/
# scp /usr/lib/systemd/system/etcd.service root@node2:/usr/lib/systemd/system/
1.3.2.9 修改其他节点的etcd配置
##node1：修改etcd.conf 配置文件中的节点名称和节点地址
# vim /opt/etcd/cfg/etcd.conf
##node2：修改etcd.conf 配置文件中的节点名称和节点地址
# vim /opt/etcd/cfg/etcd.conf
1.3.2.10 启动etcd集群
##启动etcd服务并设置开机启动，若在node上启动失败，可以先启动master。
# systemctl daemon-reload && systemctl start etcd && systemctl enable etcd
##检查集群状态
# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://10.27.134.250:2379,https://10.27.134.251:2379,https://10.27.134.252:2379" endpoint health --write-out=table
1.3.3 部署Master Node
1.3.3.1 部署kube-apiserver
1.3.3.1.1 生成kube-apiserver证书
（1）创建自签证书颁发机构（CA）配置
# cd ~/TLS/k8s
##创建ca-config配置文件
# cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
##创建ca-csr配置文件
# cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
（2） 生成证书
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
#会生成ca.pem和ca-key.pem文件
# ls *pem
（3） 使用自签证书签发kube-apiserver HTTPS证书
##创建证书申请文件
# cat > server-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"10.27.134.250",
"10.27.134.251",
"10.27.134.252",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
（4）生成https证书
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
##会生成server.pem和server-key.pem文件
# ls server*pem
1.3.3.1.2 下载二进制包
# cd /usr/local/k8s/
# wget https://dl.k8s.io/v1.23.1/kubernetes-server-linux-amd64.tar.gz
# tar zxvf kubernetes-server-linux-amd64.tar.gz
##创建kubernetes工作目录
# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
##从软件包内拷贝组件
# cd /usr/local/k8s/kubernetes/server/bin
# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
# cp kubectl /usr/bin/
1.3.3.1.3 生成kube-apiserver的token文件
# echo $(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" > /opt/kubernetes/cfg/token.csv
1.3.3.1.4 创建配置文件
# cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://10.27.134.250:2379,https://10.27.134.251:2379,https://10.27.134.252:2379 \\
--bind-address=10.27.134.250 \\
--secure-port=6443 \\
--advertise-address=10.27.134.250 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=api \\
--service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF
–logtostderr：启用日志
––v：日志等级
–log-dir：日志目录
–etcd-servers：etcd 集群地址
–bind-address：监听地址
–secure-port：https 安全端口
–advertise-address：集群通告地址
–allow-privileged：启用授权
–service-cluster-ip-range：Service 虚拟IP 地址段
–enable-admission-plugins：准入控制模块
–authorization-mode：认证授权，启用RBAC 授权和节点自管理
–enable-bootstrap-token-auth：启用TLS bootstrap 机制
–token-auth-file：bootstrap token 文件
–service-node-port-range：Service nodeport 类型默认分配端口范围
–kubelet-client-xxx：apiserver 访问kubelet 客户端证书
–tls-xxx-file：apiserver https 证书
–etcd-xxxfile：连接Etcd 集群证书
–audit-log-xxx：审计日志
–service-account-issuer=string 指定service account token issuer的标识符;该issuer将在iss声明中分发Token以使标识符生效;该参数的值为字符串或URL
–service-account-signing-key-file=string含有当前service account token issuer私钥的文件路径 ;该issuer将以此私钥签名发布的ID token（需开启TokenRequest特性）
1.3.3.1.5 拷贝生成的k8s证书
# cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl
1.3.3.1.6 system管理apiserver
# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
1.3.3.1.7 启动 kube-apiserver并设置开机启动
# systemctl daemon-reload && systemctl start kube-apiserver && systemctl enable kube-apiserver
1.3.3.2 部署kube-controller-manager
1.3.3.2.1 创建证书申请文件
# cd ~/TLS/k8s
# cat > kube-controller-manager-csr.json << EOF
{
"CN": "system:kube-controller-manager",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
1.3.3.2.2 生成证书
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
1.3.3.2.3 生成kube-controller-manager.kubeconfig配置文件
# KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
# KUBE_APISERVER="https://10.27.134.250:6443"
# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-credentials kube-controller-manager \
--client-certificate=./kube-controller-manager.pem \
--client-key=./kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-controller-manager \
--kubeconfig=${KUBE_CONFIG}
# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
1.3.3.2.4 创建配置文件
# cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF
##解释：
–master：通过本地非安全本地端口8080 连接apiserver
–leader-elect：当该组件启动多个时，自动选举（HA）
–cluster-signing-cert-file/–cluster-signing-key-file：自动为kubelet 颁发证书的CA，与apiserver 保持一致
1.3.3.2.5 systemd 管理controller-manager
# cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/Kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
1.3.3.2.6 启动kube-controller-manager并设置开机启动
# systemctl daemon-reload && systemctl start kube-controller-manager && systemctl enable kube-controller-manager
1.3.3.3 部署kube-scheduler
1.3.3.3.1 创建证书申请文件
# cd ~/TLS/k8s
# cat > kube-scheduler-csr.json << EOF
{
"CN": "system:kube-scheduler",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
1.3.3.3.2 生成证书
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
1.3.3.3.3 生成kube-scheduler.kubeconfig配置文件
# KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
# KUBE_APISERVER="https://10.27.134.250:6443"
# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-credentials kube-scheduler \
--client-certificate=./kube-scheduler.pem \
--client-key=./kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-scheduler \
--kubeconfig=${KUBE_CONFIG}
# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
1.3.3.3.4 创建配置文件
# cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect \\
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF
##解释：
–master：通过本地非安全本地端口8080 连接apiserver。
–leader-elect：当该组件启动多个时，自动选举（HA）
1.3.3.3.5 systemd 管理scheduler
# cat > /usr/lib/systemd/system/kube-scheduler.service<< EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/Kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
1.3.3.3.6 启动kube-scheduler并设置开机启动
# systemctl daemon-reload && systemctl start kube-scheduler && systemctl enable kube-scheduler
1.3.3.4 查看集群状态
1.3.3.4.1 生成kubectl连接集群的证书
##创建证书申请文件
# cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
1.3.3.4.2 生成证书
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
1.3.3.4.3 生成kubectl连接集群的配置文件
##创建配置文件位置
# mkdir /root/.kube
##创建配置文件
# KUBE_CONFIG="/root/.kube/config"
# KUBE_APISERVER="https://10.27.134.250:6443"
# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-credentials cluster-admin \
--client-certificate=./admin.pem \
--client-key=./admin-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-context default \
--cluster=kubernetes \
--user=cluster-admin \
--kubeconfig=${KUBE_CONFIG}
# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
1.3.3.4.4 查看集群组件状态
# kubectl get cs
1.3.3.5 授权kubelet-bootstrap用户允许请求证书
# kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
1.3.4 部署Worker Node
##master节点也部署成worker节点，一并参加调度
1.3.4.1 创建工作目录并拷贝二进制文件
##在worker node中创建目录
# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
##进入解压的kubernetes文件夹中
# cd /usr/local/k8s/kubernetes/server/bin
##将kubelet、kube-proxy组件拷贝至本地文件中
# cp kubelet kube-proxy /opt/kubernetes/bin
1.3.4.2 部署kubelet
1.3.4.2.1 创建配置文件
# cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=master \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF
##解释：
--hostname-override：显示名称，集群中唯一
--network-plugin：启用CNI
--kubeconfig：空路径，会自动生成，后面用于连接apiserver
--bootstrap-kubeconfig：首次启动向apiserver申请证书
--config：配置参数文件
--cert-dir：kubelet证书生成目录
--pod-infra-container-image：管理Pod网络容器的镜像
1.3.4.2.2 配置参数文件
# cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
1.3.4.2.3 生成kubelet初次加入集群引导kubeconfig文件
# KUBE_CONFIG="/opt/kubernetes/cfg/bootstrap.kubeconfig"
# KUBE_APISERVER=https://10.27.134.250:6443
##TOKEN值与token.csv里保持一致
# TOKEN="1b4c126d51e07e21bb272b144b255c14"
##生成 kubelet bootstrap kubeconfig 配置文件
# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-credentials "kubelet-bootstrap" \
--token=${TOKEN} \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=${KUBE_CONFIG}
# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
1.3.4.2.4 systemctl管理kubelet
# cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
1.3.4.2.5 启动kubelet并设置开机启动
# systemctl daemon-reload && systemctl start kubelet && systemctl enable kubelet
##注意：kubernetes官方推荐docker等使用systemd作为Cgroup Driver
##如果kubelet服务状态异常，查看日志是cgroupDriver不匹配的话
##先查看docker使用的Cgroup Driver
# docker info|grep "Cgroup Driver"
# 再修改kubelet的yaml文件中Cgroup Driver的参数
1.3.4.3 批准 kubelet 证书申请
##每个节点部署申请后都需被批准才能加入集群
（1）在master节点查看 kubelet 证书请求
# kubectl get csr
（2）批准申请
# kubectl certificate approve node-csr-31V7IbsN3zDicNmLUTyIs-s9uv9biTzpkMRE6vPrx7w
（3）查看节点
# kubectl get nodes
1.3.4.4 部署kube-proxy
1.3.4.4.1 创建证书申请文件
# cd ~/TLS/k8s
# cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
1.3.4.4.2 生成证书
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
1.3.4.4.3 创建配置文件
# cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF
1.3.4.4.4 创建参数文件
cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: master
clusterCIDR: 10.244.0.0/16
EOF
1.3.4.4.4 生成kube-proxy.kubeconfig配置文件
# KUBE_CONFIG="/opt/kubernetes/cfg/kube-proxy.kubeconfig"
# KUBE_APISERVER="https://10.27.134.250:6443"
# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-credentials kube-proxy \
--client-certificate=./kube-proxy.pem \
--client-key=./kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=${KUBE_CONFIG}
# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
1.3.4.4.5 systemd管理kube-proxy
# cat > /usr/lib/systemd/system/kube-proxy.service<< EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
1.3.4.4.6 启动kube-proxy并设置开机启动
# systemctl daemon-reload && systemctl start kube-proxy && systemctl enable kube-proxy
1.3.4.5 部署网络组件（在master节点部署）
1.3.4.5.1 Calico
##Calico是一个纯三层的数据中心网络方案，是目前Kubernetes的主流方案
（1）下载配置文件
# cd /usr/local/k8s
# wget https://docs.projectcalico.org/v3.20/manifests/calico.yaml --no-check-certificate
（2）应用yaml文件
# kubectl apply -f calico.yaml
（3）在apply或create命令无法下载镜像时通过docker pull 来拉取镜像
# cat calico.yaml |grep image
1.3.4.5.2 查看组件状态
（1）查看部署结果
# kubectl get pods -n kube-system
（2）结果实时查看
# kubectl get pods -n kube-system -w
（3）查看节点状态
# kubectl get node
1.3.4.6 授权apiserver访问kubelet
应用场景：例如kubectl logs
# cd /usr/local/k8s/
# cat > apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF
##应用yaml文件
# kubectl apply -f apiserver-to-kubelet-rbac.yaml
##查看应用
# kubectl get ClusterRole
1.3.5 新增Worker Node
1.3.5.1 拷贝已部署好的Node相关文件到新节点
##在Master节点将Worker Node涉及的文件拷贝到新节点node1、node2
# scp -r /opt/kubernetes root@node1:/opt/
# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@node1:/usr/lib/systemd/system
# scp /opt/kubernetes/ssl/ca.pem root@node1:/opt/kubernetes/ssl
1.3.5.2 删除kubelet证书和kubeconfig文件
# rm -f /opt/kubernetes/cfg/kubelet.kubeconfig
# rm -f /opt/kubernetes/ssl/kubelet*
注：这几个文件是证书申请审批后自动生成的，每个Node不同，必须删除
否则master节点无法获得node节点信息
1.3.5.3 修改主机名
# vim /opt/kubernetes/cfg/kubelet.conf
--hostname-override=node1
# vim /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: node1
1.3.5.4 启动kubelet、kube-proxy并设置开机启动
# systemctl daemon-reload && systemctl start kubelet kube-proxy && systemctl enable kubelet kube-proxy
1.3.5.5 在Master上批准新Node kubelet证书申请
##查看证书请求
# kubectl get csr
##授权请求
# kubectl certificate approve node-csr-31V7IbsN3zDicNmLUTyIs-s9uv9biTzpkMRE6vPrx7w
1.3.5.6 查看Node状态
# kubectl get node
##node2节点及更多节点同上。记得删除kubelet证书、配置文件、修改主机名！
1.3.6 部署dashboard可视化模块
1.3.6.1 服务部署
##下载yml文件
# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc5/aio/deploy/recommended.yaml
##修改yam文件，service改为NodePort类型
##创建服务
# kubectl create -f recommended.yaml
##查看服务对外暴露的端口号
# kubectl get svc -n kubernetes-dashboard
1.3.6.2 访问测试
##浏览器访问https://NodeIP:Port
1.3.6.3 创建ServiceAccount进行登录
##创建admin.conf配置文件
# cat > /opt/kubernetes/cfg/admin.conf << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: admin
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: admin
namespace: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin
namespace: kubernetes-dashboard
EOF
##查看命名空间内的加密信息是否有token和名称
# kubectl get secret -n kubernetes-dashboard
##使用该命令拿到token，替换上条命令查询的名称default-token-9xlqd，然后粘贴到浏览器中进行登录
# kubectl -n kubernetes-dashboard get secret default-token-9xlqd -o jsonpath={.data.token}|base64 -d
##使用token登录dashboard
本文来自博客园，作者：苏冷，转载请注明原文链接：https://www.cnblogs.com/jiangshusheng/p/15853195.html
