K8s集群离线环境部署
1.环境说明
虚拟机工具:VirtualBox 7.0
虚拟机操作系统:Centos 7(CentOS-7-x86_64-Minimal-2003.iso)
虚拟机配置:2核CPU、2G内存、40G硬盘
虚拟机网络模式:仅主机(Host-Only)网络,此网络模式下物理机能ping通虚拟机,虚拟机之间能相互ping通,虚拟机无法联网
K8s版本:v1.20
Docker版本:19.03.9
SSH工具:FinalShell
单Master节点虚拟机角色及IP规划:
Master1:192.168.56.201
Node1:192.168.56.202
Node2:192.168.56.203
软件部署规划:
Master1:kube-apiserver,kube-controller-manager,kube-scheduler,kubelet,kube-proxy,etcd,registry,ntp
Node1:kubelet,kube-proxy,docker,etcd,ntp
Node2:kubelet,kube-proxy,docker,etcd,ntp
部署所用到的文件百度网盘下载:
链接:https://pan.baidu.com/s/14teRvxhy2RcsduGWGMgUzw?pwd=navy
提取码:navy
2.准备工作
2.1 设置Master1、Node1、Node2虚拟机为静态ip(Master&Node)
2.2 将网盘下载的文件放到3台虚拟机的/opt/release目录下(Master&Node)
2.3 升级linux内核为5.4.228并重启(Master&Node)
cd /opt/release/rpm/kernel
rpm -ivh *.rpm --force --nodeps
grub2-set-default 'CentOS Linux (5.4.228-1.el7.elrepo.x86_64) 7 (Core)'
reboot
2.4 重启后验证是否升级成功(Master&Node)
2.5 系统初始化设置,注意主机名需要改(Master&Node)
# 1、关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 2、关闭selinux sed -i 's/enforcing/disabled/' /etc/selinux/config setenforce 0 # 3、关闭swap swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab # 4、根据规划设置主机名 hostnamectl set-hostname k8s-master1 # 5、在master添加hosts cat >> /etc/hosts << EOF 192.168.56.201 k8s-master1 192.168.56.202 k8s-node1 192.168.56.203 k8s-node2 EOF
# 6、将桥接的IPv4流量传递到iptables的链 cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system
# 7、安装常用工具 ipvsadm ipset(ipvs管理工具)conntrack(连接跟踪,service和ipvs会用到)nfs-utils(NFS客户端)
yum install vim nfs-utils wget conntrack ipvsadm ipset -y
3.部署NTP
3.1 说明
NTP服务用于k8s集群时间同步,如果内网环境有时间服务器,则Master和Node部署为客户端即可;没有时间服务器,则Master充当时间服务器部署为服务端,Node节点部署为客户端同步Master节点的时间。
3.2 部署NTP服务端(Master)
3.2.1 安装ntp
cd /opt/release/rpm/ntp/
rpm -ivh *.rpm --force --nodeps
3.2.2 修改ntp.conf
cat > /etc/ntp.conf << EOF
driftfile /var/lib/ntp/drift
restrict default nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict ::1
restrict 192.168.56.0 mask 255.255.255.0 nomodify notrap
server ntp.aliyun.com iburst
server ntp1.aliyun.com iburst
server 127.127.1.0
fudge 127.127.1.0 stratum 10
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
disable monitor
EOF
3.2.3 启动服务并设置开机启动
systemctl start ntpd
systemctl enable ntpd
3.3 部署NTP客户端(Node)
3.3.1 安装ntp
cd /opt/release/rpm/ntp/
rpm -ivh *.rpm --force --nodeps
3.3.2 修改ntp.conf
cat > /etc/ntp.conf << EOF driftfile /var/lib/ntp/drift restrict default nomodify notrap nopeer noquery restrict 127.0.0.1 restrict ::1 server 192.168.56.201 iburst includefile /etc/ntp/crypto/pw keys /etc/ntp/keys disable monitor EOF
3.3.3 启动服务并设置开机启动
systemctl start ntpd
systemctl enable ntpd
3.4 查看同步情况
ntpq -p
字段说明:
第一列中的字符指示源的质量。星号 ( * ) 表示该源是当前引用。
remote 列出源的 IP 地址或主机名。
when 指出从轮询源开始已过去的时间(秒)。
poll 指出轮询间隔时间。该值会根据本地时钟的精度相应增加。
reach 是一个八进制数字,指出源的可存取性。值 377 表示源已应答了前八个连续轮询。
offset 是源时钟与本地时钟的时间差(毫秒)。
4 安装cfssl证书管理工具(Master)
cd /opt/release/cfssl chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
5 部署etcd
5.1 生成自签证书颁发机构(CA)(Master)
mkdir -p ~/TLS/{etcd,k8s} && cd ~/TLS/etcd cat > ca-config.json << EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "www": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json << EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
5.2 生成证书请求文件(Master)
cat > server-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"192.168.56.201",
"192.168.56.202",
"192.168.56.203"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
5.3 生成证书(Master)
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
5.4 安装etcd(Master)
mkdir /opt/etcd/{bin,cfg,ssl} -p mv /opt/release/bin/etcd-v3.4.9-linux-amd64.tar.gz /opt/etcd/bin/ cd /opt/etcd/bin tar zxvf etcd-v3.4.9-linux-amd64.tar.gz mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
5.5 创建etcd配置文件(Master)
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.56.201:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.56.201:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.201:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.201:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.56.201:2380,etcd-2=https://192.168.56.202:2380,etcd-3=https://192.168.56.203:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
配置文件说明:
ETCD_NAME:节点名称,集群中唯一
ETCD_DATA_DIR:数据目录
ETCD_LISTEN_PEER_URLS:集群通信监听地址
ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEERURLS:集群通告地址
ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
ETCD_INITIAL_CLUSTER:集群节点地址
ETCD_INITIALCLUSTER_TOKEN:集群Token
ETCD_INITIALCLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群
5.6 systemd管理etcd(Master)
cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
5.7 拷贝证书到指定位置(Master)
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/
5.8 启动并设置开机启动(Master)
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
出现这个没关系,等另外两个etcd起来就好了
5.9 将文件拷贝到192.168.56.202和192.168.56.203,一行一行执行要输入密码(Master)
scp -r /opt/etcd/ root@192.168.56.202:/opt/ scp /usr/lib/systemd/system/etcd.service root@192.168.56.202:/usr/lib/systemd/system/ scp -r /opt/etcd/ root@192.168.56.203:/opt/ scp /usr/lib/systemd/system/etcd.service root@192.168.56.203:/usr/lib/systemd/system/
5.10 修改Node节点上的etcd.conf(Node)
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.56.202:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.56.202:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.202:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.202:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.56.201:2380,etcd-2=https://192.168.56.202:2380,etcd-3=https://192.168.56.203:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.56.203:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.56.203:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.203:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.203:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.56.201:2380,etcd-2=https://192.168.56.202:2380,etcd-3=https://192.168.56.203:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
5.11 启动并设置开机启动(Node)
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
5.12 查看etcd集群状态(Master)
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.56.201:2379,https://192.168.56.202:2379,https://192.168.56.203:2379" endpoint health --write-out=table
6 部署Docker(Master&Node)
6.1 安装Docker
mkdir /opt/docker/bin -p mv /opt/release/bin/docker-19.03.9.tgz /opt/docker/bin/ cd /opt/docker/bin tar zxvf docker-19.03.9.tgz mv docker/* /usr/bin
6.2 systemd管理docker
cat > /usr/lib/systemd/system/docker.service << EOF [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF
6.3 生成镜像源配置文件
mkdir /etc/docker cat > /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
"insecure-registries": ["192.168.56.201:5000"] } EOF
insecure-registries:私有仓库地址,即registry的地址,后面会部署
6.4 启动并设置开机启动
systemctl daemon-reload
systemctl start docker
systemctl enable docker
7 部署Registry(Master)
7.1 说明
如果内网中已经有私有仓库,则可以跳过此步骤,docker配置文件中的insecure-registries改为现有的私有仓库地址;如果没有,也可以单独弄一台虚拟机部署镜像仓库比如harbor,我本机电脑配置有限,就在Master节点上部署。
7.2 加载离线registry镜像到本地
docker load -i /opt/release/image/registry/registry.docker
7.3 创建挂载目录
mkdir /opt/registry
7.4 运行registry镜像
docker run -dit --restart=always --name=docker-registry -p 5000:5000 -v /opt/registry:/var/lib/registry hub.c.163.com/library/registry:latest
7.5 测试
curl http://192.168.56.201:5000/v2/_catalog
7.6 将离线镜像上传到私有仓库
7.6.1 创建shell脚本
cd /opt/release/image touch loadandpushimages.sh chmod +x loadandpushimages.sh
7.6.2 编辑shell脚本
#! /bin/bash # 装载离线镜像到本地 for directory in "amd64" "busybox" "calico" "coredns" "kubernetes-dashboard" do for file in /opt/release/image/$directory/* do docker load -i $file done done # 遍历本地镜像并推送registry image_id=$(docker images -q | sort -u) for i in ${image_id}; do image_tag=`docker inspect $i --format='{{.RepoTags}}' | sed 's/\[//g' | sed 's/\]//g'` docker tag $image_tag 192.168.56.201:5000/$image_tag docker push 192.168.56.201:5000/$image_tag done
7.6.3 执行shell脚本
./loadandpushimages.sh
7.6.4 验证镜像是否推送成功
curl http://192.168.56.201:5000/v2/_catalog
8 部署kube-apiserver(Master)
8.1 创建自签CA
cd ~/TLS/k8s cat > ca-config.json << EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json << EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] } EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
8.2 使用自签CA签发kube-apiserver HTTPS证书
cat > server-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.56.201",
"192.168.56.202",
"192.168.56.203",
"192.168.56.204",
"192.168.56.205",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
192.168.56.201为master1节点,202和203为node节点,192.168.56.204和192.168.56.205是预留的ip,可以是master2节点和vip(虚拟ip)
8.3 解压kubernetes二进制包
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
cd /opt/release/bin
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd /opt/release/bin/kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager kubelet kube-proxy /opt/kubernetes/bin
cp kubectl /usr/bin/
8.4 创建kube-apiserver配置文件
cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://192.168.56.201:2379,https://192.168.56.202:2379,https://192.168.56.203:2379 \\
--bind-address=192.168.56.201 \\
--secure-port=6443 \\
--advertise-address=192.168.56.201 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=api \\
--service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF
参数说明:
--logtostderr:启用日志
--v:日志等级
--log-dir:日志目录
--etcd-servers:etcd集群地址
--bind-address:监听地址
--secure-port:https安全端口
--advertise-address:集群通告地址
--allow-privileged:启用授权
--service-cluster-ip-range:Service虚拟IP地址段
--enable-admission-plugins:准入控制模块
--authorization-mode:认证授权,启用RBAC授权和节点自管理
--enable-bootstrap-token-auth:启用TLS bootstrap机制
--token-auth-file:bootstrap token文件
--service-node-port-range:Service nodeport类型默认分配端口范围
--kubelet-client-xxx:apiserver访问kubelet客户端证书
--tls-xxx-file:apiserver https证书
1.20版本必须加的参数:--service-account-issuer,--service-account-signing-key-file
--etcd-xxxfile:连接Etcd集群证书
--audit-log-xxx:审计日志
启动聚合层相关配置:--requestheader-client-ca-file,--proxy-client-cert-file,--proxy-client-key-file,--requestheader-allowed-names,--requestheader-extra-headers-prefix,--requestheader-group-headers,--requestheader-username-headers,--enable-aggregator-routing
8.5 拷贝证书
cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/
8.6 创建token文件
cat > /opt/kubernetes/cfg/token.csv << EOF c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper" EOF
8.7 systemd管理kube-apiserver
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
8.8 启动并设置开机启动
systemctl daemon-reload systemctl start kube-apiserver systemctl enable kube-apiserver
9 部署kube-controller-manager(Master)
9.1 创建kube-controller-manager配置文件
cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF
配置说明:
--kubeconfig:连接apiserver配置文件
--leader-elect:当该组件启动多个时,自动选举(HA)
--cluster-signing-cert-file/--cluster-signing-key-file:自动为kubelet颁发证书的CA,与apiserver保持一致
9.2 生成证书
cd ~/TLS/k8s cat > kube-controller-manager-csr.json << EOF { "CN": "system:kube-controller-manager", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
9.3 生成kubeconfig文件(以下是shell命令,直接在终端执行)
KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://192.168.56.201:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-controller-manager \
--client-certificate=./kube-controller-manager.pem \
--client-key=./kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-controller-manager \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
9.4 systemd管理kube-controller-manager
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF
9.5 启动并设置开机启动
systemctl daemon-reload systemctl start kube-controller-manager systemctl enable kube-controller-manager
10 部署kube-scheduler
10.1 创建配置文件
cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF KUBE_SCHEDULER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs \\ --leader-elect \\ --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\ --bind-address=127.0.0.1" EOF
参数说明:
--kubeconfig:连接apiserver配置文件
--leader-elect:当该组件启动多个时,自动选举(HA)
10.2 生成证书
cd ~/TLS/k8s cat > kube-scheduler-csr.json << EOF { "CN": "system:kube-scheduler", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
10.3 生成kubeconfig文件(以下是shell命令,直接在终端执行)
KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://192.168.56.201:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-scheduler \
--client-certificate=./kube-scheduler.pem \
--client-key=./kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-scheduler \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
10.4 systemd管理scheduler
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF
10.5 启动并设置开机启动
systemctl daemon-reload systemctl start kube-scheduler systemctl enable kube-scheduler
10.6 生成kubectl连接集群的证书
cd ~/TLS/k8s
cat > ~/TLS/k8s/admin-csr.json <<EOF { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
10.7 生成kubeconfig文件
mkdir /root/.kube
KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://192.168.56.201:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials cluster-admin \
--client-certificate=./admin.pem \
--client-key=./admin-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user=cluster-admin \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
10.8 通过kubectl工具查看当前集群组件状态
kubectl get cs
11 部署kubelet(Master)
11.1 创建kubelet配置文件
cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=k8s-master1 \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=192.168.56.201:5000/lizhenliang/pause-amd64:3.0"
EOF
参数说明:
--hostname-override:显示名称,集群中唯一
--network-plugin:启用CNI
--kubeconfig:空路径,会自动生成,后面用于连接apiserver
--bootstrap-kubeconfig:首次启动向apiserver申请证书
--config:配置参数文件
--cert-dir:kubelet证书生成目录
--pod-infra-container-image:管理Pod网络容器的镜像
11.2 拉取镜像
docker pull 192.168.56.201:5000/lizhenliang/pause-amd64:3.0
11.3 创建kubelet参数文件
cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0.0.2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 1000000 maxPods: 110 EOF
11.4 授权kubelet-bootstrap用户允许请求证书
kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap
11.5 生成kubelet初次加入集群引导kubeconfig文件
KUBE_CONFIG="/opt/kubernetes/cfg/bootstrap.kubeconfig"
KUBE_APISERVER="https://192.168.56.201:6443"
TOKEN="c47ffb939f5ca36231d9e3121a252940"
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials "kubelet-bootstrap" \
--token=${TOKEN} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
11.6 systemd管理kubelet
cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
11.7 启动并设置开机启动
systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
11.8 批准kubelet证书申请并加入集群
11.8.1 查看申请
kubectl get csr
11.8.2 批准申请
kubectl certificate approve node-csr-g9jyBhvprZch2yeikec30zUqBiYKkQIf10HrFgXLWiM
11.8.3 查看节点状态
kubectl get node
由于网络插件还没有部署,节点会没有准备就绪
12 部署kube-proxy(Master)
12.1 创建配置文件
cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--proxy-mode=ipvs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF
12.2 配置参数文件
cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
mode: "ipvs"
clientConnection:
kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master1
clusterCIDR: 10.244.0.0/16
EOF
12.3 生成kube-proxy.kubeconfig文件
12.3.1 生成kube-proxy证书
cd ~/TLS/k8s
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
12.3.2 生成kubeconfig文件
KUBE_CONFIG="/opt/kubernetes/cfg/kube-proxy.kubeconfig"
KUBE_APISERVER="https://192.168.56.201:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-proxy \
--client-certificate=./kube-proxy.pem \
--client-key=./kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
12.4 systemd管理kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
12.5 创建加载ipvs模块脚本
vim /etc/sysconfig/modules/ipvs.modules
将下面的代码写入
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs" for mod in $(ls $ipvs_mods_dir |grep -o "^[^.]*");do /sbin/modinfo -F filename $mod &>/dev/null if [ $? -eq 0 ];then /sbin/modprobe $mod fi done
赋予可执行权限并执行
chmod +x /etc/sysconfig/modules/ipvs.modules
cd /etc/sysconfig/modules/
./ipvs.modules
查看加载的ipvs模块
lsmod | grep ip_vs
12.6 启动并设置开机启动
systemctl daemon-reload systemctl start kube-proxy systemctl enable kube-proxy
13 部署calico(Master)
13.1 修改calico.yaml文件中的镜像地址
13.2 部署calico
kubectl apply -f /opt/release/yaml/calico.yaml
13.3 查看pod验证calico是否部署成功
kubectl get pods -n kube-system
直到两个pod的状态都为running说明部署成功
13.4 查看节点
13.5 授权apiserver访问kubelet
应用场景:例如kubectl logs
cd /opt/kubernetes/cfg
cat > apiserver-to-kubelet-rbac.yaml << EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics - pods/log verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes EOF kubectl apply -f apiserver-to-kubelet-rbac.yaml
14 部署Node节点的kubelet和kube-proxy
14.1 拷贝相关文件到Node节点(Master)
scp -r /opt/kubernetes root@192.168.56.202:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.56.202:/usr/lib/systemd/system
scp -r /opt/kubernetes root@192.168.56.203:/opt/
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.56.203:/usr/lib/systemd/system
14.2 删除kubelet证书和kubeconfig文件(Node)
注:这几个文件是证书申请审批后自动生成的,每个Node不同,必须删除
rm -f /opt/kubernetes/cfg/kubelet.kubeconfig rm -f /opt/kubernetes/ssl/kubelet*
14.3 修改配置文件中的主机名(Node)
14.3.1 修改/opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=k8s-node1 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=192.168.56.201:5000/lizhenliang/pause-amd64:3.0"
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=k8s-node2 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=192.168.56.201:5000/lizhenliang/pause-amd64:3.0"
14.3.2 修改/opt/kubernetes/cfg/kube-proxy-config.yml
kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249
mode: "ipvs" clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: k8s-node1 clusterCIDR: 10.244.0.0/16
kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249
mode: "ipvs" clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: k8s-node2 clusterCIDR: 10.244.0.0/16
14.4 创建加载ipvs模块脚本(Node)
vim /etc/sysconfig/modules/ipvs.modules
将下面的代码写入:
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs" for mod in $(ls $ipvs_mods_dir |grep -o "^[^.]*");do /sbin/modinfo -F filename $mod &>/dev/null if [ $? -eq 0 ];then /sbin/modprobe $mod fi done
赋予可执行权限并执行
chmod +x /etc/sysconfig/modules/ipvs.modules cd /etc/sysconfig/modules/ ./ipvs.modules
查看加载的ipvs模块
lsmod | grep ip_vs
启动并设置开机启动
systemctl daemon-reload systemctl start kubelet kube-proxy systemctl enable kubelet kube-proxy
14.5 在Master上批准新Node kubelet证书申请
14.5.1 查看申请
14.5.2 批准申请
kubectl certificate approve node-csr-HRpInSO4Gj_6tm-M9K39jhm3xy-H795AJ2xBa1a_kvo
kubectl certificate approve node-csr-aXpH7szQOnDY81e3SsqTKh83OnEPfEvdCysB9X0LLns
14.6 查看集群状态
kubectl get node
kubectl get pod -n kube-system
15 部署Dashboard
15.1 修改/opt/release/yaml/kubernetes-dashboard.yaml文件中的镜像地址
15.2 部署
kubectl apply -f /opt/release/yaml/kubernetes-dashboard.yaml
15.3 查看部署
kubectl get pods,svc -n kubernetes-dashboard -o wide
15.4 创建service account并绑定默认cluster-admin管理员集群角色
kubectl create serviceaccount dashboard-admin -n kube-system kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
15.5 用上面的token登录Dashboard
地址:https://192.168.56.201:30001/#/login
16 部署CoreDNS
16.1 修改/opt/release/yaml/coredns.yaml文件的镜像地址
image: 192.168.56.201:5000/coredns/coredns:1.2.2
16.2 部署
kubectl apply -f /opt/release/yaml/coredns.yaml
16.3 查看部署
kubectl get pod -n kube-system
16.4 DNS解析测试
kubectl run -it --rm dns-test --image=192.168.56.201:5000/busybox:1.28.4 sh
17 扩展多Master集群
待补充
18 参考
https://www.cnblogs.com/hujinzhong/p/14648961.html