前言
团队成员有DBA、运维、Python开发，由于需要跨部门向公司私有云团队申请虚拟机， 此时我在思考能否在现有已申请的虚拟机之上，再进行更加细粒度的资源隔离和划分，让本团队的成员使用， 也就是在私有云上构建docker容器云，于是研究下Kubernetes，看一下能否找到一些突破点？Kubernetes （库伯耐踢死），省略了Kubernetes中间8个字母，简称K8S；
什么是Kubernets?
kubernets是一种容器编排的解决方案；（The solution to do container orchestrate）
什么是容器技术?（what's the container technology?）
很久以前app上线流程是：
选择1台物理机（First you should choice a phycisal server）然后在这个物理机上部署您的app（second intalling your application in the phycisal server）；
缺点：
部署慢
效率低
很难迁移
虚拟化技术出现之后（virtualization）
优势 The advantage of Virtualization
Resourse pool like CPU Memory（CPU、内存资源可以动态分配）
Easy to extension: Add more VMs or phycial mechine （便于扩展）
缺点 The disadvantage of virtualization
每个虚拟机都需要安装自己的操作系统 Each pyhsical server requires one OS
Container Technology
如果我开发了一个非常成熟的运维平台，我想把源码买给个世界各地的运维团队；
我的要求：1次打包后能运行在世界各地；
打包（build/packet）
运输（ship）
让它运行在世界各地（run）
容器就像装满了各种杂物的集装箱，方便各种运输工具运输到世界各地；
Lowser:熟练使用各种工具并可以解决领导提出的问题
Senior:深谙各种工具的设计理念
First rate: 各种理念模型在大脑中形成，在某时某地 创造出更好的工具；
2.容器技术和虚拟化技术的区别？
虚拟化技术隔离了操作系统层，而容器技术更加彻底它隔离了应用层；
3.容器技术和虚拟化技术的联系？
通常情况下我们线上docker环境是这样的
在虚拟机上搭建docker进行
4.什么是容器编排技术？container orchestration
容器编排就是在docker容器之上再封装1层（Container Orchestration layer 横跨了 docker层 和 全部主机节点）；
这层的功能是把我们的货物（app）智能调度到1个最佳的集装箱上（docker），并集中管理这些集装箱（docker）（动态创建和销毁）
5.编排技术实现工具
Docker Swarm
Mesos
Kubernetes（最火）
Kubernetes诞生于谷歌公司的1个Brog项目
在2015年7月 released 1.0版本之后把它贡献给了 Cloud Native Computing Foundtion(CFCF)基金会
简称 K8S
Kuberbetes的架构
Kubernetes是一款容器编排工具/容器管理（不仅仅支持docker）管理平台；
Kubernetes的设计采用了传统的分布式架构：由于Master节点和Node节点共同组成了Kubernetes集群；
我们可以通过调用其API or CLI(Comand Iine Interface)命令行kubect的方式和Kubernetes的Master节点进行交互；
1.Kubernetes Master节点
Master节点相当于K8S的大脑
Kubectl：控制Kubernetes的命令行工具
API Server：基于REST api的对外管理接口
Scheduler：调度任务（创建1组容器）到指定Node上执行
Controler Manager：控制管理器Controler-manager里面有1个复制控制器(如果客户端要创建3个容器，我会检查node中创建的是不是3个？)
etcd：etcd集群存储Kubernetes集群中所有的数据
2.Node节点组件
node节点就是干活，一般来说我们需要部署1个APP，用户发起创建容器操作之后master节点会智能调度到node节点上
Kubelet：相当于master装在各个Node节点上的1个agent（管理Pod以及容器、镜像、Volum等，实现对节点进行管理）
Kube-porxy：为容器中的web服务，提供网络代理和负载均衡功能，支持Iptables和LVS
Docker Engin：负责节点容器的真正创建、管理
3.Kubernetes工作流程
假设我现在要使用Kubectl命令行创建1组容器出来，大致都会经历那些流程呢？
A.Kubectl把命令提交到API Server
B.Secheduler获取任务，算出最佳的node
C.API Server通知最佳node上的代理（Kublet）
D.Kublet调用docker-engin进行实际的容器创建工作
实验环境准备
本文主要介绍使用Centos7 安装 Kubernetes 1.10.1版本
1.设置VMware Workstation的为NAT模式
2.修改网卡名称并设置静态IP地址
[root@remote network-scripts]# cd /etc/sysconfig/network-scripts/ [root@remote network-scripts]# mv ifcfg-ems33 ifcfg-eth0
GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet net.ifnames=0 biosdevname=0" GRUB_DISABLE_RECOVERY="true"
[root@localhost network-scripts]# grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... Found linux image: /boot/vmlinuz-3.10.0-514.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-514.el7.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-eb3d805e301049b0a680718a9cc3bec0 Found initrd image: /boot/initramfs-0-rescue-eb3d805e301049b0a680718a9cc3bec0.img
TYPE="Ethernet" BOOTPROTO="static" DEFROUTE="yes" PEERDNS="no" PEERROUTES="yes" NAME="eth0" DEVICE="eth0" ONBOOT="yes" IPADDR="192.168.56.11" NETMASK="255.255.255.0" GATEWAY="192.168.56.2" DNS="8.8.8.8"
重启
3.关闭 firewalld&selinux服务
# This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
[root@localhost zhanggen]# setenforce 0 [root@localhost zhanggen]# getenforce Permissive [root@localhost zhanggen]# systemctl stop firewalld
4.系统环境准备
[root@linux-node1 ~]# cd /etc/yum.repos.d/ [root@linux-node1 yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo [root@linux-node1 ~]# yum install -y docker-ce [root@linux-node1 ~]# systemctl start docker
5.安装包
Kubernetes的安装包分为
kubernetes.tar.gz 源码包
kubernetes-server-linux-amd64.tar.gz 服务端包
kubernetes-node-linux-amd64.tar.gz node节点包
kubernetes-client-linux-amd64.tar.gz 客户端工具包
[root@linux-node1 桌面]# mv k8s-v1.10.1-manual /usr/local/src/ [root@linux-node1 桌面]# cd /usr/src/ [root@linux-node1 src]# ls debug kernels [root@linux-node1 src]# cd /usr/local/src/k8s-v1.10.1-manual/ [root@linux-node1 k8s-v1.10.1-manual]# ls k8s-v1.10.1 [root@linux-node1 k8s-v1.10.1-manual]# cd k8s-v1.10.1/ [root@linux-node1 k8s-v1.10.1]# ls cfssl-certinfo_linux-amd64 flannel-v0.10.0-linux-amd64.tar.gz cfssljson_linux-amd64 kubernetes-client-linux-amd64.tar.gz cfssl_linux-amd64 kubernetes-node-linux-amd64.tar.gz cni-plugins-amd64-v0.7.1.tgz kubernetes-server-linux-amd64.tar.gz etcd-v3.2.18-linux-amd64.tar.gz kubernetes.tar.gz [root@linux-node1 k8s-v1.10.1]#
6.Kubernetes部署目录
配置文件、二进制文件、ssl证书、日志
mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}
7.配置Kubernetes的环境变量
# .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin:/opt/kubernetes/bin export PATH
source ~/.bash_profile
Centos7安装 Kubernetes 1.10.1版本
kubernetes项目是由Go写的，所以都是一堆已经编译好的二进制文件，想要安装它们无需编译；
比较麻烦的地方就是Kubernetes的组件间通信是基于SSL协议，所以要为安装每个组件都要生成、分发证书；
下载安装包----->复制到安装目录---------->写配置文件------->生成证书----------> 分发证书--------->启动
集群CA证书的创建和分发
从Kubernetes 1.8.x开始部署Kubernetes都需要使用TLS证书进行通信加密
本环节主要是安装cfssl，生成证书、并把证书存放在 /usr/local/src/ssl目录下，然后分发到其他节点去；
1.安装 cfssl
[root@linux-node1 ~]# cd /usr/local/src [root@linux-node1 src]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 [root@linux-node1 src]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 [root@linux-node1 src]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 [root@linux-node1 src]# chmod +x cfssl* [root@linux-node1 src]# mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo [root@linux-node1 src]# mv cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljson [root@linux-node1 src]# mv cfssl_linux-amd64 /opt/kubernetes/bin/cfssl ---------------同步到其他node节点--------------------------------------------------------------- [root@linux-node1 bin]# scp /opt/kubernetes/bin/cfssl* 192.168.56.12:/opt/kubernetes/bin root@192.168.56.12's password: cfssl-certinfo 100% 6441KB 6.3MB/s 00:00 cfssljson
2.创建证书存放目录
[root@linux-node1 src]# mkdir -p /usr/local/src/ssl [root@linux-node1 src]# cd usr/local/src/ssl bash: cd: usr/local/src/ssl: 没有那个文件或目录 [root@linux-node1 src]# cd /usr/local/src/ssl [root@linux-node1 ssl]# pwd /usr/local/src/ssl
3.编辑证书配置文件
{ "signing": { "default": { "expiry": "8760h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } } }
4.编辑ca-csr配置文件
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
5.生成证书
[root@linux-node1 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca 2019/04/28 11:09:23 [INFO] generating a new CA key and certificate from CSR 2019/04/28 11:09:23 [INFO] generate received request 2019/04/28 11:09:23 [INFO] received CSR 2019/04/28 11:09:23 [INFO] generating key: rsa-2048 2019/04/28 11:09:23 [INFO] encoded CSR 2019/04/28 11:09:23 [INFO] signed certificate with serial number 201551104749556046170924668398043154803671249365 [root@linux-node1 ssl
6.把在/usr/local/ssl目录中生成的证书 cp 到 本机和其他节点的 /opt/kubernetes/ssl
[root@linux-node1 ssl]# cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl [root@linux-node1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.56.12:/opt/kubernetes/ssl root@192.168.56.12's password: ca.csr 100% 1001 1.0KB/s 00:00 ca.pem 100% 1359 1.3KB/s 00:00 ca-key.pem 100% 1679 1.6KB/s 00:00 ca-config.json 100% 283 0.3KB/s 00:00 [root@linux-node1 ssl]#
ETCD集群部署
ETCD集群类似于Hadoop中的zookper分布式协同服务，
也可以是一种分布式的专门为分布式系统设计的K-V存储。
0.准备etcd软件包
wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz [root@linux-node1 src]# tar zxf etcd-v3.2.18-linux-amd64.tar.gz [root@linux-node1 src]# cd etcd-v3.2.18-linux-amd64 [root@linux-node1 etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/ [root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.12:/opt/kubernetes/bin/ [root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.13:/opt/kubernetes/bin/
1.创建 etcd 证书签名配置文件：
etcd集群的通信也需要 CA证书进行认证，所以要使用搭建好的自签名证书给，生成证书，并分发给其他节点；
{ "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.56.11", "192.168.56.12", "192.168.56.13" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
2.生成 etcd 证书和私钥：
[root@linux-node1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes etcd-csr.json | cfssljson -bare etcd 会生成以下证书文件 [root@k8s-master ~]# ls -l etcd* -rw-r--r-- 1 root root 1045 Mar 5 11:27 etcd.csr -rw-r--r-- 1 root root 257 Mar 5 11:25 etcd-csr.json -rw------- 1 root root 1679 Mar 5 11:27 etcd-key.pem -rw-r--r-- 1 root root 1419 Mar 5 11:27 etcd.pem
3.将证书移动到/opt/kubernetes/ssl目录下
[root@k8s-master ~]# cp etcd*.pem /opt/kubernetes/ssl [root@linux-node1 ~]# scp etcd*.pem 192.168.56.12:/opt/kubernetes/ssl [root@linux-node1 ~]# scp etcd*.pem 192.168.56.13:/opt/kubernetes/ssl [root@k8s-master ~]# rm -f etcd.csr etcd-csr.json
4.设置ETCD配置文件并发发给其他节点
[root@linux-node1 ~]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://192.168.56.11:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.56.11:2379,https://127.0.0.1:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.11:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.11:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
5.创建ETCD系统服务
[root@linux-node1 ~]# vim /etc/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target [Service] Type=simple WorkingDirectory=/var/lib/etcd EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf # set GOMAXPROCS to number of processors ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd" Type=notify [Install] WantedBy=multi-user.target
6.重新加载系统服务
[root@linux-node1 ~]# systemctl daemon-reload [root@linux-node1 ~]# systemctl enable etcd # scp /opt/kubernetes/cfg/etcd.conf 192.168.56.12:/opt/kubernetes/cfg/ # scp /etc/systemd/system/etcd.service 192.168.56.12:/etc/systemd/system/ # scp /opt/kubernetes/cfg/etcd.conf 192.168.56.13:/opt/kubernetes/cfg/ # scp /etc/systemd/system/etcd.service 192.168.56.13:/etc/systemd/system/ 在所有节点上创建etcd存储目录并启动etcd [root@linux-node1 ~]# mkdir /var/lib/etcd [root@linux-node1 ~]# systemctl start etcd [root@linux-node1 ~]# systemctl status etcd
7.验证集群
[root@linux-node1 ~]# etcdctl --endpoints=https://192.168.56.11:2379 \ --ca-file=/opt/kubernetes/ssl/ca.pem \ --cert-file=/opt/kubernetes/ssl/etcd.pem \ --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health member 435fb0a8da627a4c is healthy: got healthy result from https://192.168.56.12:2379 member 6566e06d7343e1bb is healthy: got healthy result from https://192.168.56.11:2379 member ce7b884e428b6c8c is healthy: got healthy result from https://192.168.56.13:2379 cluster is healthy
Kubernetes Master节点部署
Kubernetes的Master节点主要包含3个服务：
API Server：Kubernetes组件间的数据交换和通信枢纽，
只有Apiserver才可以操作etcd集群，其他模块只能通过ApiServer间接查询或修改数据；
Scheduler：分配调度Pod（Kubernetes中的逻辑单位，包含容器）到集群的node节点
Controller-manager:有一系列的控制器组成，它通过ApiServer监控整个集群的状态，并确保集群处在预期的工作状态；
部署Kubernetes API服务部署
0.准备软件包
[root@linux-node1 ~]# cd /usr/local/src/kubernetes [root@linux-node1 kubernetes]# cp server/bin/kube-apiserver /opt/kubernetes/bin/ [root@linux-node1 kubernetes]# cp server/bin/kube-controller-manager /opt/kubernetes/bin/ [root@linux-node1 kubernetes]# cp server/bin/kube-scheduler /opt/kubernetes/bin/
1.创建生成CSR的 JSON 配置文件
{ "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.56.11", "10.1.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
2.生成 kubernetes 证书和私钥
生成kubernetes 证书和私钥存放在/opt/kubernetes/ssl/目录下
[root@linux-node1 src]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes [root@linux-node1 src]# cp kubernetes*.pem /opt/kubernetes/ssl/ [root@linux-node1 ~]# scp kubernetes*.pem 192.168.56.12:/opt/kubernetes/ssl/ [root@linux-node1 ~]# scp kubernetes*.pem 192.168.56.13:/opt/kubernetes/ssl/
3.创建 kube-apiserver 使用的客户端 token 文件
在/opt/kubernetes/ssl/bootstrap-token.csv生成token文件
[root@linux-node1 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' ' ad6d5bb607a186796d8861557df0d17f [root@linux-node1 ~]# vim /opt/kubernetes/ssl/ bootstrap-token.csv ad6d5bb607a186796d8861557df0d17f,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
4.创建基础用户名/密码认证配置
在/opt/kubernetes/ssl/basic-auth.csv用户密码认证文件
admin,admin,1 readonly,readonly,2
5.设置Kubernetes API Server的启动项
[root@linux-node1 ~]# vim /usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] ExecStart=/opt/kubernetes/bin/kube-apiserver \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \ --bind-address=192.168.56.11 \ --insecure-bind-address=127.0.0.1 \ --authorization-mode=Node,RBAC \ --runtime-config=rbac.authorization.k8s.io/v1 \ --kubelet-https=true \ --anonymous-auth=false \ --basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \ --enable-bootstrap-token-auth \ --token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \ --service-cluster-ip-range=10.1.0.0/16 \ --service-node-port-range=20000-40000 \ --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \ --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \ --client-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \ --etcd-cafile=/opt/kubernetes/ssl/ca.pem \ --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \ --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \ --etcd-servers=https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/opt/kubernetes/log/api-audit.log \ --event-ttl=1h \ --v=2 \ --logtostderr=false \ --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target
ps:
192.168.56.11:6443端口是kubernetes-api对外的socket（需要认证）
127.0.0.1:8080端口是给Controller-manager、Scheduler节点内部通信使用；（无需认证）
6.启动API Server服务
[root@linux-node1 ~]# systemctl daemon-reload [root@linux-node1 ~]# systemctl enable kube-apiserver [root@linux-node1 ~]# systemctl start kube-apiserver 查看API Server服务状态 [root@linux-node1 ~]# systemctl status kube-apiserver
部署Controller Manager服务
1.设置 kube-controller-manager的启动项
[root@linux-node1 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/opt/kubernetes/bin/kube-controller-manager \ --address=127.0.0.1 \ --master=http://127.0.0.1:8080 \ --allocate-node-cidrs=true \ --service-cluster-ip-range=10.1.0.0/16 \ --cluster-cidr=10.2.0.0/16 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \ --root-ca-file=/opt/kubernetes/ssl/ca.pem \ --leader-elect=true \ --v=2 \ --logtostderr=false \ --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
ps:kube-controller-manager监听在内网127.0.0.1:10252
2.启动Controller Manager服务
[root@linux-node1 ~]# systemctl daemon-reload [root@linux-node1 scripts]# systemctl enable kube-controller-manager [root@linux-node1 scripts]# systemctl start kube-controller-manager
部署Kubernetes-Scheduler
0.设置Kubernetes-Scheduler的启动项
[root@linux-node1 ~]# vim /usr/lib/systemd/system/kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/opt/kubernetes/bin/kube-scheduler \ --address=127.0.0.1 \ --master=http://127.0.0.1:8080 \ --leader-elect=true \ --v=2 \ --logtostderr=false \ --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
2.启动Kubernetes-Scheduler
[root@linux-node1 ~]# systemctl daemon-reload [root@linux-node1 scripts]# systemctl enable kube-scheduler [root@linux-node1 scripts]# systemctl start kube-scheduler [root@linux-node1 scripts]# systemctl status kube-scheduler
部署kubectl 命令行工具
管理K8S集群除了可以使用程序调用 Api-Server，还可以通过kubectl命令行进行调用
1.准备二进制命令包
[root@linux-node1 ~]# cd /usr/local/src/kubernetes/client/bin [root@linux-node1 bin]# cp kubectl /opt/kubernetes/bin/
2.创建 admin 证书签名请求
[root@linux-node1 ~]# cd /usr/local/src/ssl/ [root@linux-node1 ssl]# vim admin-csr.json { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] }
3.生成 admin 证书和私钥
[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes admin-csr.json | cfssljson -bare admin [root@linux-node1 ssl]# ls -l admin* -rw-r--r-- 1 root root 1009 Mar 5 12:29 admin.csr -rw-r--r-- 1 root root 229 Mar 5 12:28 admin-csr.json -rw------- 1 root root 1675 Mar 5 12:29 admin-key.pem -rw-r--r-- 1 root root 1399 Mar 5 12:29 admin.pem [root@linux-node1 src]# mv admin*.pem /opt/kubernetes/ssl/
以下操作都是为了帮你在当前用户加目录下生成1个config文件，该文件在kuberctl 和api通信就使用这个文件进行加密
apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVSTAzZE5wWjExNmVWc0hEdUY2RDZjb29sTWRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9EQXpNRFF3TUZvWERUSTBNRFF5TmpBek1EUXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeG10WHBRZVpxUUxaRnpyQ0dwRFUKUE9JNDlFclZSd3ZaV2toMmVYSzZsMVpCTXZSWGY0QTczbHU5UkdpanErZ3EwRGNsSmMwWHRoSXNJMlVUNFJ4Qwp5QlpGNzIyVmkrZHlDTHNnUHFYSFFTYUVVVDdQeGhIOUt6c0dMbWV6M0YwUUJCbWNXNU00clJ5U2E1VDYxbkpnCm5QWmQvdjVFZ1VEMDI2TldFcWM2aWp0blVvQ1hFdDFteDRhbWE1YTk1OFBQTm5OSXVJUlFSUnp6Z1U0L3NFVGQKSUpPR2l2N043RysrdWU0Z3pLemZPRFJUU0FDK1FUVnB6c0RNN05sY29ITWpnOGNSL0ZxYWVjQXJoZ05xckxPbQpMamoxUjZDR0d3a2FnUG40SWhGQVkxamJQVXBHSnRQSkN4RlUzY0RQeXQrVEZwblFOVmxCYmMrWE5HTGo0QkFUCkx3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVM1ZqU2gyVWp5akFicTgranB5dTM2OThua1lJd0h3WURWUjBqQkJnd0ZvQVUzVmpTaDJVagp5akFicTgranB5dTM2OThua1lJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMaHdXWExjMmMyREprRW41Y2VrClhxQlJqbXNIVVhKYzhiQWN5aXBrL0Y5OXBKaDRoYjJCMXcvd011aGdTZStRWFptSFVhZUdWbFZJUGhuTkMxM00KallnajZGenM4RGJXbVQ4TWViVHJtVXVjMSttMnQ1clpSdENDeGZocHdhSmJHcURPU29vYUpBVWdvdWdVS00vQQppU2t2N3J6OC9BYjdramFNY2ZFRzJsbmEzdkNXRXhUTW9PL2V3RkR3THZnWUgxMXcybU9ZSjRSV1gxaUFlNVlxCnAzclRscVdQNmM3U1RsNkpyem1EOVUwWkpkMzQ0SmNxcDFORkNpUzJYcGZIdFMySkhxRVVVN1Y4Zi81RzRkeWIKcmRQYVRNNGJsZzlaWUMvcGtlbUZoRjIvRm50Y3hrQVhzWXR4eUVzOVdHeHZyK0JvRnJqeXBpdzlhMDNLeVlDTwo3NGM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://192.168.56.11:6443 name: kubernetes contexts: - context: cluster: kubernetes user: admin name: kubernetes current-context: kubernetes kind: Config preferences: {} users: - name: admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzVENDQXNXZ0F3SUJBZ0lVRlNwVzZHcCt6YmtmaloyeDZFMFpGS01mTkpnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9URTBNall3TUZvWERUSXdNRFF5T0RFME1qWXdNRm93YXpFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RnpBVgpCZ05WQkFvVERuTjVjM1JsYlRwdFlYTjBaWEp6TVE4d0RRWURWUVFMRXdaVGVYTjBaVzB4RGpBTUJnTlZCQU1UCkJXRmtiV2x1TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6bzFtSEtrdW54OWMKTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXR2lCSHNxQUZPUEhJU3VtTnc1RWE4S29aNDMzRQp6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2tocUU3WjlqTWxYTGpHVEhtM0txWWIrNXE0SWdZCkUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cGhmZ1dWVytiUWdnLzhrdmVxM0JqbXZhTmJTZTIKRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sQWFFZm9tM0N4bHlPR285YzJMamhqcG9MdUpXTgp5U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGg5TXVUS01FOGhQT21TbzdSaHRPR0hPM2NDK2JkCng3NUFQcE9tRFFJREFRQUJvMzh3ZlRBT0JnTlZIUThCQWY4RUJBTUNCYUF3SFFZRFZSMGxCQll3RkFZSUt3WUIKQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWURWUjBPQkJZRUZPdDFlNXo3OS9rdwo5QVU4dEhhcm9TOWs4bnpCTUI4R0ExVWRJd1FZTUJhQUZOMVkwb2RsSThvd0c2dlBvNmNydCt2Zko1R0NNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFDdGpJYWZaeWlYWHhyU2lzM3Z4TWk2MGJDeXRTekkvWEhzcFhyWUR3bnUKRlBxNkJDSzQrbXpvUDREcVhWVTR1NG5PZzhncDFXa0QyVk12YWphUjlUOFRDbUt0NEhFSWR4ai9uSTU0ekJZUAp2eTRrRXVrbUhyWXFPTlRBT1pKMzAzQWxFQUNZRFFQNTFTU3poaThDak15N3NWdmdlUkdCOVo1WVV1TitCaFFpCmxsWm5nVXBHVlBNT0dhMUVYdlY5T01GajVWQ2taa2l1dk5NVzFOWFUvTzNXQVNoWUhhK21NNzJJTHVWdDhJOWMKRlFqTk9rUjNFamU1eHpvOTNEVFMwZGZtbFRXc3dPK0grM3pEcVFKZU9KSzNyYnROeEM2Z0ZLcUl3dHZsOUF0awpkQ2N0YzcveXdTT0NObVc4WDc1azRvaXhNY3RvamVFc3FjMWw5SnBMR1dwMwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBem8xbUhLa3VueDljTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXCkdpQkhzcUFGT1BISVN1bU53NUVhOEtvWjQzM0V6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2sKaHFFN1o5ak1sWExqR1RIbTNLcVliKzVxNElnWUUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cApoZmdXVlcrYlFnZy84a3ZlcTNCam12YU5iU2UyRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sCkFhRWZvbTNDeGx5T0dvOWMyTGpoanBvTHVKV055U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGgKOU11VEtNRThoUE9tU283Umh0T0dITzNjQytiZHg3NUFQcE9tRFFJREFRQUJBb0lCQUJxbllucm1WZzdRbEN2NgpxQVhBQW9xck1hcUN1UmZhSXVuZ0xFRVpYcmZSZzNtMmdjV2pUcjA5S3c5YkcvYVd4dVZxcnloSk93Z0JMY2t0Cjd6aStlSEVBTEQ3UzExTjhhVmYyTU10SG9xN0xOMTlsK25PcEVLcG83cFdHZXNuQWg4TUgvSjNORFZ1bUZEMUIKV05RQzNJdEhMeml0WTZpZnVIZFQzZG9STGt4aU9TTUZaQ3l0cG9hRkJtc0I4dVBtSXcvajduTHRpWklGRko0eApaZ1pEaXRWTnM5c25xMFNlWVpKZVJvQm5sS0w1QmhpL2VJZDNnd3dONlplZ0ptbnFlVytydXZqUVEzV2pyNXVPCnNncFVWbDZVOW1aN1ZNYXVVWEdZVHA2Qk1UaFFKWlVUQVl5bzZGMzE3blZaRndWVWJ3WnZEU0VXM0I0VzVNS0cKZHRsY3ZpRUNnWUVBNElNOFhxNHozdTlTeVBYZFZEdER4MUNkbkZVak5TMHZ1R0E4ZGpFUGFNbng1STVpTnpFeQpwTjhrUnpaWEVHSW55b3pFVzlzOStHSjJwb3VxZnRJV3pOYUxmZUFRbzBzU3lSTmtuclNsLzlLaERFdWN6alNECkM0dURsb3poTGRmb3VWZi9BdE02SldkWDVZN3ByQXdLcFhkSjJCNU5vQXRxeGxqbnpjeU9zSWtDZ1lFQTY0VlQKMHNHc2tabkVwUUx1ZVIxSkZYSm5qSTA4R01rVFhlU1U0KzJrZzRWdkJTRmxBNzBiOUtaL2l4L3BOcUhveFVYbQpSSTdwUE1GWUJJcU93MFZxdjViN3daQzJnTXgzUWczUUpaRURZb2hZblpFZ3M5UUZyb1NmT2M1dm1seForSG02CmNLN2pkalI5OFFBcHdDd1JVR08vc2lyeFV0UGRnZW16TTFZSUFHVUNnWUFBaFUxbWl0RGorM29kclRST05iVDYKaVYxVU4zNVZhVDFyR0E0TDJDRkpCTzdpc05IWmZ1dTNKaTFYWFBEbXdOT0d6THpIMmNKVENTZHRTM1doeGFyMwozcWVFS3pqZXFCWHJFWGh5UmNqOHh1aEl0d1F1RmtFWGpjTklYaHRIbC9DYVBYSUI5NnR5MnNLQmJjdHM4cm96Cm1Bczd6Ull4QU5YR2ovNDVvL2ZRd1FLQmdRQ0p0Q1MwZjBTVXhPRXkwYW40Nm1TR3c2TkRqSGhzelhRalc3aXEKSTVJaXkrdURobWozYktSaHdNK2wybnlTMHN1MFBCQk1XWHFKYVVvN0xZQVhNQWtnQi9rZXphdkhWc1VMdStQTgpjeUlWaEQ2N3NkVDdENlpheVhRSGFtbmFTTkRaOU9KTGJCWFdVUTZwMER5WS9hc0o0Nmg3Vlp4UG4weE4zd1JGCnFiRjMrUUtCZ1FEYXhTRzgvU1RsNmkyeitIY0F2c25lem1iVXJ6QmFKc3BJM3NOMG9tQ1IvZVJXcTM5dVk5VTEKODk2VXVBQ3RDTnk4VS9EWkY3RnNsakpOK3FkYjYrVXNlN0FZRTVEejlxTDl1SFI2am80dHZoNzFTUkFreE9nRgptSCtDMnJkbDNzdStPU2ZHTlQ4dzFId2dBaDFIdExJRHVsNTFvekduTGFOWWNzNjZ0M0lQbHc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo= [root@linux-node1 ~]# ^C [root@linux-node1 ~]# pwd /root [root@linux-node1 ~]# cat /root/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVSTAzZE5wWjExNmVWc0hEdUY2RDZjb29sTWRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9EQXpNRFF3TUZvWERUSTBNRFF5TmpBek1EUXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeG10WHBRZVpxUUxaRnpyQ0dwRFUKUE9JNDlFclZSd3ZaV2toMmVYSzZsMVpCTXZSWGY0QTczbHU5UkdpanErZ3EwRGNsSmMwWHRoSXNJMlVUNFJ4Qwp5QlpGNzIyVmkrZHlDTHNnUHFYSFFTYUVVVDdQeGhIOUt6c0dMbWV6M0YwUUJCbWNXNU00clJ5U2E1VDYxbkpnCm5QWmQvdjVFZ1VEMDI2TldFcWM2aWp0blVvQ1hFdDFteDRhbWE1YTk1OFBQTm5OSXVJUlFSUnp6Z1U0L3NFVGQKSUpPR2l2N043RysrdWU0Z3pLemZPRFJUU0FDK1FUVnB6c0RNN05sY29ITWpnOGNSL0ZxYWVjQXJoZ05xckxPbQpMamoxUjZDR0d3a2FnUG40SWhGQVkxamJQVXBHSnRQSkN4RlUzY0RQeXQrVEZwblFOVmxCYmMrWE5HTGo0QkFUCkx3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVM1ZqU2gyVWp5akFicTgranB5dTM2OThua1lJd0h3WURWUjBqQkJnd0ZvQVUzVmpTaDJVagp5akFicTgranB5dTM2OThua1lJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMaHdXWExjMmMyREprRW41Y2VrClhxQlJqbXNIVVhKYzhiQWN5aXBrL0Y5OXBKaDRoYjJCMXcvd011aGdTZStRWFptSFVhZUdWbFZJUGhuTkMxM00KallnajZGenM4RGJXbVQ4TWViVHJtVXVjMSttMnQ1clpSdENDeGZocHdhSmJHcURPU29vYUpBVWdvdWdVS00vQQppU2t2N3J6OC9BYjdramFNY2ZFRzJsbmEzdkNXRXhUTW9PL2V3RkR3THZnWUgxMXcybU9ZSjRSV1gxaUFlNVlxCnAzclRscVdQNmM3U1RsNkpyem1EOVUwWkpkMzQ0SmNxcDFORkNpUzJYcGZIdFMySkhxRVVVN1Y4Zi81RzRkeWIKcmRQYVRNNGJsZzlaWUMvcGtlbUZoRjIvRm50Y3hrQVhzWXR4eUVzOVdHeHZyK0JvRnJqeXBpdzlhMDNLeVlDTwo3NGM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://192.168.56.11:6443 name: kubernetes contexts: - context: cluster: kubernetes user: admin name: kubernetes current-context: kubernetes kind: Config preferences: {} users: - name: admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzVENDQXNXZ0F3SUJBZ0lVRlNwVzZHcCt6YmtmaloyeDZFMFpGS01mTkpnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9URTBNall3TUZvWERUSXdNRFF5T0RFME1qWXdNRm93YXpFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RnpBVgpCZ05WQkFvVERuTjVjM1JsYlRwdFlYTjBaWEp6TVE4d0RRWURWUVFMRXdaVGVYTjBaVzB4RGpBTUJnTlZCQU1UCkJXRmtiV2x1TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6bzFtSEtrdW54OWMKTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXR2lCSHNxQUZPUEhJU3VtTnc1RWE4S29aNDMzRQp6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2tocUU3WjlqTWxYTGpHVEhtM0txWWIrNXE0SWdZCkUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cGhmZ1dWVytiUWdnLzhrdmVxM0JqbXZhTmJTZTIKRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sQWFFZm9tM0N4bHlPR285YzJMamhqcG9MdUpXTgp5U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGg5TXVUS01FOGhQT21TbzdSaHRPR0hPM2NDK2JkCng3NUFQcE9tRFFJREFRQUJvMzh3ZlRBT0JnTlZIUThCQWY4RUJBTUNCYUF3SFFZRFZSMGxCQll3RkFZSUt3WUIKQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWURWUjBPQkJZRUZPdDFlNXo3OS9rdwo5QVU4dEhhcm9TOWs4bnpCTUI4R0ExVWRJd1FZTUJhQUZOMVkwb2RsSThvd0c2dlBvNmNydCt2Zko1R0NNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFDdGpJYWZaeWlYWHhyU2lzM3Z4TWk2MGJDeXRTekkvWEhzcFhyWUR3bnUKRlBxNkJDSzQrbXpvUDREcVhWVTR1NG5PZzhncDFXa0QyVk12YWphUjlUOFRDbUt0NEhFSWR4ai9uSTU0ekJZUAp2eTRrRXVrbUhyWXFPTlRBT1pKMzAzQWxFQUNZRFFQNTFTU3poaThDak15N3NWdmdlUkdCOVo1WVV1TitCaFFpCmxsWm5nVXBHVlBNT0dhMUVYdlY5T01GajVWQ2taa2l1dk5NVzFOWFUvTzNXQVNoWUhhK21NNzJJTHVWdDhJOWMKRlFqTk9rUjNFamU1eHpvOTNEVFMwZGZtbFRXc3dPK0grM3pEcVFKZU9KSzNyYnROeEM2Z0ZLcUl3dHZsOUF0awpkQ2N0YzcveXdTT0NObVc4WDc1azRvaXhNY3RvamVFc3FjMWw5SnBMR1dwMwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBem8xbUhLa3VueDljTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXCkdpQkhzcUFGT1BISVN1bU53NUVhOEtvWjQzM0V6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2sKaHFFN1o5ak1sWExqR1RIbTNLcVliKzVxNElnWUUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cApoZmdXVlcrYlFnZy84a3ZlcTNCam12YU5iU2UyRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sCkFhRWZvbTNDeGx5T0dvOWMyTGpoanBvTHVKV055U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGgKOU11VEtNRThoUE9tU283Umh0T0dITzNjQytiZHg3NUFQcE9tRFFJREFRQUJBb0lCQUJxbllucm1WZzdRbEN2NgpxQVhBQW9xck1hcUN1UmZhSXVuZ0xFRVpYcmZSZzNtMmdjV2pUcjA5S3c5YkcvYVd4dVZxcnloSk93Z0JMY2t0Cjd6aStlSEVBTEQ3UzExTjhhVmYyTU10SG9xN0xOMTlsK25PcEVLcG83cFdHZXNuQWg4TUgvSjNORFZ1bUZEMUIKV05RQzNJdEhMeml0WTZpZnVIZFQzZG9STGt4aU9TTUZaQ3l0cG9hRkJtc0I4dVBtSXcvajduTHRpWklGRko0eApaZ1pEaXRWTnM5c25xMFNlWVpKZVJvQm5sS0w1QmhpL2VJZDNnd3dONlplZ0ptbnFlVytydXZqUVEzV2pyNXVPCnNncFVWbDZVOW1aN1ZNYXVVWEdZVHA2Qk1UaFFKWlVUQVl5bzZGMzE3blZaRndWVWJ3WnZEU0VXM0I0VzVNS0cKZHRsY3ZpRUNnWUVBNElNOFhxNHozdTlTeVBYZFZEdER4MUNkbkZVak5TMHZ1R0E4ZGpFUGFNbng1STVpTnpFeQpwTjhrUnpaWEVHSW55b3pFVzlzOStHSjJwb3VxZnRJV3pOYUxmZUFRbzBzU3lSTmtuclNsLzlLaERFdWN6alNECkM0dURsb3poTGRmb3VWZi9BdE02SldkWDVZN3ByQXdLcFhkSjJCNU5vQXRxeGxqbnpjeU9zSWtDZ1lFQTY0VlQKMHNHc2tabkVwUUx1ZVIxSkZYSm5qSTA4R01rVFhlU1U0KzJrZzRWdkJTRmxBNzBiOUtaL2l4L3BOcUhveFVYbQpSSTdwUE1GWUJJcU93MFZxdjViN3daQzJnTXgzUWczUUpaRURZb2hZblpFZ3M5UUZyb1NmT2M1dm1seForSG02CmNLN2pkalI5OFFBcHdDd1JVR08vc2lyeFV0UGRnZW16TTFZSUFHVUNnWUFBaFUxbWl0RGorM29kclRST05iVDYKaVYxVU4zNVZhVDFyR0E0TDJDRkpCTzdpc05IWmZ1dTNKaTFYWFBEbXdOT0d6THpIMmNKVENTZHRTM1doeGFyMwozcWVFS3pqZXFCWHJFWGh5UmNqOHh1aEl0d1F1RmtFWGpjTklYaHRIbC9DYVBYSUI5NnR5MnNLQmJjdHM4cm96Cm1Bczd6Ull4QU5YR2ovNDVvL2ZRd1FLQmdRQ0p0Q1MwZjBTVXhPRXkwYW40Nm1TR3c2TkRqSGhzelhRalc3aXEKSTVJaXkrdURobWozYktSaHdNK2wybnlTMHN1MFBCQk1XWHFKYVVvN0xZQVhNQWtnQi9rZXphdkhWc1VMdStQTgpjeUlWaEQ2N3NkVDdENlpheVhRSGFtbmFTTkRaOU9KTGJCWFdVUTZwMER5WS9hc0o0Nmg3Vlp4UG4weE4zd1JGCnFiRjMrUUtCZ1FEYXhTRzgvU1RsNmkyeitIY0F2c25lem1iVXJ6QmFKc3BJM3NOMG9tQ1IvZVJXcTM5dVk5VTEKODk2VXVBQ3RDTnk4VS9EWkY3RnNsakpOK3FkYjYrVXNlN0FZRTVEejlxTDl1SFI2am80dHZoNzFTUkFreE9nRgptSCtDMnJkbDNzdStPU2ZHTlQ4dzFId2dBaDFIdExJRHVsNTFvekduTGFOWWNzNjZ0M0lQbHc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
[root@linux-node1 src]# kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.56.11:6443 Cluster "kubernetes" set.
[root@linux-node1 src]# kubectl config set-credentials admin \ --client-certificate=/opt/kubernetes/ssl/admin.pem \ --embed-certs=true \ --client-key=/opt/kubernetes/ssl/admin-key.pem User "admin" set.
[root@linux-node1 src]# kubectl config set-context kubernetes \ --cluster=kubernetes \ --user=admin Context "kubernetes" created.
[root@linux-node1 src]# kubectl config use-context kubernetes Switched to context "kubernetes".
[root@linux-node1 ~]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Unhealthy Get https://192.168.56.12:2379/health: remote error: tls: bad certificate etcd-2 Unhealthy Get https://192.168.56.13:2379/health: remote error: tls: bad certificate etcd-0 Healthy {"health": "true"} [root@linux-node1 ~]#
Node节点部署
现在Master节点的组件已经部署完毕，可是光杆司令是无法工作的，所以需要部署node节点；
Node节点部署主要包括：kubelet(master的agent)、Kubernetes Proxy（提供负载均衡服务）
1.二进制包准备 将软件包从linux-node1复制到linux-node2中去。
把kubelet、kube-proxy复制到所有node节点
[root@linux-node1 ~]# cd /usr/local/src/kubernetes/server/bin/ [root@linux-node1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin/ [root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.12:/opt/kubernetes/bin/ [root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.13:/opt/kubernetes/bin/
2.创建角色绑定
[root@linux-node1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap clusterrolebinding "kubelet-bootstrap" created
切换到/usr/local/src/ssl，一些操作是在 /opt/kubernetes/ssl目录下生成1个bootstrap.kubeconfig 文件，
当node节点的启动kubelet服务时，node节点会携带 bootstrap.kubeconfig 发送post请求进行crs请求验证
所以以后增加Node节点，都需要把该文件复制过去；
3.创建 kubelet bootstrapping kubeconfig 文件 设置集群参数
[root@linux-node1 ~]# kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.56.11:6443 \ --kubeconfig=bootstrap.kubeconfig Cluster "kubernetes" set.
[root@linux-node1 ~]# kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig Context "default" created.
[root@linux-node1 ~]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig Switched to context "default". [root@linux-node1 kubernetes]# cp bootstrap.kubeconfig /opt/kubernetes/cfg [root@linux-node1 kubernetes]# scp bootstrap.kubeconfig 192.168.56.12:/opt/kubernetes/cfg [root@linux-node1 kubernetes]# scp bootstrap.kubeconfig 192.168.56.13:/opt/kubernetes/cfg
4.部署kubelet 1.设置所有Node支持CNI
[root@linux-node2 ~]# mkdir -p /etc/cni/net.d [root@linux-node2 ~]# vim /etc/cni/net.d/10-default.conf { "name": "flannel", "type": "flannel", "delegate": { "bridge": "docker0", "isDefaultGateway": true, "mtu": 1400 } }
5.创建kubelet目录
[root@linux-node2 ~]# mkdir /var/lib/kubelet
6.node节点创建kubelet服务配置
[root@k8s-node2 ~]# vim /usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/opt/kubernetes/bin/kubelet \ --address=192.168.56.12 \ --hostname-override=192.168.56.12 \ --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \ --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ --cert-dir=/opt/kubernetes/ssl \ --network-plugin=cni \ --cni-conf-dir=/etc/cni/net.d \ --cni-bin-dir=/opt/kubernetes/bin/cni \ --cluster-dns=10.1.0.2 \ --cluster-domain=cluster.local. \ --hairpin-mode hairpin-veth \ --allow-privileged=true \ --fail-swap-on=false \ --logtostderr=true \ --v=2 \ --logtostderr=false \ --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5
[root@k8s-node3 ~]# vim /usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/opt/kubernetes/bin/kubelet \ --address=192.168.56.13 \ --hostname-override=192.168.56.13 \ --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \ --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ --cert-dir=/opt/kubernetes/ssl \ --network-plugin=cni \ --cni-conf-dir=/etc/cni/net.d \ --cni-bin-dir=/opt/kubernetes/bin/cni \ --cluster-dns=10.1.0.2 \ --cluster-domain=cluster.local. \ --hairpin-mode hairpin-veth \ --allow-privileged=true \ --fail-swap-on=false \ --logtostderr=true \ --v=2 \ --logtostderr=false \ --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5
7.在node上4.启动Kubelet
[root@linux-node2 ~]# systemctl daemon-reload [root@linux-node2 ~]# systemctl enable kubelet [root@linux-node2 ~]# systemctl start kubelet
[root@linux-node2 kubernetes]# systemctl status kubelet
8.在master节点上查看csr请求
注意是在linux-node1（master节点）上执行。
[root@linux-node1 ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-0_w5F1FM_la_SeGiu3Y5xELRpYUjjT2icIFk9gO9KOU 1m kubelet-bootstrap Pending
可以看到node节点post发送csr请求处在 Pending状态，下面我们批准kubelet 的 TLS 证书请求
[root@linux-node1 ssl]# kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve [root@linux-node1 ssl]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-YgMzRMD3GrQsJdEUwAKMDp6Zazi-NU_h28DmOlohevc 5m kubelet-bootstrap Approved,Issued node-csr-mu9Ptdy93UCSoGLrk--AVrG1DxKImzgTv5O3kYL1TQE 6m kubelet-bootstrap Approved,Issued
csr请求处在 由于Pending状态进入Approved状态
与此同时在node节点的/opt/kubernetes/ssl目录下回生成1个名为kubelet-client.crt 的证书
8.查看节点的状态
[root@linux-node1 ssl]# kubectl get node NAME STATUS ROLES AGE VERSION 192.168.56.11 NotReady <none> 8h v1.10.1 192.168.56.12 Ready <none> 4m v1.10.1 192.168.56.13 Ready <none> 4m v1.10.1 [root@linux-node1 ssl]#
部署Kubernetes Proxy
1.配置kube-proxy使用LVS
[root@linux-node2 ~]# yum install -y ipvsadm ipset conntrack
2.创建 kube-proxy 证书请求
[root@linux-node1 ~]# cd /usr/local/src/ssl/ [root@linux-node1 ~]# vim kube-proxy-csr.json { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
3.生成证书
[root@linux-node1~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
4.分发证书到所有Node节点
[root@linux-node1 ssl]# cp kube-proxy*.pem /opt/kubernetes/ssl/ [root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.12:/opt/kubernetes/ssl/ [root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.12:/opt/kubernetes/ssl/
5.创建kube-proxy配置文件
[root@linux-node2 ~]# kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=https://192.168.56.11:6443 \ --kubeconfig=kube-proxy.kubeconfig Cluster "kubernetes" set. [root@linux-node2 ~]# kubectl config set-credentials kube-proxy \ --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \ --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig User "kube-proxy" set. [root@linux-node2 ~]# kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig Context "default" created. [root@linux-node2 ~]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig Switched to context "default".
6.分发kubeconfig配置文件
[root@linux-node1 ssl]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg/ [root@linux-node1 ~]# scp kube-proxy.kubeconfig 192.168.56.12:/opt/kubernetes/cfg/ [root@linux-node1 ~]# scp kube-proxy.kubeconfig 192.168.56.13:/opt/kubernetes/cfg/
7.创建kube-proxy服务配置
[root@linux-node2 bin]# mkdir /var/lib/kube-proxy [root@k8s-node2 ~]# vim /usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/opt/kubernetes/bin/kube-proxy \ --bind-address=192.168.56.12 \ --hostname-override=192.168.56.12 \ --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \ --masquerade-all \ --feature-gates=SupportIPVSProxyMode=true \ --proxy-mode=ipvs \ --ipvs-min-sync-period=5s \ --ipvs-sync-period=5s \ --ipvs-scheduler=rr \ --logtostderr=true \ --v=2 \ --logtostderr=false \ --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target 8.启动Kubernetes Proxy [root@linux-node2 ~]# systemctl daemon-reload [root@linux-node2 ~]# systemctl enable kube-proxy [root@linux-node2 ~]# systemctl start kube-proxy
9.查看服务状态 查看kube-proxy服务状态
[root@linux-node2 cfg]# systemctl status kube-proxy ● kube-proxy.service - Kubernetes Kube-Proxy Server Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2019-04-30 08:50:57 CST; 13min ago Docs: https://github.com/GoogleCloudPlatform/kubernetes Main PID: 28022 (kube-proxy) Memory: 38.1M CGroup: /system.slice/kube-proxy.service ‣ 28022 /opt/kubernetes/bin/kube-proxy --bind-address=192.168.56.12 --hos... Apr 30 08:50:57 linux-node2.example.com systemd[1]: kube-proxy.service holdoff time.... Apr 30 08:50:57 linux-node2.example.com systemd[1]: Started Kubernetes Kube-Proxy S.... Apr 30 08:50:57 linux-node2.example.com systemd[1]: Starting Kubernetes Kube-Proxy .... Hint: Some lines were ellipsized, use -l to show in full. [root@linux-node2 cfg]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.0.1:443 rr persistent 10800 -> 192.168.56.11:6443 Masq 1 0 0 [root@linux-node2 cfg]#
Flannel网络部署
即使我们部署好了所有Kubernetes集群的所有组件，但是它们没有网络是无法通行的！
Poid：Poid是Kubernetes中1个逻辑的概念，Kubernets直接管理的不是容器而是Poid；
每个Poild里面又包含了1个或者N个容器；
如果你需要的容器之间需要网络互通，你就可以在1个poind里面跑多个容器，它们之间通过host来通行；
每个Poid都要1个IP地址：用来替代docker0网桥，实现Kubernetes中不同容器间的通信
Replication Ctronller（复杂控制）简称RC
RC:是保证Kubernetes中Pod高可用，通过监控运行中的Pod来保证集群中指定Pod的副本数量；
RS(replica set)Kubernetes感觉RC的功能有点单一就对RC升级新增了新功能，RS是RC的升级版
Deployment是1个比RS应用模式更广的API对象：它在保证Pod副本数量的前提下，还可以创建、更新、滚动升级服务；
RC、RS和Deplyment只是保证了支持服务的Pod数量，但没有解决如何高效访问这些服务的问题；
在K8S集群中客户端要访问的服务就是1个server对象，每个servers对象对应集群内有效的虚拟IP
Flunel：就是通过对docker网络的封装实现 Kubernetes中Pod之间的通信，它可以每个Node都可以分配到不同的IP地址段
1.为Flannel生成证书
在/usr/local/src/ssl
{ "CN": "flanneld", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ]
2.生成证书
[root@linux-node1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
3.分发证书
[root@linux-node1 ~]# cp flanneld*.pem /opt/kubernetes/ssl/ [root@linux-node1 ~]# scp flanneld*.pem 192.168.56.12:/opt/kubernetes/ssl/ [root@linux-node1 ~]# scp flanneld*.pem 192.168.56.13:/opt/kubernetes/ssl/
4.下载Flannel软件包
[root@linux-node1 ~]# cd /usr/local/src # wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz [root@linux-node1 src]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz [root@linux-node1 src]# cp flanneld mk-docker-opts.sh /opt/kubernetes/bin/ 复制到linux-node2节点 [root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.12:/opt/kubernetes/bin/ [root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.13:/opt/kubernetes/bin/ 复制对应脚本到/opt/kubernetes/bin目录下。 [root@linux-node1 ~]# cd /usr/local/src/kubernetes/cluster/centos/node/bin/ [root@linux-node1 bin]# cp remove-docker0.sh /opt/kubernetes/bin/ [root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.12:/opt/kubernetes/bin/ [root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.13:/opt/kubernetes/bin/
5.配置Flannel
[root@linux-node1 ~]# vim /opt/kubernetes/cfg/flannel FLANNEL_ETCD="-etcd-endpoints=https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379" FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network" FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem" FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem" FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem" 复制配置到其它节点上 [root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.12:/opt/kubernetes/cfg/ [root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.13:/opt/kubernetes/cfg/
6.设置Flannel系统服务
[root@linux-node1 ~]# vim /usr/lib/systemd/system/flannel.service [Unit] Description=Flanneld overlay address etcd agent After=network.target Before=docker.service [Service] EnvironmentFile=-/opt/kubernetes/cfg/flannel ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE} ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker Type=notify [Install] WantedBy=multi-user.target RequiredBy=docker.service
7.复制系统服务脚本到其它节点上
scp /usr/lib/systemd/system/flannel.service 192.168.56.12:/usr/lib/systemd/system/ scp /usr/lib/systemd/system/flannel.service 192.168.56.13:/usr/lib/systemd/system/
8.Flannel 和CNI集成
https://github.com/containernetworking/plugins/releases wget https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz [root@linux-node1 ~]# mkdir /opt/kubernetes/bin/cni [root@linux-node1 src]# tar zxf cni-plugins-amd64-v0.7.1.tgz -C /opt/kubernetes/bin/cni # scp -r /opt/kubernetes/bin/cni/* 192.168.56.12:/opt/kubernetes/bin/cni/ # scp -r /opt/kubernetes/bin/cni/* 192.168.56.13:/opt/kubernetes/bin/cni/
创建Etcd的key
/opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem \ --no-sync -C https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379 \ mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null 2>&1
9.启动flannel
[root@linux-node1 ~]# systemctl daemon-reload [root@linux-node1 ~]# systemctl enable flannel [root@linux-node1 ~]# chmod +x /opt/kubernetes/bin/* [root@linux-node1 ~]# systemctl start flannel[root@linux-node1 ~]# systemctl status flannel
10.修改docker配置文件，让Docker使用Flannel
[root@linux-node1 ~]# vim /usr/lib/systemd/system/docker.service [Unit] #在Unit下面修改After和增加Requires After=network-online.target firewalld.service flannel.service Wants=network-online.target Requires=flannel.service [Service] #增加EnvironmentFile=-/run/flannel/docker Type=notify EnvironmentFile=-/run/flannel/docker ExecStart=/usr/bin/dockerd $DOCKER_OPTS
11.docker配置文件同步
scp /usr/lib/systemd/system/docker.service 192.168.56.12:/usr/lib/systemd/system/ scp /usr/lib/systemd/system/docker.service 192.168.56.13:/usr/lib/systemd/system/
12.重启Docker
[root@linux-node1 ~]# systemctl daemon-reload
[root@linux-node1 ~]# systemctl restart docker
[root@linux-node1 ssl]# kubectl run net-test --image=alpine --replicas=2 sleep 360000 deployment.apps "net-test" created [root@linux-node1 ssl]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE net-test-5767cb94df-mkj4p 0/1 ContainerCreating 0 10s <none> 192.168.56.12 net-test-5767cb94df-q8vbt 0/1 ContainerCreating 0 10s <none> 192.168.56.13 [root@linux-node1 ssl]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE net-test-5767cb94df-mkj4p 0/1 ContainerCreating 0 27s <none> 192.168.56.12 net-test-5767cb94df-q8vbt 0/1 ContainerCreating 0 27s <none> 192.168.56.13 [root@linux-node1 ssl]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE net-test-5767cb94df-mkj4p 1/1 Running 0 50s 10.2.101.2 192.168.56.12 net-test-5767cb94df-q8vbt 1/1 Running 0 50s 10.2.34.2 192.168.56.13 [root@linux-node1 ssl]# ping 10.2.101.2 PING 10.2.101.2 (10.2.101.2) 56(84) bytes of data. 64 bytes from 10.2.101.2: icmp_seq=1 ttl=63 time=1.88 ms 64 bytes from 10.2.101.2: icmp_seq=2 ttl=63 time=0.400 ms ^X64 bytes from 10.2.101.2: icmp_seq=3 ttl=63 time=0.348 ms 64 bytes from 10.2.101.2: icmp_seq=4 ttl=63 time=0.406 ms ^C --- 10.2.101.2 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3000ms rtt min/avg/max/mdev = 0.348/0.760/1.889
如果最后容器处在 running状态说明你的K8S已经搭建完毕；
