Kubernetes04-安装Kubernetes，多master（使用二进制）
1、系统要求
1.1、系统要求
- Kubernetes系统由一组可执行程序组成，用户可以通过GitHub上的Kubernetes项目页下载编译好的二进制包，或者下载源代码并编译后进行安装。
- 安装Kubernetes对软件和硬件的系统要求如表所示：
- Kubernetes需要容器运行时（Container Runtime Interface，CRI）的支持，目前官方支持的容器运行时包括：Docker、Containerd、CRI-O和frakti。
1.2、部署环境
- 软件版本
- Kubernetes：1.20.14
- etcd：etcd-v3.4.13-linux-amd64.tar.gz
- docker：docker-ce-19.03.10
- 系统环境
- 10.1.1.11：CentOSLinuxrelease7.7.1908(Core)
- 10.1.1.12：CentOSLinuxrelease7.7.1908(Core)
- 10.1.1.13：CentOSLinuxrelease7.7.1908(Core)
- 10.1.1.14：CentOSLinuxrelease7.7.1908(Core)
- 10.1.1.15：CentOSLinuxrelease7.7.1908(Core)
- 各个虚拟机的要求
- 核数最少两个
- 内存最少2G
1.3、安装前的准备（所有节点）
1、IP规划
- 10.1.1.11：k8s-master1
- 10.1.1.12：k8s-master2
- 10.1.1.13：k8s-master3
- 10.1.1.14：k8s-node1
- 10.1.1.15：k8s-node2
- 10.1.1.250：vip
- 172.21.0.0/16：PodIP
- 172.22.0.0/16：ServiceIP
2、配置机器主机名
#在10.1.1.11上执行如下： hostnamectl set-hostname k8s-master1 && bash #在10.1.1.12上执行如下： hostnamectl set-hostname k8s-master2 && bash #在10.1.1.13上执行如下： hostnamectl set-hostname k8s-master3 && bash #在10.1.1.14上执行如下： hostnamectl set-hostname k8s-node1 && bash #在10.1.1.15上执行如下： hostnamectl set-hostname k8s-node2 && bash
2、配置主机hosts文件，相互之间通过主机名互相访问
- 修改每台机器的/etc/hosts文件：
cat >> /etc/hosts << EOF 10.1.1.11 k8s-master1 10.1.1.12 k8s-master2 10.1.1.13 k8s-master3 10.1.1.14 k8s-node1 10.1.1.15 k8s-node2 EOF
3、关闭交换分区swap，提升性能
- Swap是交换分区，如果机器内存不够，会使用swap分区，但是swap分区的性能较低，k8s设计的时候为了能提升性能，默认是不允许使用交换分区的。Kubeadm初始化的时候会检测swap是否关闭，如果没关闭，那就初始化失败。如果不想要关闭交换分区，初始化k8s的时候可以指定--ignore-preflight-errors=Swap来解决。
#查看swap状态 free -h #关闭swap，临时有效 swapoff -a #注释掉swap，重启机器，永久有效 vim /etc/fstab #/dev/mapper/centos-swap swap swap defaults 0 0
4、修改机器内核参数
#加载br_netfilter模块 modprobe br_netfilter echo "modprobe br_netfilter" >> /etc/profile #修改内核参数 cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.ipv6.conf.all.disable_ipv6 = 1 EOF #使修改的参数生效 sysctl -p /etc/sysctl.d/k8s.conf
- 问题1：sysctl是做什么的？
- 在运行时配置内核参数。-p从指定的文件加载系统参数，如不指定即从/etc/sysctl.conf中加载
- 问题2：为什么要执行modprobe br_netfilter？
- sysctl -p /etc/sysctl.d/k8s.conf出现报错：
- sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
- sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
- sysctl -p /etc/sysctl.d/k8s.conf出现报错：
- 问题3：为什么开启net.bridge.bridge-nf-call-iptables内核参数？
- 在centos下安装docker，执行docker info出现如下警告：
- WARNING: bridge-nf-call-iptables is disabled
- WARNING: bridge-nf-call-ip6tables is disabled
- 在centos下安装docker，执行docker info出现如下警告：
- 问题4：为什么要开启net.ipv4.ip_forward = 1参数？
- 出于安全考虑，Linux系统默认是禁止数据包转发的。所谓转发即当主机拥有多于一块的网卡时，其中一块收到数据包，根据数据包的目的ip地址将数据包发往本机另一块网卡，该网卡根据路由表继续发送数据包。这通常是路由器所要实现的功能。
- 要让Linux系统具有路由转发功能，需要配置一个Linux的内核参数net.ipv4.ip_forward。这个参数指定了Linux系统当前对路由转发功能的支持情况；其值为0时表示
5、关闭firewalld防火墙
- 需要注意的是，CentOS Linux 7默认启动了防火墙服务（firewalld），而Kubernetes的Master与工作Node之间会有大量的网络通信，安全的做法是在防火墙上配置各组件需要相互通信的端口号。
- 在安全的内部网络环境中可以关闭防火墙服务：
#查看防护墙状态 systemctl status firewalld.service #关闭防火墙 systemctl stop firewalld.service systemctl disable firewalld.service
6、关闭selinux
- 禁用SELinux，让容器可以读取主机文件系统：
#查看selinux的状态 getenforce #临时有效 setenforce 0 #永久有效。改系统文件/etc/sysconfig/selinux，然后重启机器 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
7、配置时间同步
#安装ntpdate yum install ntpdate -y #同步时间 ntpdate ntp1.aliyun.com #把时间同步做成计划任务 crontab -e * */1 * * * /usr/sbin/ntpdate ntp1.aliyun.com #重启crond服务 service crond restart
8、开启ipvs
cat > /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in \${ipvs_modules}; do
/sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1
if [ 0 -eq 0 ]; then
/sbin/modprobe \${kernel_module}
fi
done
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
#查看是否开启ip_vs
lsmod | grep ip_vs
- 问题1：ipvs是什么？
- ipvs (IP Virtual Server) 实现了传输层负载均衡，也就是我们常说的4层LAN交换，作为 Linux 内核的一部分。ipvs运行在主机上，在真实服务器集群前充当负载均衡器。ipvs可以将基于TCP和UDP的服务请求转发到真实服务器上，并使真实服务器的服务在单个 IP 地址上显示为虚拟服务。
- 问题2：ipvs和iptable对比分析
- kube-proxy支持 iptables 和 ipvs 两种模式， 在kubernetes v1.8 中引入了 ipvs 模式，在 v1.9 中处于 beta 阶段，在 v1.11 中已经正式可用了。iptables 模式在 v1.1 中就添加支持了，从 v1.2 版本开始 iptables 就是 kube-proxy 默认的操作模式，ipvs 和 iptables 都是基于netfilter的，但是ipvs采用的是hash表，因此当service数量达到一定规模时，hash查表的速度优势就会显现出来，从而提高service的服务性能。
- ipvs模式和iptables模式之间有哪些差异呢？
- 1、ipvs 为大型集群提供了更好的可扩展性和性能
- 2、ipvs 支持比 iptables 更复杂的复制均衡算法（最小负载、最少连接、加权等等）
- 3、ipvs 支持服务器健康检查和连接重试等功能
9、安装基础包（非必须）
yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools \
nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl \
curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf \
automake zlib-devel python-devel epel-release openssh-server socat ipvsadm \
conntrack ntpdate telnet ipvsadm
10、安装iptables（非必须）
- 如果用firewalld不习惯，可以安装iptables
#安装iptables yum install iptables-services -y #禁用iptables service iptables stop && systemctl disable iptables #清空防火墙规则 iptables -F
11、查看内核版本、cpu核数和内存大小
#内核至少要3.10 uname -a #cpu核数最少要2个 cat /proc/cpuinfo | grep "processor" | wc -l #内存大小至少要2G free -h #centos版本最好在7.5以上 cat /etc/redhat-release
2、安装docker（所有节点）
- 二进制安装或yum安装都可以，为了方便，这里使用yum安装docker。
1、配置docker的yum源
- http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
- https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo
wget http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -P /etc/yum.repos.d/
2、安装docker
yum install docker-ce-19.03.10 docker-ce-cli-19.03.10 containerd.io -y
3、镜像加速和文件驱动器
- 修改docker文件驱动为systemd，默认为cgroupfs，kubelet默认使用systemd，两者必须一致才可以。
- 不修改，在进行初始化master时会有警告信息（[Docker SystemdCheck]: detected cgroupfs" as the Docker cgroup dr iver. The r ecommended dr fiver is" systemd"）
mkdir -p /etc/docker/
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": [
"https://rsbud4vc.mirror.aliyuncs.com",
"https://registry.docker-cn.com",
"https://docker.mirrors.ustc.edu.cn",
"https://dockerhub.azk8s.cn",
"http://hub-mirror.c.163.com",
"http://qtid6917.mirror.aliyuncs.com",
"https://rncxm540.mirror.aliyuncs.com"],
"exec-opts": [
"native.cgroupdriver=systemd"]
}
EOF
#查看配置是否生效（启动docker后）
docker info
5、启动docker
systemctl enable docker.service systemctl start docker.service systemctl status docker.service
- 修改网络参数，将下列文件的值改为1
#要先启动docker，才会有前两个文件 cat /proc/sys/net/bridge/bridge-nf-call-ip6tables cat /proc/sys/net/bridge/bridge-nf-call-iptables cat /proc/sys/net/ipv4/ip_forward cat /proc/sys/net/ipv6/conf/default/disable_ipv6 cat /proc/sys/net/ipv6/conf/all/disable_ipv6
3、搭建私有CA
- 从kubernetes 1.8版本开始，kubernetes各组件需要使用SSL进行通信。每一个kubernetes集群都需要独立的CA证书。
- CA证书有三种：easyrsa、openssl、cfssl。这里使用cfssl证书，也是目前使用最多的，相对来说配置简单。
#创建CA相关的目录 mkdir -p /usr/local/k8s/ca
3.1、搭建CA的工具
#下载地址，本文使用的是v1.5.0 https://github.com/cloudflare/cfssl #将制作CA的工具放到ca目录中 mv ./cfssl_1.5.0_linux_amd64 /usr/local/k8s/ca/cfssl mv ./cfssl-certinfo_1.5.0_linux_amd64 /usr/local/k8s/ca/cfssl-certinfo mv ./cfssljson_1.5.0_linux_amd64 /usr/local/k8s/ca/cfssljson #给工具添加可执行权限 chmod +x /usr/local/k8s/ca/cfssl* ]# ls -lh /usr/local/k8s/ca/ -rwxr-xr-x 1 root root 15M 3月 2 22:01 cfssl -rwxr-xr-x 1 root root 12M 3月 2 22:01 cfssl-certinfo -rwxr-xr-x 1 root root 9.3M 3月 2 22:01 cfssljson
3.2、搭建CA
1、生成CA的自签证书
#创建用来生成CA证书签名请求（CSR）的JSON 配置文件
cat > /usr/local/k8s/ca/ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
#生成证书和密钥
cd /usr/local/k8s/ca/
./cfssl gencert -initca ca-csr.json | ./cfssljson -bare ca
#查看生成的证书（ca.pem）和密钥（ca-key.pem）
]# ls -lh /usr/local/k8s/ca/ca*
-rw-r--r-- 1 root root 1001 3月 2 22:15 /usr/local/k8s/ca/ca.csr
-rw------- 1 root root 1.7K 3月 2 22:15 /usr/local/k8s/ca/ca-key.pem
-rw-r--r-- 1 root root 1.3K 3月 2 22:15 /usr/local/k8s/ca/ca.pem
2、创建用来生成 CA文件的 JSON 配置文件
cat > /usr/local/k8s/ca/ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "8760h"
}
}
}
}
EOF
4、部署etcd集群
- etcd服务作为Kubernetes集群的主数据库，在安装Kubernetes各服务之前需要首先安装和启动。（etcd是独立于kubernetes的）
- 所有持久化的状态信息以KV的形式存储在ETCD中。类似zookeeper，提供分布式协调服务。之所以说kubenetes各个组件是无状态的，就是因为其中把数据都存放在ETCD中。
- 在下面三台机器上搭建etcd集群
- 10.1.1.11
- 10.1.1.12
- 10.1.1.13
#创建ectd相关的目录
mkdir -p /usr/local/k8s/etcd/{bin,conf,ssl}
4.1、生成etcd的证书
1、创建etcd证书的签名请求
- 注意，hosts字段中是etcd集群的各个节点ip，必须全部包含，不能少。可以多预留几个，方便以后扩容。
cat > /usr/local/k8s/ca/etcd-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"10.1.1.11",
"10.1.1.12",
"10.1.1.13",
"10.1.1.250"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
2、生成etcd证书和私钥
cd /usr/local/k8s/ca/
./cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=kubernetes etcd-csr.json | ./cfssljson -bare etcd
3、发放etcd的证书和密钥
#将etcd的证书和密钥到etcd/ssl目录中
mv /usr/local/k8s/ca/etcd*.{pem,csr} /usr/local/k8s/etcd/ssl
4.2、安装etcd
1、下载etcd
GitHub官网：https://github.com/etcd-io/etcd
2、解压etcd
#解压tecd tar zfx ./etcd-v3.4.13-linux-amd64.tar.gz #移动etcd的两个可执行文件 cp ./etcd-v3.4.13-linux-amd64/etcdctl /usr/local/bin/ mv ./etcd-v3.4.13-linux-amd64/etcd* /usr/local/k8s/etcd/bin/
3、配置etcd的配置文件
- 2379端口用于外部通信，2380用于内部通信
cat > /usr/local/k8s/etcd/conf/etcd.conf << EOF #[member] ETCD_NAME="etcd-node1" ETCD_LISTEN_PEER_URLS="https://10.1.1.11:2380" ETCD_LISTEN_CLIENT_URLS="https://10.1.1.11:2379,https://127.0.0.1:2379" ETCD_DATA_DIR="/data/etcd" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.1.1.11:2380" ETCD_ADVERTISE_CLIENT_URLS="https://10.1.1.11:2379" ETCD_INITIAL_CLUSTER="etcd-node1=https://10.1.1.11:2380,etcd-node2=https://10.1.1.12:2380,etcd-node3=https://10.1.1.13:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" EOF
- 配置文件中各指令释义：
- ETCD_NAME：节点名称，集群中唯一
- ETCD_DATA_DIR：数据目录
- ETCD_LISTEN_PEER_URLS：集群通信监听地址
- ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
- ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
- ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
- ETCD_INITIAL_CLUSTER：集群节点地址
- ETCD_INITIAL_CLUSTER_TOKEN：集群Token
- ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
- 创建ETCD_DATA_DIR指定的etcd数据存储目录
mkdir -p /data/etc
4、创建etcd系统服务
- 指令说明：https://etcd.io/docs/v3.4/op-guide/configuration/
cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] EnvironmentFile=-/usr/local/k8s/etcd/conf/etcd.conf ExecStart=/usr/local/k8s/etcd/bin/etcd \\ --cert-file=/usr/local/k8s/etcd/ssl/etcd.pem \\ --key-file=/usr/local/k8s/etcd/ssl/etcd-key.pem \\ --trusted-ca-file=/usr/local/k8s/ca/ca.pem \\ --peer-cert-file=/usr/local/k8s/etcd/ssl/etcd.pem \\ --peer-key-file=/usr/local/k8s/etcd/ssl/etcd-key.pem \\ --peer-trusted-ca-file=/usr/local/k8s/ca/ca.pem \\ --peer-client-cert-auth \\ --client-cert-auth Restart=on-failure RestartSec=5 LimitNOFILE=65536 Type=notify [Install] WantedBy=multi-user.target EOF
4.3、将配置同步到其他etcd节点
- 以将配置同步到10.1.1.12机器为例
#同步k8s目录 scp -r /usr/local/k8s root@10.1.1.12:/usr/local/ #同步系统服务文件 scp /usr/lib/systemd/system/etcd.service root@10.1.1.12:/usr/lib/systemd/system/etcd.service #修改etcd.conf文件（在10.1.1.12上执行，修改ETCD_NAME和对应的IP） vim /usr/local/k8s/etcd/conf/etcd.conf #创建/data/etc（在10.1.1.12上执行） mkdir -p /data/etc
4.4、启动etcd
1、启动etcd
#重读系统服务配置，使其生效 systemctl daemon-reload #启动etcd服务 systemctl enable etcd.service systemctl start etcd.service systemctl status etcd.service
2、查看etcd的各个节点的状态
- 注意，如果日志中出现如下告警，说明你的请求中没带证书，或证书路径错误，再或者你的证书有问题（如果启动时出现的）。
- etcd[3065]: {"level":"warn",......,"error":"remote error: tls: bad certificate"}
]# etcdctl --cacert=/usr/local/k8s/ca/ca.pem --cert=/usr/local/k8s/etcd/ssl/etcd.pem \
--key=/usr/local/k8s/etcd/ssl/etcd-key.pem --endpoints="https://10.1.1.11:2379,https://10.1.1.12:2379,https://10.1.1.13:2379" \
endpoint health --write-out=table
+------------------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+------------------------+--------+-------------+-------+
| https://10.1.1.11:2379 | true | 21.006099ms | |
| https://10.1.1.12:2379 | true | 23.821027ms | |
| https://10.1.1.13:2379 | true | 25.212676ms | |
+------------------------+--------+-------------+-------+
]# etcdctl --cacert=/usr/local/k8s/ca/ca.pem --cert=/usr/local/k8s/etcd/ssl/etcd.pem \
--key=/usr/local/k8s/etcd/ssl/etcd-key.pem --endpoints="https://10.1.1.11:2379,https://10.1.1.12:2379,https://10.1.1.13:2379" \
endpoint status --write-out=table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.1.1.11:2379 | f5859395d8fc536b | 3.4.14 | 25 kB | true | false | 6 | 9 | 9 | |
| https://10.1.1.12:2379 | d3d05727a4b463da | 3.4.14 | 37 kB | false | false | 6 | 9 | 9 | |
| https://10.1.1.13:2379 | 281bdcce6f4633c7 | 3.4.14 | 20 kB | false | false | 6 | 9 | 9 | |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
5、部署Kubernetes集群的master和node
5.1、下载Kubernetes二进制文件
- 从Kubernetes发布官网https://github.com/kubernetes/kubernetes/releases找到对应的版本号，单击CHANGELOG，找到已编译好的二进制文件的下载页面。
- 在压缩包kubernetes.tar.gz内包含了Kubernetes的服务程序文件、文档和示例；
- 在压缩包kubernetes-src.tar.gz内则包含了全部源代码。
- 也可以直接下载Server Binaries中的kubernetes-server-linux-amd64.tar.gz文件，其中包含了Kubernetes需要运行的全部服务程序文件。
- kubernetes-server-linux-arm64.tar.gz中主要的服务程序文件列表：
- 解压kubernetes-server-linux-arm64.tar.gz
tar zfx kubernetes-server-linux-amd64.tar.gz
5.2、Master和Node需要部署的组件
- Master需要部署kubeapiserver、kube-controller-manager、kube-scheduler服务进程。
- Node需要部署kubelet、kube-proxy服务进程。
6、部署kubeapiserver
- API Server提供集群管理的REST API接口，包括认证授权、数据校验以及集群状态变更等。
- 只有API Server才能直接操作etcd。
- 其他模块通过API Server查询或修改数据。
- API Server提供其他模块之间的数据交互和通信枢纽。
#创建相关目录
mkdir -p /usr/local/k8s/kubeapiserver/{bin,conf,ssl}
#将可执行文件移到bin目录下
mv ./kubernetes/server/bin/kube-apiserver /usr/local/k8s/kubeapiserver/bin
6.1、生成kubeapiserver的证书
1、创建kubeapiserver证书的签名请求
- 如果hosts字段不为空，则需要指定授权使用该证书的IP或域名列表。由于该证书后续被kubernetes master集群使用，所以需要将master节点的IP都填上，同时还需要填写service网络的首个IP(一般是kube-apiserver指定的service-cluster-ip-range网段的第一个IP，如172.22.0.1)。
cat > /usr/local/k8s/ca/apiserver-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"10.1.1.11",
"10.1.1.12",
"10.1.1.13",
"10.1.1.250",
"172.22.0.1",
"127.0.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
2、生成kubeapiserver证书和私钥
cd /usr/local/k8s/ca/ ./cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \ -profile=kubernetes apiserver-csr.json | ./cfssljson -bare apiserver
3、发放kubeapiserver的证书和密钥
#将kubernetes的证书和密钥到kubeapiserver/ssl目录中
mv /usr/local/k8s/ca/apiserver*.{pem,csr} /usr/local/k8s/kubeapiserver/ssl
6.2、配置kubeapiserver
1、创建kubeapiserver使用的客户端token文件
- token文件的格式
token，用户名，UID，用户组
- 创建token文件
- 注意，网上有不一样的（kubelet-bootstrap,10001,"system:node-bootstrapper）。如果你和我一样是小白的话，请不要使用，否者node将不能正常加入k8s集群
#生成token值 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' #创建token文件 cat > /usr/local/k8s/kubeapiserver/conf/token.csv << EOF ae805e2221dff87cac96c2345bf490e5,kubelet-bootstrap,10001,"system:node-bootstrapper" EOF
2、配置kubeapiserver的配置文件
- 指令说明：https://v1-20.docs.kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/
cat > /usr/local/k8s/kubeapiserver/conf/kubeapiserver.conf << EOF KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,ResourceQuota,LimitRanger,ServiceAccount,NodeRestriction,DefaultStorageClass \\ --anonymous-auth=false \\ --bind-address=10.1.1.11 \\ --secure-port=6443 \\ --advertise-address=10.1.1.11 \\ --authorization-mode=RBAC,Node \\ --runtime-config=api/all=true \\ --enable-bootstrap-token-auth \\ --service-cluster-ip-range=172.22.0.0/16 \\ --token-auth-file=/usr/local/k8s/kubeapiserver/conf/token.csv \\ --service-node-port-range=1-65355 \\ --tls-cert-file=/usr/local/k8s/kubeapiserver/ssl/apiserver.pem \\ --tls-private-key-file=/usr/local/k8s/kubeapiserver/ssl/apiserver-key.pem \\ --client-ca-file=/usr/local/k8s/ca/ca.pem \\ --kubelet-client-certificate=/usr/local/k8s/kubeapiserver/ssl/apiserver.pem \\ --kubelet-client-key=/usr/local/k8s/kubeapiserver/ssl/apiserver-key.pem \\ --service-account-key-file=/usr/local/k8s/ca/ca-key.pem \\ --service-account-signing-key-file=/usr/local/k8s/ca/ca-key.pem \\ --service-account-issuer=https://kubernetes.default.svc.cluster.local \\ --etcd-cafile=/usr/local/k8s/ca/ca.pem \\ --etcd-certfile=/usr/local/k8s/etcd/ssl/etcd.pem \\ --etcd-keyfile=/usr/local/k8s/etcd/ssl/etcd-key.pem \\ --etcd-servers=https://10.1.1.11:2379,https://10.1.1.12:2379,https://10.1.1.13:2379 \\ --allow-privileged=true \\ --apiserver-count=3 \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-path=/data/logs/kubernetes/k8s-audit.log \\ --event-ttl=1h \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/data/logs/kubernetes \\ --v=2 \\ --requestheader-client-ca-file=/usr/local/k8s/ca/ca.pem \\ --proxy-client-cert-file=/usr/local/k8s/kubeapiserver/ssl/apiserver.pem \\ --proxy-client-key-file=/usr/local/k8s/kubeapiserver/ssl/apiserver-key.pem \\ --requestheader-allowed-names=kubernetes \\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User \\ --enable-aggregator-routing" EOF
- 配置文件中各指令释义：
- --logtostderr：启用日志
- --v：日志等级
- --log-dir：日志目录
- --etcd-servers：etcd集群地址
- --bind-address：监听地址
- --secure-port：https安全端口
- --advertise-address：集群通告地址
- --allow-privileged：启用授权
- --service-cluster-ip-range：Service虚拟IP地址段
- --enable-admission-plugins：准入控制模块
- --authorization-mode：认证授权，启用RBAC授权和节点自管理
- --enable-bootstrap-token-auth：启用TLS bootstrap机制
- --token-auth-file：bootstrap token文件
- --service-node-port-range：Service nodeport类型默认分配端口范围
- --kubelet-client-xxx：kubeapiserver访问kubelet客户端证书
- --tls-xxx-file：apiserver https证书1.20版本必须加的参数：--service-account-issuer，--service-account-signing-key-file
- --etcd-xxxfile：连接Etcd集群证书
- --audit-log-xxx：审计日志
- 启动聚合层相关配置：--requestheader-client-ca-file，--proxy-client-cert-file，--proxy-client-key-file，--requestheader-allowed-names，--requestheader-extra-headers-prefix，--requestheader-group-headers，--requestheader-username-headers，--enable-aggregator-routing
- 创建日志目录
mkdir -p /data/logs/kubernetes
3、配置kubeapiserver的系统服务文件
cat > /usr/lib/systemd/system/kube-apiserver.service << EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=etcd.service Wants=etcd.service [Service] EnvironmentFile=-/usr/local/k8s/kubeapiserver/conf/kubeapiserver.conf ExecStart=/usr/local/k8s/kubeapiserver/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
6.3、将配置同步到其他master节点
- 以将配置同步到10.1.1.12机器为例
#同步kubeapiserver目录 scp -r /usr/local/k8s/kubeapiserver/ root@10.1.1.12:/usr/local/k8s/ #同步系统服务文件 scp /usr/lib/systemd/system/kube-apiserver.service root@10.1.1.12:/usr/lib/systemd/system/kube-apiserver.service #修改kubeapiserver.conf文件（在10.1.1.12上执行，修改对应的IP） vim /usr/local/k8s/kubeapiserver/conf/kubeapiserver.conf #创建目录（在10.1.1.12上执行） mkdir -p /data/logs/kubernetes
6.4、启动kubeapiserver
1、启动kubeapiserver服务
#重读系统服务配置，使其生效 systemctl daemon-reload #启动etcd服务 systemctl enable kube-apiserver.service systemctl start kube-apiserver.service systemctl status kube-apiserver.service
2、查看机器状态
- 返回的401，这个是正常的，因为还没认证。
]# curl --insecure https://10.1.1.11:6443/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}
7、部署kubectl
- kubectl命令使用手册：https://kubernetes.io/zh/docs/reference/kubectl/cheatsheet/
- Kubectl是kubernetes集群的客户端工具，操作kubernetes的资源，如增删改查等。
- Kubectl操作资源的时候，需要一个文件/etc/kubernetes/admin.conf，kubectl会根据这个文件的配置，去访问kubernetes资源。/etc/kubernetes/admin.con文件记录了要访问的k8s集群和要用到的证书。
- (1)可以将/etc/kubernetes/admin.conf设置为一个环境变量KUBECONFIG。在使用kubectl的时候，就会自动加载KUBECONFIG来操作要管理的哪个kubernetes集群的资源了。
- export KUBECONFIG =/etc/kubernetes/admin.conf
- (2)也可以将/etc/kubernetes/admin.conf放到${home}/.kube/config中。在使用kubectl的时候，就会加载${home}/.kube/config文件来操作要管理的哪个kubernetes集群的资源了。
- (3)如果设置了KUBECONFIG，那就会找到KUBECONFIG去操作kubernetes。如果没有KUBECONFIG变量，就会使用${home}/.kube/config文件决定管理哪个kubernetes集群的资源。
#创建相关目录
mkdir -p /usr/local/k8s/kubectl/{bin,conf,ssl}
#将可执行文件移到bin目录下
cp ./kubernetes/server/bin/kubectl /usr/bin
mv ./kubernetes/server/bin/kubectl /usr/local/k8s/kubectl/bin
7.1、生成kubectl连接集群的kubeconfig文件
1、创建kubectlr证书的签名请求
cat > /usr/local/k8s/ca/admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
2、生成kubectlr证书和私钥
cd /usr/local/k8s/ca/
./cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes \
admin-csr.json | ./cfssljson -bare admin
3、创建kubeconfig配置文件
mkdir /root/.kube
cd /usr/local/k8s/ca/
KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://10.1.1.11:6443"
#设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
#设置客户端认证参数
kubectl config set-credentials cluster-admin \
--client-certificate=./admin.pem \
--client-key=./admin-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
#设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=cluster-admin \
--kubeconfig=${KUBE_CONFIG}
#设置当前上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
4、通过kubectl工具查看当前集群组件状态
- 还没有安装controller-manager和scheduler，因此其状态是Unhealthy。这是正常的，先不用管。
]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused
scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
7.2、 授权kubelet-bootstrap用户允许请求证书
- 允许node节点加入k8s集群。
- 注意，下面命令与token关联。让kubelet可以加入集群。
kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap
8、部署kube-controller-manager
- kube-controller-manager由一系列的控制器组成，它通过kubeapiserver监控整个集群的状态，并确保集群处于预期的工作状态。
#创建相关目录
mkdir -p /usr/local/k8s/kube-controller-manager/{bin,conf,ssl}
#将可执行文件移到bin目录下
mv ./kubernetes/server/bin/kube-controller-manager /usr/local/k8s/kube-controller-manager/bin
8.1、生成kube-controller-manager的证书
1、创建kube-controller-manager证书的签名请求
cat > /usr/local/k8s/ca/kube-controller-manager-csr.json << EOF
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"10.1.1.11",
"10.1.1.12",
"10.1.1.13",
"10.1.1.250"
],
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:kube-controller-manager",
"OU": "System"
}
]
}
EOF
- hosts列表包含所有kube-controller-manager节点IP；
- CN为system:kube-controller-manager。
- O为system:kube-controller-manager，kubernetes内置的ClusterRoleBindingssystem:kube-controller-manager赋予kube-controller-manager工作所需的权限
2、生成kube-controller-manager证书和私钥
cd /usr/local/k8s/ca/
./cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes \
kube-controller-manager-csr.json | ./cfssljson -bare kube-controller-manager
3、发放kube-controller-manager的证书和密钥
mv /usr/local/k8s/ca/kube-controller-manager*.{pem,csr} /usr/local/k8s/kube-controller-manager/ssl
8.2、配置kube-controller-manager
1、创建kube-controller-manager的kubeconfig
- 生成kubeconfig文件（以下是shell命令，直接在终端执行）
cd /usr/local/k8s/ca
KUBE_CONFIG="/usr/local/k8s/kube-controller-manager/conf/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://10.1.1.11:6443"
#设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
#设置客户端认证参数
kubectl config set-credentials kube-controller-manager \
--client-certificate=/usr/local/k8s/kube-controller-manager/ssl/kube-controller-manager.pem \
--client-key=/usr/local/k8s/kube-controller-manager/ssl/kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
#设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-controller-manager \
--kubeconfig=${KUBE_CONFIG}
#设置当前上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
2、配置kube-controller-manager的配置文件
- 注意：Service、Pod和本机所在的网络，三者的IP网段不要重合。Pod的网段要尽量大一些。
cat > /usr/local/k8s/kube-controller-manager/conf/kube-controller-manager.conf << EOF KUBE_CONTROLLER_MANAGER_OPTS="--bind-address=127.0.0.1 \\ --kubeconfig=/usr/local/k8s/kube-controller-manager/conf/kube-controller-manager.kubeconfig \\ --authentication-kubeconfig=/usr/local/k8s/kube-controller-manager/conf/kube-controller-manager.kubeconfig \\ --authorization-kubeconfig=/usr/local/k8s/kube-controller-manager/conf/kube-controller-manager.kubeconfig \\ --service-cluster-ip-range=172.22.0.0/16 \\ --cluster-name=kubernetes \\ --cluster-signing-cert-file=/usr/local/k8s/ca/ca.pem \\ --cluster-signing-key-file=/usr/local/k8s/ca/ca-key.pem \\ --allocate-node-cidrs=true \\ --cluster-cidr=172.21.0.0/16 \\ --experimental-cluster-signing-duration=87600h0m0s \\ --root-ca-file=/usr/local/k8s/ca/ca.pem \\ --service-account-private-key-file=/usr/local/k8s/ca/ca-key.pem \\ --leader-elect=true \\ --feature-gates=RotateKubeletServerCertificate=true \\ --controllers=*,bootstrapsigner,tokencleaner \\ --horizontal-pod-autoscaler-use-rest-clients=true \\ --horizontal-pod-autoscaler-sync-period=10s \\ --tls-cert-file=/usr/local/k8s/kube-controller-manager/ssl/kube-controller-manager.pem \\ --tls-private-key-file=/usr/local/k8s/kube-controller-manager/ssl/kube-controller-manager-key.pem \\ --use-service-account-credentials=true \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/data/logs/kubernetes \\ --v=2" EOF
- --kubeconfig：连接kubeapiserver配置文件
- --leader-elect：当该组件启动多个时，自动选举（HA）
- --cluster-signing-cert-file/--cluster-signing-key-file：自动为kubelet颁发证书的CA，与kubeapiserver保持一致
- --cluster-cidr：集群中Pod的IP范围
- --service-cluster-ip-range：集群中Service的IP范围
3、配置kube-controller-manager的系统服务文件
cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/usr/local/k8s/kube-controller-manager/conf/kube-controller-manager.conf ExecStart=/usr/local/k8s/kube-controller-manager/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
8.3、将配置同步到其他master节点
- 以将配置同步到10.1.1.12机器为例
#同步kubeapiserver目录 scp -r /usr/local/k8s/kube-controller-manager root@10.1.1.12:/usr/local/k8s/ #同步系统服务文件 scp /usr/lib/systemd/system/kube-controller-manager.service root@10.1.1.12:/usr/lib/systemd/system/kube-controller-manager.service
8.4、启动kube-controller-manager
#重读系统服务配置，使其生效 systemctl daemon-reload #启动kube-controller-manager systemctl enable kube-controller-manager.service systemctl start kube-controller-manager.service systemctl status kube-controller-manager.service
9、部署kube-scheduler
- scheduler负责分配调度Pod到集群内的node节点。
- 监听kube-apiserver，查询还未分配的Node的Pod。
- 根据调度策略为这些Pod分配节点。
#创建相关目录
mkdir -p /usr/local/k8s/kube-scheduler/{bin,conf,ssl}
#将可执行文件移到bin目录下
mv ./kubernetes/server/bin/kube-scheduler /usr/local/k8s/kube-scheduler/bin
9.1、生成kube-scheduler的证书
1、创建kube-scheduler证书的签名请求
cat > /usr/local/k8s/ca/kube-scheduler-csr.json << EOF
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"10.1.1.11",
"10.1.1.12",
"10.1.1.13",
"10.1.1.250"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:kube-scheduler",
"OU": "System"
}
]
}
EOF
2、生成kube-scheduler证书和私钥
cd /usr/local/k8s/ca/
./cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes \
kube-scheduler-csr.json | ./cfssljson -bare kube-scheduler
3、发放kube-scheduler的证书和密钥
mv /usr/local/k8s/ca/kube-scheduler*.{pem,csr} /usr/local/k8s/kube-scheduler/ssl
9.2、配置kube-scheduler
1、创建kube-scheduler的kubeconfig
- 生成kubeconfig文件（以下是shell命令，直接在终端执行）
cd /usr/local/k8s/ca
KUBE_CONFIG="/usr/local/k8s/kube-scheduler/conf/kube-scheduler.kubeconfig "
KUBE_APISERVER="https://10.1.1.11:6443"
#设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
#设置客户端认证参数
kubectl config set-credentials kube-scheduler \
--client-certificate=/usr/local/k8s/kube-scheduler/ssl/kube-scheduler.pem \
--client-key=/usr/local/k8s/kube-scheduler/ssl/kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
#设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-scheduler \
--kubeconfig=${KUBE_CONFIG}
#设置当前上下文
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
2、配置kube-scheduler的配置文件
cat > /usr/local/k8s/kube-scheduler/conf/kube-scheduler.conf << EOF KUBE_SCHEDULER_OPTS="--bind-address=127.0.0.1 \\ --kubeconfig=/usr/local/k8s/kube-scheduler/conf/kube-scheduler.kubeconfig \\ --leader-elect=true \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/data/logs/kubernetes \\ --v=2" EOF
3、配置kube-scheduler的系统服务文件
cat > /usr/lib/systemd/system/kube-scheduler.service << EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/usr/local/k8s/kube-scheduler/conf/kube-scheduler.conf ExecStart=/usr/local/k8s/kube-scheduler/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
9.3、将配置同步到其他master节点
- 以将配置同步到10.1.1.12机器为例
#同步kubeapiserver目录 scp -r /usr/local/k8s/kube-scheduler root@10.1.1.12:/usr/local/k8s/ #同步系统服务文件 scp /usr/lib/systemd/system/kube-scheduler.service root@10.1.1.12:/usr/lib/systemd/system/kube-scheduler.service
9.4、启动kube-scheduler
#重读系统服务配置，使其生效 systemctl daemon-reload #启动kube-scheduler systemctl enable kube-scheduler.service systemctl start kube-scheduler.service systemctl status kube-scheduler.service
-
查看组件的信息
]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
10、部署kubelet，并将node加入集群
- 每个Node节点上的kubelet定期调用API Server的REST接口报告自身状态，API Server接收这些信息后，将节点d的状态信息更新到etcd中。
- kubelet也通过API Server监听Pod信息，从而对Node机器上的Pod进行管理，如创建、删除、更新pod。
#创建相关目录
mkdir -p /usr/local/k8s/kubelet/{bin,conf,ssl}
#将可执行文件移到bin目录下
mv ./kubernetes/server/bin/kubelet /usr/local/k8s/kubelet/bin
10.1、配置kubelet
- 在一台master机器上进行一下操作，然后同步到各个node机器上
1、生成kubelet初次加入集群的引导文件kubeconfig
KUBE_CONFIG="/usr/local/k8s/kubelet/conf/bootstrap.kubeconfig"
KUBE_APISERVER="https://10.1.1.11:6443" #apiserver的IP:PORT
TOKEN="ae805e2221dff87cac96c2345bf490e5" #与token.csv里保持一致
kubectl config set-cluster kubernetes \
--certificate-authority=/usr/local/k8s/ca/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials "kubelet-bootstrap" \
--token=${TOKEN} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
2、创建配置参数文件
- "cgroupDriver": "systemd"要和docker的驱动一致，否者可能导致kubelet无法正常启动（没有监听端口）
- docker info | grep "Cgroup Driver"
- address替换为node的IP地址。（以10.1.1.14的node机器为例）
- clusterDNS一般是kube-apiserver指定的service-cluster-ip-range网段的第二个IP，如172.22.0.2
cat > /usr/local/k8s/kubelet/conf/kubelet-config.json << EOF
{
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1",
"authentication": {
"x509": {
"clientCAFile": "/usr/local/k8s/ca/ca.pem"
},
"webhook": {
"enabled": true,
"cacheTTL": "2m0s"
},
"anonymous": {
"enabled": false
}
},
"authorization": {
"mode": "Webhook",
"webhook": {
"cacheAuthorizedTTL": "5m0s",
"cacheUnauthorizedTTL": "30s"
}
},
"address": "10.1.1.14",
"port": 10250,
"readOnlyPort": 10255,
"cgroupDriver": "systemd",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"featureGates": {
"RotateKubeletClientCertificate": true,
"RotateKubeletServerCertificate": true
},
"clusterDomain": "cluster.local.",
"clusterDNS": ["172.22.0.2"]
}
EOF
3、创建kubelet的配置文件
- 注意：--hostname-override的值一旦指定尽量不要修改。
cat > /usr/local/k8s/kubelet/conf/kubelet.conf << EOF KUBELET_OPTS="--bootstrap-kubeconfig=/usr/local/k8s/kubelet/conf/bootstrap.kubeconfig \\ --cert-dir=/usr/local/k8s/kubelet/ssl \\ --kubeconfig=/usr/local/k8s/kubelet/conf/kubelet.kubeconfig \\ --config=/usr/local/k8s/kubelet/conf/kubelet-config.json \\ --network-plugin=cni \\ --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause-amd64:3.0 \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/data/logs/kubernetes \\ --v=2" EOF
- --hostname-override：显示名称，集群中唯一
- --network-plugin：启用CNI
- --kubeconfig：空路径，会自动生成，后面用于连接apiserver
- --bootstrap-kubeconfig：首次启动向apiserver申请证书
- --config：配置参数文件
- --cert-dir：kubelet证书生成目录
- --pod-infra-container-image：管理Pod网络容器的镜像
4、配置kubelet的系统服务文件
cat > /usr/lib/systemd/system/kubelet.service << EOF [Unit] Description=Kubernetes Kubelet After=docker.service [Service] EnvironmentFile=-/usr/local/k8s/kubelet/conf/kubelet.conf ExecStart=/usr/local/k8s/kubelet/bin/kubelet \$KUBELET_OPTS Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target EOF
10.2、将配置同步到其他mode节点
- 以将配置同步到10.1.1.14机器为例
#创建相关目录（在10.1.1.14上执行）
mkdir -p /usr/local/k8s/kubelet/{bin,conf,ssl}
#同步kubelet目录
scp -r /usr/local/k8s/kubelet root@10.1.1.14:/usr/local/k8s/
#同步ca目录
scp -r /usr/local/k8s/ca root@10.1.1.14:/usr/local/k8s/
#同步系统服务文件
scp /usr/lib/systemd/system/kubelet.service root@10.1.1.14:/usr/lib/systemd/system/kubelet.service
#修改kubelet-config.json（在10.1.1.14上执行，修改为对应的IP）
vim /usr/local/k8s/kubelet/conf/kubelet-config.json
10.3、启动kubelet
#重读系统服务配置，使其生效 systemctl daemon-reload #启动kubelet systemctl enable kubelet.service systemctl start kubelet.service systemctl status kubelet.service
10.4、将node加入集群
- 注意，因为还没有安装网络插件，所以节点的状态是NotReady。
#查看kubelet证书请求 ]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-j_VNW7cKJDMnfXKuAj1HbaCOUac2Okwr8q0LoSZRXm0 14m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending #批准node加入集群 kubectl certificate approve node-csr-j_VNW7cKJDMnfXKuAj1HbaCOUac2Okwr8q0LoSZRXm0 #查看节点 ]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-node1 NotReady <none> 77s v1.20.14
11、部署kube-proxy
- 在一台master机器上进行一下操作，然后同步到各个node机器上
#创建相关目录
mkdir -p /usr/local/k8s/kube-proxy/{bin,conf,ssl}
#将可执行文件移到bin目录下
mv ./kubernetes/server/bin/kube-proxy /usr/local/k8s/kube-proxy/bin
11.1、生成kube-proxy的证书
1、创建kube-proxy证书的签名请求
cat > /usr/local/k8s/ca/kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
2、生成kube-proxy证书和私钥
cd /usr/local/k8s/ca/
./cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes \
kube-proxy-csr.json | ./cfssljson -bare kube-proxy
3、发放kube-proxy的证书和密钥
mv /usr/local/k8s/ca/kube-proxy*.{pem,csr} /usr/local/k8s/kube-proxy/ssl
11.2、配置kube-proxy
1、生成kube-proxy.kubeconfig文件
KUBE_CONFIG="/usr/local/k8s/kube-proxy/conf/kube-proxy.kubeconfig"
KUBE_APISERVER="https://10.1.1.11:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/usr/local/k8s/ca/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-proxy \
--client-certificate=/usr/local/k8s/kube-proxy/ssl/kube-proxy.pem \
--client-key=/usr/local/k8s/kube-proxy/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
2、创建kube-proxy的参数文件
- clusterCIDR使用的是物理机（虚拟机）所在的网段，不知道为啥。
cat > /usr/local/k8s/kube-proxy/conf/kube-proxy-config.yml << EOF kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 10.1.1.14 healthzBindAddress: 10.1.1.14:10256 metricsBindAddress: 10.1.1.14:10249 clusterCIDR: 10.1.1.0/24 clientConnection: kubeconfig: /usr/local/k8s/kube-proxy/conf/kube-proxy.kubeconfig mode: "ipvs" EOF
3、配置kube-proxy的系统服务文件
cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] WorkingDirectory=-/usr/local/k8s/kube-proxy/conf/kube-proxy.conf ExecStart=/usr/local/k8s/kube-proxy/bin/kube-proxy \\ --config=/usr/local/k8s/kube-proxy/conf/kube-proxy-config.yml \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/data/logs/kubernetes \\ --v=2 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
11.3、将配置同步到各个node节点
#创建相关目录（在10.1.1.14上执行）
mkdir -p /usr/local/k8s/kube-proxy/{bin,conf,ssl}
#同步kubelet目录
scp -r /usr/local/k8s/kube-proxy root@10.1.1.14:/usr/local/k8s/
#同步系统服务文件
scp /usr/lib/systemd/system/kube-proxy.service root@10.1.1.14:/usr/lib/systemd/system/kube-proxy.service
#修改kube-proxy-config.yml（在10.1.1.14上执行，修改为对应的IP）
vim /usr/local/k8s/kube-proxy/conf/kube-proxy-config.yml
11.4、启动kube-proxy
#重读系统服务配置，使其生效 systemctl daemon-reload #启动kube-proxy systemctl enable kube-proxy.service systemctl start kube-proxy.service systemctl status kube-proxy.service
12、部署网络插件
12.1、部署Calico
- Calico是一个纯三层的数据中心网络方案，是目前Kubernetes主流的网络方案。
- 三种不同情况，三种不同安装Calico的方式 ：
- 使用Kubernetes API数据存储安装Calico，50个节点或更少。
- 使用Kubernetes API数据存储安装Calico，超过50个节点。
- 使用etcd 数据存储安装Calico。
1、下载calico
官网：https://www.tigera.io/
安装说明：https://projectcalico.docs.tigera.io/archive/v3.19/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-etcd-datastore
#下载Calico
wget https://docs.projectcalico.org/archive/v3.19/manifests/calico.yaml --no-check-certificate \
-O /usr/local/k8s/calico.yaml
2、修改calico.yaml文件
- 取消掉对清单中CALICO_IPV4POOL_CIDR变量的注释，并将其设置为与--cluster-cidr相同。
]# vim /usr/local/k8s/calico.yaml
- name: CALICO_IPV4POOL_CIDR
value: "172.21.0.0/16"
- name: IP_AUTODETECTION_METHOD
value: "interface=ens33"
- calico.yaml文件
--- # Source: calico/templates/calico-config.yaml # This ConfigMap is used to configure a self-hosted Calico installation. kind: ConfigMap apiVersion: v1 metadata: name: calico-config namespace: kube-system data: # Typha is disabled. typha_service_name: "none" # Configure the backend to use. calico_backend: "bird" # Configure the MTU to use for workload interfaces and tunnels. # By default, MTU is auto-detected, and explicitly setting this field should not be required. # You can override auto-detection by providing a non-zero value. veth_mtu: "0" # The CNI network configuration to install on each node. The special # values in this config will be automatically populated. cni_network_config: |- { "name": "k8s-pod-network", "cniVersion": "0.3.1", "plugins": [ { "type": "calico", "log_level": "info", "log_file_path": "/var/log/calico/cni/cni.log", "datastore_type": "kubernetes", "nodename": "__KUBERNETES_NODE_NAME__", "mtu": __CNI_MTU__, "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "__KUBECONFIG_FILEPATH__" } }, { "type": "portmap", "snat": true, "capabilities": {"portMappings": true} }, { "type": "bandwidth", "capabilities": {"bandwidth": true} } ] } --- # Source: calico/templates/kdd-crds.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: bgpconfigurations.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: BGPConfiguration listKind: BGPConfigurationList plural: bgpconfigurations singular: bgpconfiguration scope: Cluster versions: - name: v1 schema: openAPIV3Schema: description: BGPConfiguration contains the configuration for any BGP routing. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: BGPConfigurationSpec contains the values of the BGP configuration. properties: asNumber: description: 'ASNumber is the default AS number used by a node. [Default: 64512]' format: int32 type: integer communities: description: Communities is a list of BGP community values and their arbitrary names for tagging routes. items: description: Community contains standard or large community value and its name. properties: name: description: Name given to community value. type: string value: description: Value must be of format `aa:nn` or `aa:nn:mm`. For standard community use `aa:nn` format, where `aa` and `nn` are 16 bit number. For large community use `aa:nn:mm` format, where `aa`, `nn` and `mm` are 32 bit number. Where, `aa` is an AS Number, `nn` and `mm` are per-AS identifier. pattern: ^(\d+):(\d+)$|^(\d+):(\d+):(\d+)$ type: string type: object type: array listenPort: description: ListenPort is the port where BGP protocol should listen. Defaults to 179 maximum: 65535 minimum: 1 type: integer logSeverityScreen: description: 'LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: INFO]' type: string nodeToNodeMeshEnabled: description: 'NodeToNodeMeshEnabled sets whether full node to node BGP mesh is enabled. [Default: true]' type: boolean prefixAdvertisements: description: PrefixAdvertisements contains per-prefix advertisement configuration. items: description: PrefixAdvertisement configures advertisement properties for the specified CIDR. properties: cidr: description: CIDR for which properties should be advertised. type: string communities: description: Communities can be list of either community names already defined in `Specs.Communities` or community value of format `aa:nn` or `aa:nn:mm`. For standard community use `aa:nn` format, where `aa` and `nn` are 16 bit number. For large community use `aa:nn:mm` format, where `aa`, `nn` and `mm` are 32 bit number. Where,`aa` is an AS Number, `nn` and `mm` are per-AS identifier. items: type: string type: array type: object type: array serviceClusterIPs: description: ServiceClusterIPs are the CIDR blocks from which service cluster IPs are allocated. If specified, Calico will advertise these blocks, as well as any cluster IPs within them. items: description: ServiceClusterIPBlock represents a single allowed ClusterIP CIDR block. properties: cidr: type: string type: object type: array serviceExternalIPs: description: ServiceExternalIPs are the CIDR blocks for Kubernetes Service External IPs. Kubernetes Service ExternalIPs will only be advertised if they are within one of these blocks. items: description: ServiceExternalIPBlock represents a single allowed External IP CIDR block. properties: cidr: type: string type: object type: array serviceLoadBalancerIPs: description: ServiceLoadBalancerIPs are the CIDR blocks for Kubernetes Service LoadBalancer IPs. Kubernetes Service status.LoadBalancer.Ingress IPs will only be advertised if they are within one of these blocks. items: description: ServiceLoadBalancerIPBlock represents a single allowed LoadBalancer IP CIDR block. properties: cidr: type: string type: object type: array type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: bgppeers.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: BGPPeer listKind: BGPPeerList plural: bgppeers singular: bgppeer scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: BGPPeerSpec contains the specification for a BGPPeer resource. properties: asNumber: description: The AS Number of the peer. format: int32 type: integer keepOriginalNextHop: description: Option to keep the original nexthop field when routes are sent to a BGP Peer. Setting "true" configures the selected BGP Peers node to use the "next hop keep;" instead of "next hop self;"(default) in the specific branch of the Node on "bird.cfg". type: boolean node: description: The node name identifying the Calico node instance that is targeted by this peer. If this is not set, and no nodeSelector is specified, then this BGP peer selects all nodes in the cluster. type: string nodeSelector: description: Selector for the nodes that should have this peering. When this is set, the Node field must be empty. type: string password: description: Optional BGP password for the peerings generated by this BGPPeer resource. properties: secretKeyRef: description: Selects a key of a secret in the node pod's namespace. properties: key: description: The key of the secret to select from. Must be a valid secret key. type: string name: description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names TODO: Add other useful fields. apiVersion, kind, uid?' type: string optional: description: Specify whether the Secret or its key must be defined type: boolean required: - key type: object type: object peerIP: description: The IP address of the peer followed by an optional port number to peer with. If port number is given, format should be `[<IPv6>]:port` or `<IPv4>:<port>` for IPv4. If optional port number is not set, and this peer IP and ASNumber belongs to a calico/node with ListenPort set in BGPConfiguration, then we use that port to peer. type: string peerSelector: description: Selector for the remote nodes to peer with. When this is set, the PeerIP and ASNumber fields must be empty. For each peering between the local node and selected remote nodes, we configure an IPv4 peering if both ends have NodeBGPSpec.IPv4Address specified, and an IPv6 peering if both ends have NodeBGPSpec.IPv6Address specified. The remote AS number comes from the remote node's NodeBGPSpec.ASNumber, or the global default if that is not set. type: string sourceAddress: description: Specifies whether and how to configure a source address for the peerings generated by this BGPPeer resource. Default value "UseNodeIP" means to configure the node IP as the source address. "None" means not to configure a source address. type: string type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: blockaffinities.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: BlockAffinity listKind: BlockAffinityList plural: blockaffinities singular: blockaffinity scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: BlockAffinitySpec contains the specification for a BlockAffinity resource. properties: cidr: type: string deleted: description: Deleted indicates that this block affinity is being deleted. This field is a string for compatibility with older releases that mistakenly treat this field as a string. type: string node: type: string state: type: string required: - cidr - deleted - node - state type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: clusterinformations.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: ClusterInformation listKind: ClusterInformationList plural: clusterinformations singular: clusterinformation scope: Cluster versions: - name: v1 schema: openAPIV3Schema: description: ClusterInformation contains the cluster specific information. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: ClusterInformationSpec contains the values of describing the cluster. properties: calicoVersion: description: CalicoVersion is the version of Calico that the cluster is running type: string clusterGUID: description: ClusterGUID is the GUID of the cluster type: string clusterType: description: ClusterType describes the type of the cluster type: string datastoreReady: description: DatastoreReady is used during significant datastore migrations to signal to components such as Felix that it should wait before accessing the datastore. type: boolean variant: description: Variant declares which variant of Calico should be active. type: string type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: felixconfigurations.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: FelixConfiguration listKind: FelixConfigurationList plural: felixconfigurations singular: felixconfiguration scope: Cluster versions: - name: v1 schema: openAPIV3Schema: description: Felix Configuration contains the configuration for Felix. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: FelixConfigurationSpec contains the values of the Felix configuration. properties: allowIPIPPacketsFromWorkloads: description: 'AllowIPIPPacketsFromWorkloads controls whether Felix will add a rule to drop IPIP encapsulated traffic from workloads [Default: false]' type: boolean allowVXLANPacketsFromWorkloads: description: 'AllowVXLANPacketsFromWorkloads controls whether Felix will add a rule to drop VXLAN encapsulated traffic from workloads [Default: false]' type: boolean awsSrcDstCheck: description: 'Set source-destination-check on AWS EC2 instances. Accepted value must be one of "DoNothing", "Enabled" or "Disabled". [Default: DoNothing]' enum: - DoNothing - Enable - Disable type: string bpfConnectTimeLoadBalancingEnabled: description: 'BPFConnectTimeLoadBalancingEnabled when in BPF mode, controls whether Felix installs the connection-time load balancer. The connect-time load balancer is required for the host to be able to reach Kubernetes services and it improves the performance of pod-to-service connections. The only reason to disable it is for debugging purposes. [Default: true]' type: boolean bpfDataIfacePattern: description: BPFDataIfacePattern is a regular expression that controls which interfaces Felix should attach BPF programs to in order to catch traffic to/from the network. This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to nodeports and services from outside the cluster. It should not match the workload interfaces (usually named cali...). type: string bpfDisableUnprivileged: description: 'BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable unprivileged use of BPF. This ensures that unprivileged users cannot access Calico''s BPF maps and cannot insert their own BPF programs to interfere with Calico''s. [Default: true]' type: boolean bpfEnabled: description: 'BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]' type: boolean bpfExtToServiceConnmark: description: 'BPFExtToServiceConnmark in BPF mode, control a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing intepreted by RPF check. [Default: 0]' type: integer bpfExternalServiceMode: description: 'BPFExternalServiceMode in BPF mode, controls how connections from outside the cluster to services (node ports and cluster IPs) are forwarded to remote workloads. If set to "Tunnel" then both request and response traffic is tunneled to the remote node. If set to "DSR", the request traffic is tunneled but the response traffic is sent directly from the remote node. In "DSR" mode, the remote node appears to use the IP of the ingress node; this requires a permissive L2 network. [Default: Tunnel]' type: string bpfKubeProxyEndpointSlicesEnabled: description: BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls whether Felix's embedded kube-proxy accepts EndpointSlices or not. type: boolean bpfKubeProxyIptablesCleanupEnabled: description: 'BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF mode, Felix will proactively clean up the upstream Kubernetes kube-proxy''s iptables chains. Should only be enabled if kube-proxy is not running. [Default: true]' type: boolean bpfKubeProxyMinSyncPeriod: description: 'BPFKubeProxyMinSyncPeriod, in BPF mode, controls the minimum time between updates to the dataplane for Felix''s embedded kube-proxy. Lower values give reduced set-up latency. Higher values reduce Felix CPU usage by batching up more work. [Default: 1s]' type: string bpfLogLevel: description: 'BPFLogLevel controls the log level of the BPF programs when in BPF dataplane mode. One of "Off", "Info", or "Debug". The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. [Default: Off].' type: string chainInsertMode: description: 'ChainInsertMode controls whether Felix hooks the kernel''s top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. insert is the safe default since it prevents Calico''s rules from being bypassed. If you switch to append mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico rules, otherwise the Calico policy will be bypassed. [Default: insert]' type: string dataplaneDriver: type: string debugDisableLogDropping: type: boolean debugMemoryProfilePath: type: string debugSimulateCalcGraphHangAfter: type: string debugSimulateDataplaneHangAfter: type: string defaultEndpointToHostAction: description: 'DefaultEndpointToHostAction controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico blocks traffic from workload endpoints to the host itself with an iptables "DROP" action. If you want to allow some or all traffic from endpoint to host, set this parameter to RETURN or ACCEPT. Use RETURN if you have your own rules in the iptables "INPUT" chain; Calico will insert its rules at the top of that chain, then "RETURN" packets to the "INPUT" chain once it has completed processing workload endpoint egress policy. Use ACCEPT to unconditionally accept packets from workloads after processing workload endpoint egress policy. [Default: Drop]' type: string deviceRouteProtocol: description: This defines the route protocol added to programmed device routes, by default this will be RTPROT_BOOT when left blank. type: integer deviceRouteSourceAddress: description: This is the source address to use on programmed device routes. By default the source address is left blank, leaving the kernel to choose the source address used. type: string disableConntrackInvalidCheck: type: boolean endpointReportingDelay: type: string endpointReportingEnabled: type: boolean externalNodesList: description: ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes which may source tunnel traffic and have the tunneled traffic be accepted at calico nodes. items: type: string type: array failsafeInboundHostPorts: description: 'FailsafeInboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to "tcp". If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value none. The default value allows ssh access and DHCP. [Default: tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]' items: description: ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified. properties: net: type: string port: type: integer protocol: type: string required: - port - protocol type: object type: array failsafeOutboundHostPorts: description: 'FailsafeOutboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to "tcp". If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value none. The default value opens etcd''s standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP and DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, udp:53, udp:67]' items: description: ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified. properties: net: type: string port: type: integer protocol: type: string required: - port - protocol type: object type: array featureDetectOverride: description: FeatureDetectOverride is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=". "true" or "false" will force the feature, empty or omitted values are auto-detected. type: string genericXDPEnabled: description: 'GenericXDPEnabled enables Generic XDP so network cards that don''t support XDP offload or driver modes can use XDP. This is not recommended since it doesn''t provide better performance than iptables. [Default: false]' type: boolean healthEnabled: type: boolean healthHost: type: string healthPort: type: integer interfaceExclude: description: 'InterfaceExclude is a comma-separated list of interfaces that Felix should exclude when monitoring for host endpoints. The default value ensures that Felix ignores Kubernetes'' IPVS dummy interface, which is used internally by kube-proxy. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with ''/''. For example having values ''/^kube/,veth1'' will exclude all interfaces that begin with ''kube'' and also the interface ''veth1''. [Default: kube-ipvs0]' type: string interfacePrefix: description: 'InterfacePrefix is the interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the ''cali'' value, and our OpenStack integration sets the ''tap'' value. [Default: cali]' type: string interfaceRefreshInterval: description: InterfaceRefreshInterval is the period at which Felix rescans local interfaces to verify their state. The rescan can be disabled by setting the interval to 0. type: string ipipEnabled: type: boolean ipipMTU: description: 'IPIPMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]' type: integer ipsetsRefreshInterval: description: 'IpsetsRefreshInterval is the period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico''s rules. Set to 0 to disable iptables refresh. [Default: 90s]' type: string iptablesBackend: description: IptablesBackend specifies which backend of iptables will be used. The default is legacy. type: string iptablesFilterAllowAction: type: string iptablesLockFilePath: description: 'IptablesLockFilePath is the location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix''s container at a different path). [Default: /run/xtables.lock]' type: string iptablesLockProbeInterval: description: 'IptablesLockProbeInterval is the time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. [Default: 50ms]' type: string iptablesLockTimeout: description: 'IptablesLockTimeout is the time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the calico/node or calico/felix container. [Default: 0s disabled]' type: string iptablesMangleAllowAction: type: string iptablesMarkMask: description: 'IptablesMarkMask is the mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. [Default: 0xff000000]' format: int32 type: integer iptablesNATOutgoingInterfaceFilter: type: string iptablesPostWriteCheckInterval: description: 'IptablesPostWriteCheckInterval is the period after Felix has done a write to the dataplane that it schedules an extra read back in order to check the write was not clobbered by another process. This should only occur if another application on the system doesn''t respect the iptables lock. [Default: 1s]' type: string iptablesRefreshInterval: description: 'IptablesRefreshInterval is the period at which Felix re-checks the IP sets in the dataplane to ensure that no other process has accidentally broken Calico''s rules. Set to 0 to disable IP sets refresh. Note: the default for this value is lower than the other refresh intervals as a workaround for a Linux kernel bug that was fixed in kernel version 4.11. If you are using v4.11 or greater you may want to set this to, a higher value to reduce Felix CPU usage. [Default: 10s]' type: string ipv6Support: type: boolean kubeNodePortRanges: description: 'KubeNodePortRanges holds list of port ranges used for service node ports. Only used if felix detects kube-proxy running in ipvs mode. Felix uses these ranges to separate host and workload traffic. [Default: 30000:32767].' items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array logFilePath: description: 'LogFilePath is the full path to the Felix log. Set to none to disable file logging. [Default: /var/log/calico/felix.log]' type: string logPrefix: description: 'LogPrefix is the log prefix that Felix uses when rendering LOG rules. [Default: calico-packet]' type: string logSeverityFile: description: 'LogSeverityFile is the log severity above which logs are sent to the log file. [Default: Info]' type: string logSeverityScreen: description: 'LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]' type: string logSeveritySys: description: 'LogSeveritySys is the log severity above which logs are sent to the syslog. Set to None for no logging to syslog. [Default: Info]' type: string maxIpsetSize: type: integer metadataAddr: description: 'MetadataAddr is the IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of none (case insensitive) means that Felix should not set up any NAT rule for the metadata path. [Default: 127.0.0.1]' type: string metadataPort: description: 'MetadataPort is the port of the metadata server. This, combined with global.MetadataAddr (if not ''None''), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed [Default: 8775].' type: integer mtuIfacePattern: description: MTUIfacePattern is a regular expression that controls which interfaces Felix should scan in order to calculate the host's MTU. This should not match workload interfaces (usually named cali...). type: string natOutgoingAddress: description: NATOutgoingAddress specifies an address to use when performing source NAT for traffic in a natOutgoing pool that is leaving the network. By default the address used is an address on the interface the traffic is leaving on (ie it uses the iptables MASQUERADE target) type: string natPortRange: anyOf: - type: integer - type: string description: NATPortRange specifies the range of ports that is used for port mapping when doing outgoing NAT. When unset the default behavior of the network stack is used. pattern: ^.* x-kubernetes-int-or-string: true netlinkTimeout: type: string openstackRegion: description: 'OpenstackRegion is the name of the region that a particular Felix belongs to. In a multi-region Calico/OpenStack deployment, this must be configured somehow for each Felix (here in the datamodel, or in felix.cfg or the environment on each compute node), and must match the [calico] openstack_region value configured in neutron.conf on each node. [Default: Empty]' type: string policySyncPathPrefix: description: 'PolicySyncPathPrefix is used to by Felix to communicate policy changes to external services, like Application layer policy. [Default: Empty]' type: string prometheusGoMetricsEnabled: description: 'PrometheusGoMetricsEnabled disables Go runtime metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]' type: boolean prometheusMetricsEnabled: description: 'PrometheusMetricsEnabled enables the Prometheus metrics server in Felix if set to true. [Default: false]' type: boolean prometheusMetricsHost: description: 'PrometheusMetricsHost is the host that the Prometheus metrics server should bind to. [Default: empty]' type: string prometheusMetricsPort: description: 'PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. [Default: 9091]' type: integer prometheusProcessMetricsEnabled: description: 'PrometheusProcessMetricsEnabled disables process metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]' type: boolean removeExternalRoutes: description: Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. This is enabled by default which means we will remove externally added routes. type: boolean reportingInterval: description: 'ReportingInterval is the interval at which Felix reports its status into the datastore or 0 to disable. Must be non-zero in OpenStack deployments. [Default: 30s]' type: string reportingTTL: description: 'ReportingTTL is the time-to-live setting for process-wide status reports. [Default: 90s]' type: string routeRefreshInterval: description: 'RouteRefreshInterval is the period at which Felix re-checks the routes in the dataplane to ensure that no other process has accidentally broken Calico''s rules. Set to 0 to disable route refresh. [Default: 90s]' type: string routeSource: description: 'RouteSource configures where Felix gets its routing information. - WorkloadIPs: use workload endpoints to construct routes. - CalicoIPAM: the default - use IPAM data to construct routes.' type: string routeTableRange: description: Calico programs additional Linux route tables for various purposes. RouteTableRange specifies the indices of the route tables that Calico should use. properties: max: type: integer min: type: integer required: - max - min type: object serviceLoopPrevention: description: 'When service IP advertisement is enabled, prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT''d by kube-proxy. Unless set to "Disabled", in which case such routing loops continue to be allowed. [Default: Drop]' type: string sidecarAccelerationEnabled: description: 'SidecarAccelerationEnabled enables experimental sidecar acceleration [Default: false]' type: boolean usageReportingEnabled: description: 'UsageReportingEnabled reports anonymous Calico version number and cluster size to projectcalico.org. Logs warnings returned by the usage server. For example, if a significant security vulnerability has been discovered in the version of Calico being used. [Default: true]' type: boolean usageReportingInitialDelay: description: 'UsageReportingInitialDelay controls the minimum delay before Felix makes a report. [Default: 300s]' type: string usageReportingInterval: description: 'UsageReportingInterval controls the interval at which Felix makes reports. [Default: 86400s]' type: string useInternalDataplaneDriver: type: boolean vxlanEnabled: type: boolean vxlanMTU: description: 'VXLANMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]' type: integer vxlanPort: type: integer vxlanVNI: type: integer wireguardEnabled: description: 'WireguardEnabled controls whether Wireguard is enabled. [Default: false]' type: boolean wireguardInterfaceName: description: 'WireguardInterfaceName specifies the name to use for the Wireguard interface. [Default: wg.calico]' type: string wireguardListeningPort: description: 'WireguardListeningPort controls the listening port used by Wireguard. [Default: 51820]' type: integer wireguardMTU: description: 'WireguardMTU controls the MTU on the Wireguard interface. See Configuring MTU [Default: 1420]' type: integer wireguardRoutingRulePriority: description: 'WireguardRoutingRulePriority controls the priority value to use for the Wireguard routing rule. [Default: 99]' type: integer xdpEnabled: description: 'XDPEnabled enables XDP acceleration for suitable untracked incoming deny rules. [Default: true]' type: boolean xdpRefreshInterval: description: 'XDPRefreshInterval is the period at which Felix re-checks all XDP state to ensure that no other process has accidentally broken Calico''s BPF maps or attached programs. Set to 0 to disable XDP refresh. [Default: 90s]' type: string type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: globalnetworkpolicies.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: GlobalNetworkPolicy listKind: GlobalNetworkPolicyList plural: globalnetworkpolicies singular: globalnetworkpolicy scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: properties: applyOnForward: description: ApplyOnForward indicates to apply the rules in this policy on forward traffic. type: boolean doNotTrack: description: DoNotTrack indicates whether packets matched by the rules in this policy should go through the data plane's connection tracking, such as Linux conntrack. If True, the rules in this policy are applied before any data plane connection tracking, and packets allowed by this policy are marked as not to be tracked. type: boolean egress: description: The ordered set of egress rules. Each rule contains a set of packet match criteria and a corresponding action to apply. items: description: "A Rule encapsulates a set of match criteria and an action. Both selector-based security Policy and security Profiles reference rules - separated out as a list of rules for both ingress and egress packet matching. \n Each positive match criteria has a negated version, prefixed with \"Not\". All the match criteria within a rule must be satisfied for a packet to match. A single rule can contain the positive and negative version of a match and both must be satisfied for the rule to match." properties: action: type: string destination: description: Destination contains the match criteria that apply to destination entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object http: description: HTTP contains match criteria that apply to HTTP requests. properties: methods: description: Methods is an optional field that restricts the rule to apply only to HTTP requests that use one of the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple methods are OR'd together. items: type: string type: array paths: description: 'Paths is an optional field that restricts the rule to apply to HTTP requests that use one of the listed HTTP Paths. Multiple paths are OR''d together. e.g: - exact: /foo - prefix: /bar NOTE: Each entry may ONLY specify either a `exact` or a `prefix` match. The validator will check for it.' items: description: 'HTTPPath specifies an HTTP path to match. It may be either of the form: exact: <path>: which matches the path exactly or prefix: <path-prefix>: which matches the path prefix' properties: exact: type: string prefix: type: string type: object type: array type: object icmp: description: ICMP is an optional field that restricts the rule to apply to a specific type and code of ICMP traffic. This should only be specified if the Protocol field is set to "ICMP" or "ICMPv6". properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object ipVersion: description: IPVersion is an optional field that restricts the rule to only match a specific IP version. type: integer metadata: description: Metadata contains additional information for this rule properties: annotations: additionalProperties: type: string description: Annotations is a set of key value pairs that give extra information about the rule type: object type: object notICMP: description: NotICMP is the negated version of the ICMP field. properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object notProtocol: anyOf: - type: integer - type: string description: NotProtocol is the negated version of the Protocol field. pattern: ^.* x-kubernetes-int-or-string: true protocol: anyOf: - type: integer - type: string description: "Protocol is an optional field that restricts the rule to only apply to traffic of a specific IP protocol. Required if any of the EntityRules contain Ports (because ports only apply to certain protocols). \n Must be one of these string values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\", \"UDPLite\" or an integer in the range 1-255." pattern: ^.* x-kubernetes-int-or-string: true source: description: Source contains the match criteria that apply to source entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object required: - action type: object type: array ingress: description: The ordered set of ingress rules. Each rule contains a set of packet match criteria and a corresponding action to apply. items: description: "A Rule encapsulates a set of match criteria and an action. Both selector-based security Policy and security Profiles reference rules - separated out as a list of rules for both ingress and egress packet matching. \n Each positive match criteria has a negated version, prefixed with \"Not\". All the match criteria within a rule must be satisfied for a packet to match. A single rule can contain the positive and negative version of a match and both must be satisfied for the rule to match." properties: action: type: string destination: description: Destination contains the match criteria that apply to destination entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object http: description: HTTP contains match criteria that apply to HTTP requests. properties: methods: description: Methods is an optional field that restricts the rule to apply only to HTTP requests that use one of the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple methods are OR'd together. items: type: string type: array paths: description: 'Paths is an optional field that restricts the rule to apply to HTTP requests that use one of the listed HTTP Paths. Multiple paths are OR''d together. e.g: - exact: /foo - prefix: /bar NOTE: Each entry may ONLY specify either a `exact` or a `prefix` match. The validator will check for it.' items: description: 'HTTPPath specifies an HTTP path to match. It may be either of the form: exact: <path>: which matches the path exactly or prefix: <path-prefix>: which matches the path prefix' properties: exact: type: string prefix: type: string type: object type: array type: object icmp: description: ICMP is an optional field that restricts the rule to apply to a specific type and code of ICMP traffic. This should only be specified if the Protocol field is set to "ICMP" or "ICMPv6". properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object ipVersion: description: IPVersion is an optional field that restricts the rule to only match a specific IP version. type: integer metadata: description: Metadata contains additional information for this rule properties: annotations: additionalProperties: type: string description: Annotations is a set of key value pairs that give extra information about the rule type: object type: object notICMP: description: NotICMP is the negated version of the ICMP field. properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object notProtocol: anyOf: - type: integer - type: string description: NotProtocol is the negated version of the Protocol field. pattern: ^.* x-kubernetes-int-or-string: true protocol: anyOf: - type: integer - type: string description: "Protocol is an optional field that restricts the rule to only apply to traffic of a specific IP protocol. Required if any of the EntityRules contain Ports (because ports only apply to certain protocols). \n Must be one of these string values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\", \"UDPLite\" or an integer in the range 1-255." pattern: ^.* x-kubernetes-int-or-string: true source: description: Source contains the match criteria that apply to source entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object required: - action type: object type: array namespaceSelector: description: NamespaceSelector is an optional field for an expression used to select a pod based on namespaces. type: string order: description: Order is an optional field that specifies the order in which the policy is applied. Policies with higher "order" are applied after those with lower order. If the order is omitted, it may be considered to be "infinite" - i.e. the policy will be applied last. Policies with identical order will be applied in alphanumerical order based on the Policy "Name". type: number preDNAT: description: PreDNAT indicates to apply the rules in this policy before any DNAT. type: boolean selector: description: "The selector is an expression used to pick pick out the endpoints that the policy should be applied to. \n Selector expressions follow this syntax: \n \tlabel == \"string_literal\" \ -> comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\" \ -> not equal; also matches if label is not present \tlabel in { \"a\", \"b\", \"c\", ... } -> true if the value of label X is one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\", ... } -> true if the value of label X is not one of \"a\", \"b\", \"c\" \thas(label_name) -> True if that label is present \t! expr -> negation of expr \texpr && expr -> Short-circuit and \texpr || expr -> Short-circuit or \t( expr ) -> parens for grouping \tall() or the empty selector -> matches all endpoints. \n Label names are allowed to contain alphanumerics, -, _ and /. String literals are more permissive but they do not support escape characters. \n Examples (with made-up labels): \n \ttype == \"webserver\" && deployment == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment != \"dev\" \t! has(label_name)" type: string serviceAccountSelector: description: ServiceAccountSelector is an optional field for an expression used to select a pod based on service accounts. type: string types: description: "Types indicates whether this policy applies to ingress, or to egress, or to both. When not explicitly specified (and so the value on creation is empty or nil), Calico defaults Types according to what Ingress and Egress rules are present in the policy. The default is: \n - [ PolicyTypeIngress ], if there are no Egress rules (including the case where there are also no Ingress rules) \n - [ PolicyTypeEgress ], if there are Egress rules but no Ingress rules \n - [ PolicyTypeIngress, PolicyTypeEgress ], if there are both Ingress and Egress rules. \n When the policy is read back again, Types will always be one of these values, never empty or nil." items: description: PolicyType enumerates the possible values of the PolicySpec Types field. type: string type: array type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: globalnetworksets.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: GlobalNetworkSet listKind: GlobalNetworkSetList plural: globalnetworksets singular: globalnetworkset scope: Cluster versions: - name: v1 schema: openAPIV3Schema: description: GlobalNetworkSet contains a set of arbitrary IP sub-networks/CIDRs that share labels to allow rules to refer to them via selectors. The labels of GlobalNetworkSet are not namespaced. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: GlobalNetworkSetSpec contains the specification for a NetworkSet resource. properties: nets: description: The list of IP networks that belong to this set. items: type: string type: array type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: hostendpoints.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: HostEndpoint listKind: HostEndpointList plural: hostendpoints singular: hostendpoint scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: HostEndpointSpec contains the specification for a HostEndpoint resource. properties: expectedIPs: description: "The expected IP addresses (IPv4 and IPv6) of the endpoint. If \"InterfaceName\" is not present, Calico will look for an interface matching any of the IPs in the list and apply policy to that. Note: \tWhen using the selector match criteria in an ingress or egress security Policy \tor Profile, Calico converts the selector into a set of IP addresses. For host \tendpoints, the ExpectedIPs field is used for that purpose. (If only the interface \tname is specified, Calico does not learn the IPs of the interface for use in match \tcriteria.)" items: type: string type: array interfaceName: description: "Either \"*\", or the name of a specific Linux interface to apply policy to; or empty. \"*\" indicates that this HostEndpoint governs all traffic to, from or through the default network namespace of the host named by the \"Node\" field; entering and leaving that namespace via any interface, including those from/to non-host-networked local workloads. \n If InterfaceName is not \"*\", this HostEndpoint only governs traffic that enters or leaves the host through the specific interface named by InterfaceName, or - when InterfaceName is empty - through the specific interface that has one of the IPs in ExpectedIPs. Therefore, when InterfaceName is empty, at least one expected IP must be specified. Only external interfaces (such as \"eth0\") are supported here; it isn't possible for a HostEndpoint to protect traffic through a specific local workload interface. \n Note: Only some kinds of policy are implemented for \"*\" HostEndpoints; initially just pre-DNAT policy. Please check Calico documentation for the latest position." type: string node: description: The node name identifying the Calico node instance. type: string ports: description: Ports contains the endpoint's named ports, which may be referenced in security policy rules. items: properties: name: type: string port: type: integer protocol: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true required: - name - port - protocol type: object type: array profiles: description: A list of identifiers of security Profile objects that apply to this endpoint. Each profile is applied in the order that they appear in this list. Profile rules are applied after the selector-based security policy. items: type: string type: array type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: ipamblocks.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: IPAMBlock listKind: IPAMBlockList plural: ipamblocks singular: ipamblock scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IPAMBlockSpec contains the specification for an IPAMBlock resource. properties: affinity: type: string allocations: items: type: integer # TODO: This nullable is manually added in. We should update controller-gen # to handle []*int properly itself. nullable: true type: array attributes: items: properties: handle_id: type: string secondary: additionalProperties: type: string type: object type: object type: array cidr: type: string deleted: type: boolean strictAffinity: type: boolean unallocated: items: type: integer type: array required: - allocations - attributes - cidr - strictAffinity - unallocated type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: ipamconfigs.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: IPAMConfig listKind: IPAMConfigList plural: ipamconfigs singular: ipamconfig scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IPAMConfigSpec contains the specification for an IPAMConfig resource. properties: autoAllocateBlocks: type: boolean maxBlocksPerHost: description: MaxBlocksPerHost, if non-zero, is the max number of blocks that can be affine to each host. type: integer strictAffinity: type: boolean required: - autoAllocateBlocks - strictAffinity type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: ipamhandles.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: IPAMHandle listKind: IPAMHandleList plural: ipamhandles singular: ipamhandle scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IPAMHandleSpec contains the specification for an IPAMHandle resource. properties: block: additionalProperties: type: integer type: object deleted: type: boolean handleID: type: string required: - block - handleID type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: ippools.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: IPPool listKind: IPPoolList plural: ippools singular: ippool scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IPPoolSpec contains the specification for an IPPool resource. properties: blockSize: description: The block size to use for IP address assignments from this pool. Defaults to 26 for IPv4 and 112 for IPv6. type: integer cidr: description: The pool CIDR. type: string disabled: description: When disabled is true, Calico IPAM will not assign addresses from this pool. type: boolean ipip: description: 'Deprecated: this field is only used for APIv1 backwards compatibility. Setting this field is not allowed, this field is for internal use only.' properties: enabled: description: When enabled is true, ipip tunneling will be used to deliver packets to destinations within this pool. type: boolean mode: description: The IPIP mode. This can be one of "always" or "cross-subnet". A mode of "always" will also use IPIP tunneling for routing to destination IP addresses within this pool. A mode of "cross-subnet" will only use IPIP tunneling when the destination node is on a different subnet to the originating node. The default value (if not specified) is "always". type: string type: object ipipMode: description: Contains configuration for IPIP tunneling for this pool. If not specified, then this is defaulted to "Never" (i.e. IPIP tunneling is disabled). type: string nat-outgoing: description: 'Deprecated: this field is only used for APIv1 backwards compatibility. Setting this field is not allowed, this field is for internal use only.' type: boolean natOutgoing: description: When nat-outgoing is true, packets sent from Calico networked containers in this pool to destinations outside of this pool will be masqueraded. type: boolean nodeSelector: description: Allows IPPool to allocate for a specific node by label selector. type: string vxlanMode: description: Contains configuration for VXLAN tunneling for this pool. If not specified, then this is defaulted to "Never" (i.e. VXLAN tunneling is disabled). type: string required: - cidr type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: kubecontrollersconfigurations.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: KubeControllersConfiguration listKind: KubeControllersConfigurationList plural: kubecontrollersconfigurations singular: kubecontrollersconfiguration scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: KubeControllersConfigurationSpec contains the values of the Kubernetes controllers configuration. properties: controllers: description: Controllers enables and configures individual Kubernetes controllers properties: namespace: description: Namespace enables and configures the namespace controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object node: description: Node enables and configures the node controller. Enabled by default, set to nil to disable. properties: hostEndpoint: description: HostEndpoint controls syncing nodes to host endpoints. Disabled by default, set to nil to disable. properties: autoCreate: description: 'AutoCreate enables automatic creation of host endpoints for every node. [Default: Disabled]' type: string type: object reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string syncLabels: description: 'SyncLabels controls whether to copy Kubernetes node labels to Calico nodes. [Default: Enabled]' type: string type: object policy: description: Policy enables and configures the policy controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object serviceAccount: description: ServiceAccount enables and configures the service account controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object workloadEndpoint: description: WorkloadEndpoint enables and configures the workload endpoint controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object type: object etcdV3CompactionPeriod: description: 'EtcdV3CompactionPeriod is the period between etcdv3 compaction requests. Set to 0 to disable. [Default: 10m]' type: string healthChecks: description: 'HealthChecks enables or disables support for health checks [Default: Enabled]' type: string logSeverityScreen: description: 'LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]' type: string prometheusMetricsPort: description: 'PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. Set to 0 to disable. [Default: 9094]' type: integer required: - controllers type: object status: description: KubeControllersConfigurationStatus represents the status of the configuration. It's useful for admins to be able to see the actual config that was applied, which can be modified by environment variables on the kube-controllers process. properties: environmentVars: additionalProperties: type: string description: EnvironmentVars contains the environment variables on the kube-controllers that influenced the RunningConfig. type: object runningConfig: description: RunningConfig contains the effective config that is running in the kube-controllers pod, after merging the API resource with any environment variables. properties: controllers: description: Controllers enables and configures individual Kubernetes controllers properties: namespace: description: Namespace enables and configures the namespace controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object node: description: Node enables and configures the node controller. Enabled by default, set to nil to disable. properties: hostEndpoint: description: HostEndpoint controls syncing nodes to host endpoints. Disabled by default, set to nil to disable. properties: autoCreate: description: 'AutoCreate enables automatic creation of host endpoints for every node. [Default: Disabled]' type: string type: object reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string syncLabels: description: 'SyncLabels controls whether to copy Kubernetes node labels to Calico nodes. [Default: Enabled]' type: string type: object policy: description: Policy enables and configures the policy controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object serviceAccount: description: ServiceAccount enables and configures the service account controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object workloadEndpoint: description: WorkloadEndpoint enables and configures the workload endpoint controller. Enabled by default, set to nil to disable. properties: reconcilerPeriod: description: 'ReconcilerPeriod is the period to perform reconciliation with the Calico datastore. [Default: 5m]' type: string type: object type: object etcdV3CompactionPeriod: description: 'EtcdV3CompactionPeriod is the period between etcdv3 compaction requests. Set to 0 to disable. [Default: 10m]' type: string healthChecks: description: 'HealthChecks enables or disables support for health checks [Default: Enabled]' type: string logSeverityScreen: description: 'LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]' type: string prometheusMetricsPort: description: 'PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. Set to 0 to disable. [Default: 9094]' type: integer required: - controllers type: object type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: networkpolicies.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: NetworkPolicy listKind: NetworkPolicyList plural: networkpolicies singular: networkpolicy scope: Namespaced versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: properties: egress: description: The ordered set of egress rules. Each rule contains a set of packet match criteria and a corresponding action to apply. items: description: "A Rule encapsulates a set of match criteria and an action. Both selector-based security Policy and security Profiles reference rules - separated out as a list of rules for both ingress and egress packet matching. \n Each positive match criteria has a negated version, prefixed with \"Not\". All the match criteria within a rule must be satisfied for a packet to match. A single rule can contain the positive and negative version of a match and both must be satisfied for the rule to match." properties: action: type: string destination: description: Destination contains the match criteria that apply to destination entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object http: description: HTTP contains match criteria that apply to HTTP requests. properties: methods: description: Methods is an optional field that restricts the rule to apply only to HTTP requests that use one of the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple methods are OR'd together. items: type: string type: array paths: description: 'Paths is an optional field that restricts the rule to apply to HTTP requests that use one of the listed HTTP Paths. Multiple paths are OR''d together. e.g: - exact: /foo - prefix: /bar NOTE: Each entry may ONLY specify either a `exact` or a `prefix` match. The validator will check for it.' items: description: 'HTTPPath specifies an HTTP path to match. It may be either of the form: exact: <path>: which matches the path exactly or prefix: <path-prefix>: which matches the path prefix' properties: exact: type: string prefix: type: string type: object type: array type: object icmp: description: ICMP is an optional field that restricts the rule to apply to a specific type and code of ICMP traffic. This should only be specified if the Protocol field is set to "ICMP" or "ICMPv6". properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object ipVersion: description: IPVersion is an optional field that restricts the rule to only match a specific IP version. type: integer metadata: description: Metadata contains additional information for this rule properties: annotations: additionalProperties: type: string description: Annotations is a set of key value pairs that give extra information about the rule type: object type: object notICMP: description: NotICMP is the negated version of the ICMP field. properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object notProtocol: anyOf: - type: integer - type: string description: NotProtocol is the negated version of the Protocol field. pattern: ^.* x-kubernetes-int-or-string: true protocol: anyOf: - type: integer - type: string description: "Protocol is an optional field that restricts the rule to only apply to traffic of a specific IP protocol. Required if any of the EntityRules contain Ports (because ports only apply to certain protocols). \n Must be one of these string values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\", \"UDPLite\" or an integer in the range 1-255." pattern: ^.* x-kubernetes-int-or-string: true source: description: Source contains the match criteria that apply to source entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object required: - action type: object type: array ingress: description: The ordered set of ingress rules. Each rule contains a set of packet match criteria and a corresponding action to apply. items: description: "A Rule encapsulates a set of match criteria and an action. Both selector-based security Policy and security Profiles reference rules - separated out as a list of rules for both ingress and egress packet matching. \n Each positive match criteria has a negated version, prefixed with \"Not\". All the match criteria within a rule must be satisfied for a packet to match. A single rule can contain the positive and negative version of a match and both must be satisfied for the rule to match." properties: action: type: string destination: description: Destination contains the match criteria that apply to destination entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object http: description: HTTP contains match criteria that apply to HTTP requests. properties: methods: description: Methods is an optional field that restricts the rule to apply only to HTTP requests that use one of the listed HTTP Methods (e.g. GET, PUT, etc.) Multiple methods are OR'd together. items: type: string type: array paths: description: 'Paths is an optional field that restricts the rule to apply to HTTP requests that use one of the listed HTTP Paths. Multiple paths are OR''d together. e.g: - exact: /foo - prefix: /bar NOTE: Each entry may ONLY specify either a `exact` or a `prefix` match. The validator will check for it.' items: description: 'HTTPPath specifies an HTTP path to match. It may be either of the form: exact: <path>: which matches the path exactly or prefix: <path-prefix>: which matches the path prefix' properties: exact: type: string prefix: type: string type: object type: array type: object icmp: description: ICMP is an optional field that restricts the rule to apply to a specific type and code of ICMP traffic. This should only be specified if the Protocol field is set to "ICMP" or "ICMPv6". properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object ipVersion: description: IPVersion is an optional field that restricts the rule to only match a specific IP version. type: integer metadata: description: Metadata contains additional information for this rule properties: annotations: additionalProperties: type: string description: Annotations is a set of key value pairs that give extra information about the rule type: object type: object notICMP: description: NotICMP is the negated version of the ICMP field. properties: code: description: Match on a specific ICMP code. If specified, the Type value must also be specified. This is a technical limitation imposed by the kernel's iptables firewall, which Calico uses to enforce the rule. type: integer type: description: Match on a specific ICMP type. For example a value of 8 refers to ICMP Echo Request (i.e. pings). type: integer type: object notProtocol: anyOf: - type: integer - type: string description: NotProtocol is the negated version of the Protocol field. pattern: ^.* x-kubernetes-int-or-string: true protocol: anyOf: - type: integer - type: string description: "Protocol is an optional field that restricts the rule to only apply to traffic of a specific IP protocol. Required if any of the EntityRules contain Ports (because ports only apply to certain protocols). \n Must be one of these string values: \"TCP\", \"UDP\", \"ICMP\", \"ICMPv6\", \"SCTP\", \"UDPLite\" or an integer in the range 1-255." pattern: ^.* x-kubernetes-int-or-string: true source: description: Source contains the match criteria that apply to source entity. properties: namespaceSelector: description: "NamespaceSelector is an optional field that contains a selector expression. Only traffic that originates from (or terminates at) endpoints within the selected namespaces will be matched. When both NamespaceSelector and Selector are defined on the same rule, then only workload endpoints that are matched by both selectors will be selected by the rule. \n For NetworkPolicy, an empty NamespaceSelector implies that the Selector is limited to selecting only workload endpoints in the same namespace as the NetworkPolicy. \n For NetworkPolicy, `global()` NamespaceSelector implies that the Selector is limited to selecting only GlobalNetworkSet or HostEndpoint. \n For GlobalNetworkPolicy, an empty NamespaceSelector implies the Selector applies to workload endpoints across all namespaces." type: string nets: description: Nets is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) IP addresses in any of the given subnets. items: type: string type: array notNets: description: NotNets is the negated version of the Nets field. items: type: string type: array notPorts: description: NotPorts is the negated version of the Ports field. Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to "TCP" or "UDP". items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array notSelector: description: NotSelector is the negated version of the Selector field. See Selector field for subtleties with negated selectors. type: string ports: description: "Ports is an optional field that restricts the rule to only apply to traffic that has a source (destination) port that matches one of these ranges/values. This value is a list of integers or strings that represent ranges of ports. \n Since only some protocols have ports, if any ports are specified it requires the Protocol match in the Rule to be set to \"TCP\" or \"UDP\"." items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array selector: description: "Selector is an optional field that contains a selector expression (see Policy for sample syntax). \ Only traffic that originates from (terminates at) endpoints matching the selector will be matched. \n Note that: in addition to the negated version of the Selector (see NotSelector below), the selector expression syntax itself supports negation. The two types of negation are subtly different. One negates the set of matched endpoints, the other negates the whole match: \n \tSelector = \"!has(my_label)\" matches packets that are from other Calico-controlled \tendpoints that do not have the label \"my_label\". \n \tNotSelector = \"has(my_label)\" matches packets that are not from Calico-controlled \tendpoints that do have the label \"my_label\". \n The effect is that the latter will accept packets from non-Calico sources whereas the former is limited to packets from Calico-controlled endpoints." type: string serviceAccounts: description: ServiceAccounts is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a matching service account. properties: names: description: Names is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account whose name is in the list. items: type: string type: array selector: description: Selector is an optional field that restricts the rule to only apply to traffic that originates from (or terminates at) a pod running as a service account that matches the given label selector. If both Names and Selector are specified then they are AND'ed. type: string type: object type: object required: - action type: object type: array order: description: Order is an optional field that specifies the order in which the policy is applied. Policies with higher "order" are applied after those with lower order. If the order is omitted, it may be considered to be "infinite" - i.e. the policy will be applied last. Policies with identical order will be applied in alphanumerical order based on the Policy "Name". type: number selector: description: "The selector is an expression used to pick pick out the endpoints that the policy should be applied to. \n Selector expressions follow this syntax: \n \tlabel == \"string_literal\" \ -> comparison, e.g. my_label == \"foo bar\" \tlabel != \"string_literal\" \ -> not equal; also matches if label is not present \tlabel in { \"a\", \"b\", \"c\", ... } -> true if the value of label X is one of \"a\", \"b\", \"c\" \tlabel not in { \"a\", \"b\", \"c\", ... } -> true if the value of label X is not one of \"a\", \"b\", \"c\" \thas(label_name) -> True if that label is present \t! expr -> negation of expr \texpr && expr -> Short-circuit and \texpr || expr -> Short-circuit or \t( expr ) -> parens for grouping \tall() or the empty selector -> matches all endpoints. \n Label names are allowed to contain alphanumerics, -, _ and /. String literals are more permissive but they do not support escape characters. \n Examples (with made-up labels): \n \ttype == \"webserver\" && deployment == \"prod\" \ttype in {\"frontend\", \"backend\"} \tdeployment != \"dev\" \t! has(label_name)" type: string serviceAccountSelector: description: ServiceAccountSelector is an optional field for an expression used to select a pod based on service accounts. type: string types: description: "Types indicates whether this policy applies to ingress, or to egress, or to both. When not explicitly specified (and so the value on creation is empty or nil), Calico defaults Types according to what Ingress and Egress are present in the policy. The default is: \n - [ PolicyTypeIngress ], if there are no Egress rules (including the case where there are also no Ingress rules) \n - [ PolicyTypeEgress ], if there are Egress rules but no Ingress rules \n - [ PolicyTypeIngress, PolicyTypeEgress ], if there are both Ingress and Egress rules. \n When the policy is read back again, Types will always be one of these values, never empty or nil." items: description: PolicyType enumerates the possible values of the PolicySpec Types field. type: string type: array type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: networksets.crd.projectcalico.org spec: group: crd.projectcalico.org names: kind: NetworkSet listKind: NetworkSetList plural: networksets singular: networkset scope: Namespaced versions: - name: v1 schema: openAPIV3Schema: description: NetworkSet is the Namespaced-equivalent of the GlobalNetworkSet. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: NetworkSetSpec contains the specification for a NetworkSet resource. properties: nets: description: The list of IP networks that belong to this set. items: type: string type: array type: object type: object served: true storage: true status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- --- # Source: calico/templates/calico-kube-controllers-rbac.yaml # Include a clusterrole for the kube-controllers component, # and bind it to the calico-kube-controllers serviceaccount. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: calico-kube-controllers rules: # Nodes are watched to monitor for deletions. - apiGroups: [""] resources: - nodes verbs: - watch - list - get # Pods are queried to check for existence. - apiGroups: [""] resources: - pods verbs: - get # IPAM resources are manipulated when nodes are deleted. - apiGroups: ["crd.projectcalico.org"] resources: - ippools verbs: - list - apiGroups: ["crd.projectcalico.org"] resources: - blockaffinities - ipamblocks - ipamhandles verbs: - get - list - create - update - delete - watch # kube-controllers manages hostendpoints. - apiGroups: ["crd.projectcalico.org"] resources: - hostendpoints verbs: - get - list - create - update - delete # Needs access to update clusterinformations. - apiGroups: ["crd.projectcalico.org"] resources: - clusterinformations verbs: - get - create - update # KubeControllersConfiguration is where it gets its config - apiGroups: ["crd.projectcalico.org"] resources: - kubecontrollersconfigurations verbs: # read its own config - get # create a default if none exists - create # update status - update # watch for changes - watch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: calico-kube-controllers roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: calico-kube-controllers subjects: - kind: ServiceAccount name: calico-kube-controllers namespace: kube-system --- --- # Source: calico/templates/calico-node-rbac.yaml # Include a clusterrole for the calico-node DaemonSet, # and bind it to the calico-node serviceaccount. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: calico-node rules: # The CNI plugin needs to get pods, nodes, and namespaces. - apiGroups: [""] resources: - pods - nodes - namespaces verbs: - get - apiGroups: [""] resources: - endpoints - services verbs: # Used to discover service IPs for advertisement. - watch - list # Used to discover Typhas. - get # Pod CIDR auto-detection on kubeadm needs access to config maps. - apiGroups: [""] resources: - configmaps verbs: - get - apiGroups: [""] resources: - nodes/status verbs: # Needed for clearing NodeNetworkUnavailable flag. - patch # Calico stores some configuration information in node annotations. - update # Watch for changes to Kubernetes NetworkPolicies. - apiGroups: ["networking.k8s.io"] resources: - networkpolicies verbs: - watch - list # Used by Calico for policy information. - apiGroups: [""] resources: - pods - namespaces - serviceaccounts verbs: - list - watch # The CNI plugin patches pods/status. - apiGroups: [""] resources: - pods/status verbs: - patch # Calico monitors various CRDs for config. - apiGroups: ["crd.projectcalico.org"] resources: - globalfelixconfigs - felixconfigurations - bgppeers - globalbgpconfigs - bgpconfigurations - ippools - ipamblocks - globalnetworkpolicies - globalnetworksets - networkpolicies - networksets - clusterinformations - hostendpoints - blockaffinities verbs: - get - list - watch # Calico must create and update some CRDs on startup. - apiGroups: ["crd.projectcalico.org"] resources: - ippools - felixconfigurations - clusterinformations verbs: - create - update # Calico stores some configuration information on the node. - apiGroups: [""] resources: - nodes verbs: - get - list - watch # These permissions are only required for upgrade from v2.6, and can # be removed after upgrade or on fresh installations. - apiGroups: ["crd.projectcalico.org"] resources: - bgpconfigurations - bgppeers verbs: - create - update # These permissions are required for Calico CNI to perform IPAM allocations. - apiGroups: ["crd.projectcalico.org"] resources: - blockaffinities - ipamblocks - ipamhandles verbs: - get - list - create - update - delete - apiGroups: ["crd.projectcalico.org"] resources: - ipamconfigs verbs: - get # Block affinities must also be watchable by confd for route aggregation. - apiGroups: ["crd.projectcalico.org"] resources: - blockaffinities verbs: - watch # The Calico IPAM migration needs to get daemonsets. These permissions can be # removed if not upgrading from an installation using host-local IPAM. - apiGroups: ["apps"] resources: - daemonsets verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: calico-node roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: calico-node subjects: - kind: ServiceAccount name: calico-node namespace: kube-system --- # Source: calico/templates/calico-node.yaml # This manifest installs the calico-node container, as well # as the CNI plugins and network config on # each master and worker node in a Kubernetes cluster. kind: DaemonSet apiVersion: apps/v1 metadata: name: calico-node namespace: kube-system labels: k8s-app: calico-node spec: selector: matchLabels: k8s-app: calico-node updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 template: metadata: labels: k8s-app: calico-node spec: nodeSelector: kubernetes.io/os: linux hostNetwork: true tolerations: # Make sure calico-node gets scheduled on all nodes. - effect: NoSchedule operator: Exists # Mark the pod as a critical add-on for rescheduling. - key: CriticalAddonsOnly operator: Exists - effect: NoExecute operator: Exists serviceAccountName: calico-node # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods. terminationGracePeriodSeconds: 0 priorityClassName: system-node-critical initContainers: # This container performs upgrade from host-local IPAM to calico-ipam. # It can be deleted if this is a fresh installation, or if you have already # upgraded to use calico-ipam. - name: upgrade-ipam image: docker.io/calico/cni:v3.19.4 command: ["/opt/cni/bin/calico-ipam", "-upgrade"] envFrom: - configMapRef: # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode. name: kubernetes-services-endpoint optional: true env: - name: KUBERNETES_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: CALICO_NETWORKING_BACKEND valueFrom: configMapKeyRef: name: calico-config key: calico_backend volumeMounts: - mountPath: /var/lib/cni/networks name: host-local-net-dir - mountPath: /host/opt/cni/bin name: cni-bin-dir securityContext: privileged: true # This container installs the CNI binaries # and CNI network config file on each node. - name: install-cni image: docker.io/calico/cni:v3.19.4 command: ["/opt/cni/bin/install"] envFrom: - configMapRef: # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode. name: kubernetes-services-endpoint optional: true env: # Name of the CNI config file to create. - name: CNI_CONF_NAME value: "10-calico.conflist" # The CNI network config to install on each node. - name: CNI_NETWORK_CONFIG valueFrom: configMapKeyRef: name: calico-config key: cni_network_config # Set the hostname based on the k8s node name. - name: KUBERNETES_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName # CNI MTU Config variable - name: CNI_MTU valueFrom: configMapKeyRef: name: calico-config key: veth_mtu # Prevents the container from sleeping forever. - name: SLEEP value: "false" volumeMounts: - mountPath: /host/opt/cni/bin name: cni-bin-dir - mountPath: /host/etc/cni/net.d name: cni-net-dir securityContext: privileged: true # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes # to communicate with Felix over the Policy Sync API. - name: flexvol-driver image: docker.io/calico/pod2daemon-flexvol:v3.19.4 volumeMounts: - name: flexvol-driver-host mountPath: /host/driver securityContext: privileged: true containers: # Runs calico-node container on each Kubernetes node. This # container programs network policy and routes on each # host. - name: calico-node image: docker.io/calico/node:v3.19.4 envFrom: - configMapRef: # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode. name: kubernetes-services-endpoint optional: true env: # Use Kubernetes API as the backing datastore. - name: DATASTORE_TYPE value: "kubernetes" # Wait for the datastore. - name: WAIT_FOR_DATASTORE value: "true" # Set based on the k8s node name. - name: NODENAME valueFrom: fieldRef: fieldPath: spec.nodeName # Choose the backend to use. - name: CALICO_NETWORKING_BACKEND valueFrom: configMapKeyRef: name: calico-config key: calico_backend # Cluster type to identify the deployment type - name: CLUSTER_TYPE value: "k8s,bgp" # Auto-detect the BGP IP address. - name: IP value: "autodetect" # Enable IPIP - name: CALICO_IPV4POOL_IPIP value: "Always" # Enable or Disable VXLAN on the default IP pool. - name: CALICO_IPV4POOL_VXLAN value: "Never" # Set MTU for tunnel device used if ipip is enabled - name: FELIX_IPINIPMTU valueFrom: configMapKeyRef: name: calico-config key: veth_mtu # Set MTU for the VXLAN tunnel device. - name: FELIX_VXLANMTU valueFrom: configMapKeyRef: name: calico-config key: veth_mtu # Set MTU for the Wireguard tunnel device. - name: FELIX_WIREGUARDMTU valueFrom: configMapKeyRef: name: calico-config key: veth_mtu # The default IPv4 pool to create on startup if none exists. Pod IPs will be # chosen from this range. Changing this value after installation will have # no effect. This should fall within `--cluster-cidr`. # - name: CALICO_IPV4POOL_CIDR # value: "192.168.0.0/16" # Disable file logging so `kubectl logs` works. - name: CALICO_DISABLE_FILE_LOGGING value: "true" # Set Felix endpoint to host default action to ACCEPT. - name: FELIX_DEFAULTENDPOINTTOHOSTACTION value: "ACCEPT" # Disable IPv6 on Kubernetes. - name: FELIX_IPV6SUPPORT value: "false" - name: FELIX_HEALTHENABLED value: "true" securityContext: privileged: true resources: requests: cpu: 250m livenessProbe: exec: command: - /bin/calico-node - -felix-live - -bird-live periodSeconds: 10 initialDelaySeconds: 10 failureThreshold: 6 readinessProbe: exec: command: - /bin/calico-node - -felix-ready - -bird-ready periodSeconds: 10 volumeMounts: - mountPath: /lib/modules name: lib-modules readOnly: true - mountPath: /run/xtables.lock name: xtables-lock readOnly: false - mountPath: /var/run/calico name: var-run-calico readOnly: false - mountPath: /var/lib/calico name: var-lib-calico readOnly: false - name: policysync mountPath: /var/run/nodeagent # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the # parent directory. - name: sysfs mountPath: /sys/fs/ # Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host. # If the host is known to mount that filesystem already then Bidirectional can be omitted. mountPropagation: Bidirectional - name: cni-log-dir mountPath: /var/log/calico/cni readOnly: true volumes: # Used by calico-node. - name: lib-modules hostPath: path: /lib/modules - name: var-run-calico hostPath: path: /var/run/calico - name: var-lib-calico hostPath: path: /var/lib/calico - name: xtables-lock hostPath: path: /run/xtables.lock type: FileOrCreate - name: sysfs hostPath: path: /sys/fs/ type: DirectoryOrCreate # Used to install CNI. - name: cni-bin-dir hostPath: path: /opt/cni/bin - name: cni-net-dir hostPath: path: /etc/cni/net.d # Used to access CNI logs. - name: cni-log-dir hostPath: path: /var/log/calico/cni # Mount in the directory for host-local IPAM allocations. This is # used when upgrading from host-local to calico-ipam, and can be removed # if not using the upgrade-ipam init container. - name: host-local-net-dir hostPath: path: /var/lib/cni/networks # Used to create per-pod Unix Domain Sockets - name: policysync hostPath: type: DirectoryOrCreate path: /var/run/nodeagent # Used to install Flex Volume Driver - name: flexvol-driver-host hostPath: type: DirectoryOrCreate path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds --- apiVersion: v1 kind: ServiceAccount metadata: name: calico-node namespace: kube-system --- # Source: calico/templates/calico-kube-controllers.yaml # See https://github.com/projectcalico/kube-controllers apiVersion: apps/v1 kind: Deployment metadata: name: calico-kube-controllers namespace: kube-system labels: k8s-app: calico-kube-controllers spec: # The controllers can only have a single active instance. replicas: 1 selector: matchLabels: k8s-app: calico-kube-controllers strategy: type: Recreate template: metadata: name: calico-kube-controllers namespace: kube-system labels: k8s-app: calico-kube-controllers spec: nodeSelector: kubernetes.io/os: linux tolerations: # Mark the pod as a critical add-on for rescheduling. - key: CriticalAddonsOnly operator: Exists - key: node-role.kubernetes.io/master effect: NoSchedule serviceAccountName: calico-kube-controllers priorityClassName: system-cluster-critical containers: - name: calico-kube-controllers image: docker.io/calico/kube-controllers:v3.19.4 env: # Choose which controllers to run. - name: ENABLED_CONTROLLERS value: node - name: DATASTORE_TYPE value: kubernetes livenessProbe: exec: command: - /usr/bin/check-status - -l periodSeconds: 10 initialDelaySeconds: 10 failureThreshold: 6 readinessProbe: exec: command: - /usr/bin/check-status - -r periodSeconds: 10 --- apiVersion: v1 kind: ServiceAccount metadata: name: calico-kube-controllers namespace: kube-system --- # This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: calico-kube-controllers namespace: kube-system labels: k8s-app: calico-kube-controllers spec: maxUnavailable: 1 selector: matchLabels: k8s-app: calico-kube-controllers --- # Source: calico/templates/calico-etcd-secrets.yaml --- # Source: calico/templates/calico-typha.yaml --- # Source: calico/templates/configure-canal.yaml
3、部署calico
kubectl apply -f /usr/local/k8s/calico/calico.yaml
4、查看
]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-node1 Ready <none> 24h v1.20.10 ]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-848c5d445f-j7n4d 1/1 Running 0 9m25s kube-system calico-node-nc68f 1/1 Running 0 9m37s
12.2、部署CoreDNS
- 官方：https://github.com/coredns/coredns
- 官方提供的模板：https://github.com/coredns/deployment/blob/master/kubernetes/coredns.yaml.sed
1、修改官方提供的模板
- 下载的yaml需要替换如下字段
CLUSTER_DNS_IP: 172.22.0.2 #修改成kubelet-config.json中设置的clusterDNS地址 CLUSTER_DOMAIN: cluster.local REVERSE_CIDRS: in-addr.arpa ip6.arpa STUBDOMAINS: 无 UPSTREAMNAMESERVER: /etc/resolv.conf coredns/coredns:1.8.0 #镜像
- 修改后的yaml文件（coredns.yaml）
apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/name: "CoreDNS" spec: # replicas: not specified here: # 1. Default is 1. # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on. strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: priorityClassName: system-cluster-critical serviceAccountName: coredns tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname containers: - name: coredns image: coredns/coredns:1.8.0 imagePullPolicy: IfNotPresent resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile --- apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: kube-dns clusterIP: 172.22.0.2 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP
- 修改前的yaml文件（coredns.yaml.sed）
apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes CLUSTER_DOMAIN REVERSE_CIDRS { fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . UPSTREAMNAMESERVER { max_concurrent 1000 } cache 30 loop reload loadbalance }STUBDOMAINS --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/name: "CoreDNS" spec: # replicas: not specified here: # 1. Default is 1. # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on. strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: priorityClassName: system-cluster-critical serviceAccountName: coredns tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname containers: - name: coredns image: coredns/coredns:1.9.0 imagePullPolicy: IfNotPresent resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile --- apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: kube-dns clusterIP: CLUSTER_DNS_IP ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP
2、部署CoreDNS
kubectl apply -f coredns.yaml
13、验证Kubernetes集群是否安装完成
- kubeadm在Master上也安装了kubelet，在默认情况下master并不参与工作负载。如果希望master也参与工作负载，则可以执行下面的命令（删除Node的Label“node-role.kubernetes.io/master”），让Master也作为一个Node：
kubectl taint nodes --all node-role.kubernetes.io/master
13.1、等待所有节点和pod正常启动
#查看各节点的状态是否正常 ]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-node1 Ready <none> 125m v1.20.14 k8s-node2 Ready <none> 3m36s v1.20.14 #验证Kubernetes集群的相关Pod是否都创建并正常运行： ]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-848c5d445f-mbh6j 1/1 Running 0 60m kube-system calico-node-ncsm2 1/1 Running 0 3m37s kube-system calico-node-xvncs 1/1 Running 0 60m kube-system coredns-786747cbf9-md4v9 1/1 Running 0 36m
13.2、验证网络插件是否正常
- busybox的pod的yaml文件（test-busybox.yaml）
- 注意，busybox的镜像使用1.28的，不是latest，否者测试coredns会出现不正常的情况
apiVersion: v1 kind: Service metadata: name: test-busybox-svc namespace: default labels: app: test-busybox-svc spec: type: NodePort ports: - port: 54321 targetPort: 54321 nodePort: 54321 protocol: TCP name: http selector: app: test-busybox-pod --- apiVersion: apps/v1 kind: Deployment metadata: name: test-busybox-deployment namespace: default spec: selector: matchLabels: app: test-busybox-pod replicas: 2 template: metadata: labels: app: test-busybox-pod spec: imagePullSecrets: - name: svharborwx containers: - name: test-busybox-container image: busybox:1.28 imagePullPolicy: IfNotPresent command: ["/bin/sleep", "10000000000"] ports: - containerPort: 54321
- 应用test-busybox.yaml文件启动pod
]# kubectl apply -f test-busybox.yaml ]# kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES default test-busybox-deployment-5dd4f8dc7f-gwrvz 1/1 Running 0 28s 172.21.36.68 k8s-node1 <none> <none> default test-busybox-deployment-5dd4f8dc7f-nrnlb 1/1 Running 0 29s 172.21.169.129 k8s-node2 <none> <none>
1、测试Calico是否正常
]# kubectl exec test-busybox-deployment-5dd4f8dc7f-gwrvz -it -- /bin/sh / # ping -c 2 172.21.169.129 PING 172.21.169.129 (172.21.169.129): 56 data bytes 64 bytes from 172.21.169.129: seq=0 ttl=62 time=2.849 ms 64 bytes from 172.21.169.129: seq=1 ttl=62 time=0.529 ms --- 172.21.169.129 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.529/1.689/2.849 ms / # ping -c 2 www.baidu.com PING www.baidu.com (110.242.68.4): 56 data bytes 64 bytes from 110.242.68.4: seq=0 ttl=127 time=66.047 ms 64 bytes from 110.242.68.4: seq=1 ttl=127 time=61.821 ms --- www.baidu.com ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 61.821/63.934/66.047 ms / # exit
2、测试coredns是否正常
- 查看服务名称及其IP地址
]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 172.22.0.1 <none> 443/TCP 8h test-busybox-svc NodePort 172.22.185.54 <none> 54321:54321/TCP 5m43s
- 测试coredns
]# kubectl exec test-busybox-deployment-5dd4f8dc7f-gwrvz -it -- /bin/sh / # nslookup test-busybox-svc.default.svc.cluster.local Server: 172.22.0.2 Address 1: 172.22.0.2 kube-dns.kube-system.svc.cluster.local Name: test-busybox-svc.default.svc.cluster.local Address 1: 172.22.185.54 test-busybox-svc.default.svc.cluster.local / # nslookup www.baidu.com Server: 172.22.0.2 Address 1: 172.22.0.2 kube-dns.kube-system.svc.cluster.local Name: www.baidu.com Address 1: 110.242.68.4 Address 2: 110.242.68.3 / # exit
14、部署Dashboard
1、下载yaml文件，并部署
- 注意，Dashboard与kubernetes版本之间的兼容性，可以通过https://github.com/kubernetes/dashboard/releases获取。
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml kubectl apply -f recommended.yaml
- recommended.yaml文件
View Code-
查看Pod和Service资源
[root@k8s-master ~]# kubectl get pods -n kubernetes-dashboard -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES dashboard-metrics-scraper-5b8896d7fc-ggqb2 1/1 Running 0 4m28s 172.20.85.193 k8s-node01 <none> <none> kubernetes-dashboard-cb988587b-2d5tt 1/1 Running 0 4m28s 172.20.58.193 k8s-node02 <none> <none> [root@k8s-master ~]# kubectl get svc -n kubernetes-dashboard -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR dashboard-metrics-scraper ClusterIP 172.16.0.32 <none> 8000/TCP 4m38s k8s-app=dashboard-metrics-scraper kubernetes-dashboard ClusterIP 172.16.0.181 <none> 443/TCP 4m39s k8s-app=kubernetes-dashboard
2、使用NodeIP访问
- 修改Dashboard，暴露NodePort。
- 访问地址：https://NodeIP:30001
kubectl patch svc kubernetes-dashboard \
-n kubernetes-dashboard \
-p '{"spec":{"type":"NodePort","ports":[{"port":443,"targetPort":8443,"nodePort":30001}]}}'
- 创建service account并绑定默认cluster-admin集群管理员角色
- 使用输出的token登录Dashboard
--创建一个角色
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
--查看token
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}'
1
# #
