calico.yml详解
文章目录
前言
k8s创建calico网络见文档： 《calico网络安装和删除》
1. 概述
1.1 ulr
官方下载地址如下：
#wget https://docs.projectcalico.org/manifests/calico.yaml
1.2 所需对象概述
主要创建calico-node和calico-kube-controllers两个服务。需要创建如下资源：
- ConfigMap
calico-config
作用：Calico所需的配置及CNI网络配置 - DaemoSet
Clico-Node
作用：初始化node节点的网络，保证pod节点的网络互通。 - Deployment
calico-kube-controllers
作用：理k8s集群中网络策略 - RBAC规则
- CRD（自定义资源）
2. ConfigMap
作用：Calico所需的配置及CNI网络配置
kind: ConfigMap
apiVersion: v1
metadata:
name: calico-config
namespace: kube-system
data:
# Typha is disabled.
typha_service_name: "calico-typha"
# Configure the backend to use.
calico_backend: "bird"
# Configure the MTU to use for workload interfaces and tunnels.
# By default, MTU is auto-detected, and explicitly setting this field should not be required.
# You can override auto-detection by providing a non-zero value.
veth_mtu: "0"
# The CNI network configuration to install on each node. The special
# values in this config will be automatically populated.
cni_network_config: |-
{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "calico",
"log_level": "info",
"log_file_path": "/var/log/calico/cni/cni.log",
"datastore_type": "kubernetes",
"nodename": "__KUBERNETES_NODE_NAME__",
"mtu": __CNI_MTU__,
"ipam": {
"type": "calico-ipam"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "__KUBECONFIG_FILEPATH__"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
},
{
"type": "bandwidth",
"capabilities": {"bandwidth": true}
}
]
}
typha_service_name
typha模式，k8s数据存储模式超过50各节点推荐启用typha。
Typha组件可以帮助Calico扩展到大量的节点，而不会对Kubernetes API服务器造成过度的影响。
修改typha_service_name "none"改为"calico-typha"veth_mtu
网络接口的MTU（Maximum Transmission Unit ）值，即最大传输单元，它的单位是字节）calico_backend
calico的后端，默认为"bird"。cni_network_config
符合CNI规范的网络配置。
将在 /etc/cni/net.d目录下生成CNI网络配置文件。
[root@calico-master ~]# ll /etc/cni/net.d/
总用量 8
-rw-r--r-- 1 root root 664 7月 14 15:32 10-calico.conflist
-rw------- 1 root root 2778 7月 14 15:32 calico-kubeconfig
"type": "calico"表示kubelet将在/opt/cni/bin 目录下搜索名为calico的可执行文件并调用它完成容器网络设置。
[root@calico-master ~]# ll /opt/cni/bin/
总用量 181900
-rwxr-xr-x 1 root root 4159518 7月 14 15:32 bandwidth
-rwxr-xr-x 1 root root 3581192 9月 10 2020 bridge
-rwxr-xr-x 1 root root 41472000 7月 14 15:32 calico
-rwxr-xr-x 1 root root 41472000 7月 14 15:32 calico-ipam
-rwxr-xr-x 1 root root 9837552 9月 10 2020 dhcp
-rwxr-xr-x 1 root root 4699824 9月 10 2020 firewall
-rwxr-xr-x 1 root root 3069556 7月 14 15:32 flannel
-rwxr-xr-x 1 root root 3274160 9月 10 2020 host-device
-rwxr-xr-x 1 root root 3614480 7月 14 15:32 host-local
-rwxr-xr-x 1 root root 41472000 7月 14 15:32 install
-rwxr-xr-x 1 root root 3377272 9月 10 2020 ipvlan
-rwxr-xr-x 1 root root 3209463 7月 14 15:32 loopback
-rwxr-xr-x 1 root root 3440168 9月 10 2020 macvlan
-rwxr-xr-x 1 root root 3939867 7月 14 15:32 portmap
-rwxr-xr-x 1 root root 3528800 9月 10 2020 ptp
-rwxr-xr-x 1 root root 2849328 9月 10 2020 sbr
-rwxr-xr-x 1 root root 2503512 9月 10 2020 static
-rwxr-xr-x 1 root root 3356587 7月 14 15:32 tuning
-rwxr-xr-x 1 root root 3377120 9月 10 2020 vlan
3. Clico-Node
3.1 Pod和容器
以DaemonSet形式在每个Node上运行一个calico-node的Pod
- 包含三个Init Containers:
分别为：upgrade-ipam、install-cni、flexvor-driver
用describe查看该pod，结果如下：
Controlled By: DaemonSet/calico-node
upgrade-ipam:
Container ID: docker://5896e83786f452577a7cd4b8a9e9a10c2f7f4950a5072af61f9f3ea7922232cc
Image: docker.io/calico/cni:v3.19.1
……
install-cni:
Container ID: docker://86fd9456c263746a61c43d519c01abe4e75f42cbd9976d985bd06d22d1733ebc
Image: docker.io/calico/cni:v3.19.1
……
flexvol-driver:
Container ID: docker://3ab9f73561d95e90ef00c9b19f3276c030da447c09501b013f558189ea8e8ff5
Image: docker.io/calico/pod2daemon-flexvol:v3.19.1
……
- 主容器
calico-node
作用：管理Pod的网络配置，保证Pod网络与各node的互通
Containers:
calico-node:
Container ID: docker://222e79bf84bb3c58da0fe6ff0e71b14d537c68c39e49d1cd44843bafcfb90e46
Image: docker.io/calico/node:v3.19.1
3.2 yml文件
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: calico-node
namespace: kube-system
labels:
k8s-app: calico-node
spec:
selector:
matchLabels:
k8s-app: calico-node
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
k8s-app: calico-node
spec:
nodeSelector:
kubernetes.io/os: linux
hostNetwork: true
tolerations:
# Make sure calico-node gets scheduled on all nodes.
- effect: NoSchedule
operator: Exists
# Mark the pod as a critical add-on for rescheduling.
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
operator: Exists
serviceAccountName: calico-node
# Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
# deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
terminationGracePeriodSeconds: 0
priorityClassName: system-node-critical
initContainers:
# This container performs upgrade from host-local IPAM to calico-ipam.
# It can be deleted if this is a fresh installation, or if you have already
# upgraded to use calico-ipam.
- name: upgrade-ipam
image: docker.io/calico/cni:v3.19.1
command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
- name: KUBERNETES_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CALICO_NETWORKING_BACKEND
valueFrom:
configMapKeyRef:
name: calico-config
key: calico_backend
volumeMounts:
- mountPath: /var/lib/cni/networks
name: host-local-net-dir
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
securityContext:
privileged: true
# This container installs the CNI binaries
# and CNI network config file on each node.
- name: install-cni
image: docker.io/calico/cni:v3.19.1
command: ["/opt/cni/bin/install"]
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
# Name of the CNI config file to create.
- name: CNI_CONF_NAME
value: "10-calico.conflist"
# The CNI network config to install on each node.
- name: CNI_NETWORK_CONFIG
valueFrom:
configMapKeyRef:
name: calico-config
key: cni_network_config
# Set the hostname based on the k8s node name.
- name: KUBERNETES_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# CNI MTU Config variable
- name: CNI_MTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Prevents the container from sleeping forever.
- name: SLEEP
value: "false"
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
- mountPath: /host/etc/cni/net.d
name: cni-net-dir
securityContext:
privileged: true
# Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
# to communicate with Felix over the Policy Sync API.
- name: flexvol-driver
image: docker.io/calico/pod2daemon-flexvol:v3.19.1
volumeMounts:
- name: flexvol-driver-host
mountPath: /host/driver
securityContext:
privileged: true
containers:
# Runs calico-node container on each Kubernetes node. This
# container programs network policy and routes on each
# host.
- name: calico-node
image: docker.io/calico/node:v3.19.1
envFrom:
- configMapRef:
# Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
name: kubernetes-services-endpoint
optional: true
env:
# Use Kubernetes API as the backing datastore.
- name: DATASTORE_TYPE
value: "kubernetes"
# Wait for the datastore.
- name: WAIT_FOR_DATASTORE
value: "true"
# Set based on the k8s node name.
- name: NODENAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# Choose the backend to use.
- name: CALICO_NETWORKING_BACKEND
valueFrom:
configMapKeyRef:
name: calico-config
key: calico_backend
# Cluster type to identify the deployment type
- name: CLUSTER_TYPE
value: "k8s,bgp"
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
#value: "Always"
value: "off"
#value: "Always"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
value: "Never"
# Set MTU for tunnel device used if ipip is enabled
- name: FELIX_IPINIPMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Set MTU for the VXLAN tunnel device.
- name: FELIX_VXLANMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# Set MTU for the Wireguard tunnel device.
- name: FELIX_WIREGUARDMTU
valueFrom:
configMapKeyRef:
name: calico-config
key: veth_mtu
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"
# Disable file logging so `kubectl logs` works.
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"
# Set Felix endpoint to host default action to ACCEPT.
- name: FELIX_DEFAULTENDPOINTTOHOSTACTION
value: "ACCEPT"
# Disable IPv6 on Kubernetes.
- name: FELIX_IPV6SUPPORT
value: "false"
- name: FELIX_HEALTHENABLED
value: "true"
securityContext:
privileged: true
resources:
requests:
cpu: 250m
livenessProbe:
exec:
command:
- /bin/calico-node
- -felix-live
- -bird-live
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
readinessProbe:
exec:
command:
- /bin/calico-node
- -felix-ready
- -bird-ready
periodSeconds: 10
volumeMounts:
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /run/xtables.lock
name: xtables-lock
readOnly: false
- mountPath: /var/run/calico
name: var-run-calico
readOnly: false
- mountPath: /var/lib/calico
name: var-lib-calico
readOnly: false
- name: policysync
mountPath: /var/run/nodeagent
# For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
# parent directory.
- name: sysfs
mountPath: /sys/fs/
# Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
# If the host is known to mount that filesystem already then Bidirectional can be omitted.
mountPropagation: Bidirectional
- name: cni-log-dir
mountPath: /var/log/calico/cni
readOnly: true
volumes:
# Used by calico-node.
- name: lib-modules
hostPath:
path: /lib/modules
- name: var-run-calico
hostPath:
path: /var/run/calico
- name: var-lib-calico
hostPath:
path: /var/lib/calico
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
- name: sysfs
hostPath:
path: /sys/fs/
type: DirectoryOrCreate
# Used to install CNI.
- name: cni-bin-dir
hostPath:
path: /opt/cni/bin
- name: cni-net-dir
hostPath:
path: /etc/cni/net.d
# Used to access CNI logs.
- name: cni-log-dir
hostPath:
path: /var/log/calico/cni
# Mount in the directory for host-local IPAM allocations. This is
# used when upgrading from host-local to calico-ipam, and can be removed
# if not using the upgrade-ipam init container.
- name: host-local-net-dir
hostPath:
path: /var/lib/cni/networks
# Used to create per-pod Unix Domain Sockets
- name: policysync
hostPath:
type: DirectoryOrCreate
path: /var/run/nodeagent
# Used to install Flex Volume Driver
- name: flexvol-driver-host
hostPath:
type: DirectoryOrCreate
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
3.3 常用配置说明
DATASTORE_TYPE
数据后端存储，默认"kubernetes"，本质也是k8s的etcd。也可以使用"etcd"，则使用自建的etcd。CALICO_IPV4POOL_CIDR
calio IPAM（IP Address Management）的地址池，注意要配置为k8s集群的pod网段。CALICO_IPV4POOL_IPIP
是否启用IPIP模式，如果启用则会在node上创建一个tunl0的虚拟隧道。
不起用则直接使用物理机作为虚拟路由器（vRouter），不再创建额外的tunnel。
见 《k8s网络基础》中 “4.4 IP Pool 的两种模式”
4. calico-kube-controllers
用于管理k8s集群中网络策略，yml文件如下：
kind: Deployment
metadata:
name: calico-kube-controllers
namespace: kube-system
labels:
k8s-app: calico-kube-controllers
spec:
# The controllers can only have a single active instance.
replicas: 1
selector:
matchLabels:
k8s-app: calico-kube-controllers
strategy:
type: Recreate
template:
metadata:
name: calico-kube-controllers
namespace: kube-system
labels:
k8s-app: calico-kube-controllers
spec:
nodeSelector:
kubernetes.io/os: linux
tolerations:
# Mark the pod as a critical add-on for rescheduling.
- key: CriticalAddonsOnly
operator: Exists
- key: node-role.kubernetes.io/master
effect: NoSchedule
serviceAccountName: calico-kube-controllers
priorityClassName: system-cluster-critical
containers:
- name: calico-kube-controllers
image: docker.io/calico/kube-controllers:v3.19.1
env:
# Choose which controllers to run.
- name: ENABLED_CONTROLLERS
value: node
- name: DATASTORE_TYPE
value: kubernetes
livenessProbe:
exec:
command:
- /usr/bin/check-status
- -l
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
readinessProbe:
exec:
command:
- /usr/bin/check-status
- -r
periodSeconds: 10
5. 其他
RBAC规则和自定义资源对象CRD请参考官方文档，此处不做说明。
posted on 2021-08-10 22:22 运维开发玄德公 阅读(224) 评论(0) 编辑 收藏 举报 来源
