Cilium路由模式(二)

Native-Routing


原生路由/主机路由,二者是一样,二种叫法

参考官方文档 

官方Gitlab目录  cilium/install/kubernetes/cilium/values.yaml

  1. 在本地路由模式下,Cilium 会将本节点网络地址范围外的IP数据包委托给 Linux 内核的路由子系统发到网络的另一端。 这意味着数据包将被路由,就好像本地进程会发出数据包一样。 因此,连接集群节点的网络必须能够路由 PodCIDR
  2. 配置本机路由时,Cilium 会在 Linux 内核中自动启用 IP 转发。

运行要求  

  • In order to run the native routing mode, the network connecting the hosts on which Cilium is running on must be capable of forwarding IP traffic using addresses given to pods or other workloads.  差不多意思就是,如果要使用本地路由时,网络连接必须能够转发运行Cilium的节点或者负载均衡的IP流量

  • 在初始化必须指定参数  --set tunnel=disabled关闭封装模式,以开启路由模式,本机数据包转发模式利用 Cilium 运行的网络的路由功能,而不是执行封装

实现方式

根据官方文档,使用native-routing时,节点上的 Linux 内核必须知道如何转发运行 Cilium 的所有节点的 pod 或其他工作负载的数据包。 这可以通过两种方式实现:

  1. 节点本身不知道如何路由所有 pod IP,但网络上必须存在一个知道如何到达所有其他 pod 的路由器。 在这种情况下,Linux 节点配置为包含指向此类路由器的默认路由。 该模型用于云提供商网络集成。 有关更多详细信息,请参阅 Google Cloud、AWS ENI 和 Azure IPAM
  2. 每个单独的节点都知道所有其他节点的所有 pod IP,并将路由插入 Linux 内核路由表以表示这一点。 如果所有节点共享一个 L2 网络,则可以通过启用选项 auto-direct-node-routes: true(--set autoDirectNodeRoutes=true)实现Pod间的路由,此模式即DSR。 否则,必须运行其他系统组件(例如 BGP 守护程序)来分发路由。 请参阅使用 kube-router 运行 BGP 的指南,了解如何使用 kube-router 项目实现

 不管以上那种,首先必须要关闭默认模式tunnel (--set tunnel=disabled)

配置方案

使用阿里云平台提供的路由

  1. 配置过程
    # no DSR
    helm install cilium cilium/cilium --version 1.9.9 \
        --namespace kube-system \
        --set tunnel=disabled \
        --set kubeProxyReplacement=strict \
        --set nativeRoutingCIDR=172.21.0.0/20 \
        --set ipam.mode=kubernetes \
        --set ipam.operator.clusterPoolIPv4PodCIDR=172.21.0.0/20 \
        --set ipam.operator.clusterPoolIPv4MaskSize=26 \
        --set k8sServiceHost=apiserver.qiangyun.com \
        --set k8sServicePort=6443
    
    <root@PROD-K8S-CP1 ~># helm install cilium cilium/cilium --version 1.9.9 \
    >     --namespace kube-system \
    >     --set tunnel=disabled \
    >     --set kubeProxyReplacement=strict \
    >     --set nativeRoutingCIDR=172.21.0.0/20 \
    >     --set ipam.mode=kubernetes \
    >     --set ipam.operator.clusterPoolIPv4PodCIDR=172.21.0.0/20 \
    >     --set ipam.operator.clusterPoolIPv4MaskSize=26 \
    >     --set k8sServiceHost=apiserver.qiangyun.com \
    >     --set k8sServicePort=6443
    NAME: cilium
    LAST DEPLOYED: Sat Aug 28 15:30:25 2021
    NAMESPACE: kube-system
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    You have successfully installed Cilium with Hubble.
    
    Your release version is 1.9.9.
    
    For any further help, visit https://docs.cilium.io/en/v1.9/gettinghelp
    <root@PROD-K8S-CP1 ~># dps
    1e8bef8a28ac    Up Less than a second    k8s_cilium-agent_cilium-mnddn_kube-system_aa96f316-d435-4cc4-8fc3-26fe2bee35e3_0
    8b87a2f6fce0    Up 18 hours    k8s_kube-controller-manager_kube-controller-manager-prod-k8s-cp1_kube-system_c5548fca3d6f1bb0c7cbee586dff7327_3
    e13f8dc37637    Up 18 hours    k8s_etcd_etcd-prod-k8s-cp1_kube-system_30e073f094203874eecc5317ed3ce2f6_10
    998ebbddead1    Up 18 hours    k8s_kube-scheduler_kube-scheduler-prod-k8s-cp1_kube-system_10803dd5434c54168be1114c7d99a067_10
    85e2890ed099    Up 18 hours    k8s_kube-apiserver_kube-apiserver-prod-k8s-cp1_kube-system_e14dd2db1d7c352e9552e3944ff3b802_16
    <root@PROD-K8S-CP1 ~># docker logs -f 1e8
    level=info msg="Skipped reading configuration file" reason="Config File \"ciliumd\" Not Found in \"[/root]\"" subsys=config
    level=info msg="Started gops server" address="127.0.0.1:9890" subsys=daemon
    level=info msg="Memory available for map entries (0.003% of 16508948480B): 41272371B" subsys=config
    level=info msg="option bpf-ct-global-tcp-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-ct-global-any-max set by dynamic sizing to 72407" subsys=config
    level=info msg="option bpf-nat-global-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-neigh-global-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-sock-rev-map-max set by dynamic sizing to 72407" subsys=config
    level=info msg="  --agent-health-port='9876'" subsys=daemon
    level=info msg="  --agent-labels=''" subsys=daemon
    level=info msg="  --allow-icmp-frag-needed='true'" subsys=daemon
    level=info msg="  --allow-localhost='auto'" subsys=daemon
    level=info msg="  --annotate-k8s-node='true'" subsys=daemon
    level=info msg="  --api-rate-limit='map[]'" subsys=daemon
    level=info msg="  --arping-refresh-period='5m0s'" subsys=daemon
    level=info msg="  --auto-create-cilium-node-resource='true'" subsys=daemon
    level=info msg="  --auto-direct-node-routes='false'" subsys=daemon 关闭DSR模式,使用云平台的路由功能,阿里云需要指定Cilium-node所分配的PodCIDR的网段地址
    level=info msg="  --blacklist-conflicting-routes='false'" subsys=daemon
    level=info msg="  --bpf-compile-debug='false'" subsys=daemon
    level=info msg="  --bpf-ct-global-any-max='262144'" subsys=daemon
    level=info msg="  --bpf-ct-global-tcp-max='524288'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-any='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp='6h0m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-fin='10s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-syn='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-any='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-tcp='6h0m0s'" subsys=daemon
    level=info msg="  --bpf-fragments-map-max='8192'" subsys=daemon
    level=info msg="  --bpf-lb-acceleration='disabled'" subsys=daemon
    level=info msg="  --bpf-lb-algorithm='random'" subsys=daemon
    level=info msg="  --bpf-lb-maglev-hash-seed='JLfvgnHc2kaSUFaI'" subsys=daemon
    level=info msg="  --bpf-lb-maglev-table-size='16381'" subsys=daemon
    level=info msg="  --bpf-lb-map-max='65536'" subsys=daemon
    level=info msg="  --bpf-lb-mode='snat'" subsys=daemon loadbalance负载均衡转发模式SNAT,默认配置
    level=info msg="  --bpf-map-dynamic-size-ratio='0.0025'" subsys=daemon
    level=info msg="  --bpf-nat-global-max='524288'" subsys=daemon
    level=info msg="  --bpf-neigh-global-max='524288'" subsys=daemon
    level=info msg="  --bpf-policy-map-max='16384'" subsys=daemon
    level=info msg="  --bpf-root=''" subsys=daemon
    level=info msg="  --bpf-sock-rev-map-max='262144'" subsys=daemon
    level=info msg="  --certificates-directory='/var/run/cilium/certs'" subsys=daemon
    level=info msg="  --cgroup-root='/run/cilium/cgroupv2'" subsys=daemon
    level=info msg="  --cluster-id=''" subsys=daemon
    level=info msg="  --cluster-name='default'" subsys=daemon
    level=info msg="  --clustermesh-config='/var/lib/cilium/clustermesh/'" subsys=daemon
    level=info msg="  --cmdref=''" subsys=daemon
    level=info msg="  --config=''" subsys=daemon
    level=info msg="  --config-dir='/tmp/cilium/config-map'" subsys=daemon
    level=info msg="  --conntrack-gc-interval='0s'" subsys=daemon
    level=info msg="  --crd-wait-timeout='5m0s'" subsys=daemon
    level=info msg="  --datapath-mode='veth'" subsys=daemon
    level=info msg="  --debug='false'" subsys=daemon
    level=info msg="  --debug-verbose=''" subsys=daemon
    level=info msg="  --device=''" subsys=daemon
    level=info msg="  --devices=''" subsys=daemon
    level=info msg="  --direct-routing-device=''" subsys=daemon
    level=info msg="  --disable-cnp-status-updates='true'" subsys=daemon
    level=info msg="  --disable-conntrack='false'" subsys=daemon
    level=info msg="  --disable-endpoint-crd='false'" subsys=daemon
    level=info msg="  --disable-envoy-version-check='false'" subsys=daemon
    level=info msg="  --disable-iptables-feeder-rules=''" subsys=daemon
    level=info msg="  --dns-max-ips-per-restored-rule='1000'" subsys=daemon
    level=info msg="  --egress-masquerade-interfaces=''" subsys=daemon Cilium路由模式(一)提到过,Pod向外请求时,伪装地址出口设备接口,此功能是依赖传统的iptables-bases,默认是internal接口
    level=info msg="  --egress-multi-home-ip-rule-compat='false'" subsys=daemon
    level=info msg="  --enable-auto-protect-node-port-range='true'" subsys=daemon
    level=info msg="  --enable-bandwidth-manager='false'" subsys=daemon
    level=info msg="  --enable-bpf-clock-probe='true'" subsys=daemon
    level=info msg="  --enable-bpf-masquerade='true'" subsys=daemon
    level=info msg="  --enable-bpf-tproxy='false'" subsys=daemon
    level=info msg="  --enable-endpoint-health-checking='true'" subsys=daemon
    level=info msg="  --enable-endpoint-routes='false'" subsys=daemon 关闭以endpoint为单位的路由模式,就是独立的lxc1e216780d18e(使用netstat -in 即可获得,实际就是指Container的网络设备)
    level=info msg="  --enable-external-ips='true'" subsys=daemon
    level=info msg="  --enable-health-check-nodeport='true'" subsys=daemon
    level=info msg="  --enable-health-checking='true'" subsys=daemon
    level=info msg="  --enable-host-firewall='false'" subsys=daemon
    level=info msg="  --enable-host-legacy-routing='false'" subsys=daemon 关闭主机传统路由模式,个人理解在Pod向外发送请求时,使用eBPF处理数据包
    level=info msg="  --enable-host-port='true'" subsys=daemon
    level=info msg="  --enable-host-reachable-services='false'" subsys=daemon
    level=info msg="  --enable-hubble='true'" subsys=daemon
    level=info msg="  --enable-identity-mark='true'" subsys=daemon
    level=info msg="  --enable-ip-masq-agent='false'" subsys=daemon 后面再作详细补充
    level=info msg="  --enable-ipsec='false'" subsys=daemon
    level=info msg="  --enable-ipv4='true'" subsys=daemon
    level=info msg="  --enable-ipv4-fragment-tracking='true'" subsys=daemon
    level=info msg="  --enable-ipv6='false'" subsys=daemon
    level=info msg="  --enable-ipv6-ndp='false'" subsys=daemon
    level=info msg="  --enable-k8s-api-discovery='false'" subsys=daemon
    level=info msg="  --enable-k8s-endpoint-slice='true'" subsys=daemon
    level=info msg="  --enable-k8s-event-handover='false'" subsys=daemon
    level=info msg="  --enable-l7-proxy='true'" subsys=daemon
    level=info msg="  --enable-local-node-route='true'" subsys=daemon
    level=info msg="  --enable-local-redirect-policy='false'" subsys=daemon
    level=info msg="  --enable-monitor='true'" subsys=daemon
    level=info msg="  --enable-node-port='false'" subsys=daemon
    level=info msg="  --enable-policy='default'" subsys=daemon
    level=info msg="  --enable-remote-node-identity='true'" subsys=daemon
    level=info msg="  --enable-selective-regeneration='true'" subsys=daemon
    level=info msg="  --enable-session-affinity='true'" subsys=daemon
    level=info msg="  --enable-svc-source-range-check='true'" subsys=daemon
    level=info msg="  --enable-tracing='false'" subsys=daemon
    level=info msg="  --enable-well-known-identities='false'" subsys=daemon
    level=info msg="  --enable-xt-socket-fallback='true'" subsys=daemon
    level=info msg="  --encrypt-interface=''" subsys=daemon
    level=info msg="  --encrypt-node='false'" subsys=daemon
    level=info msg="  --endpoint-interface-name-prefix='lxc+'" subsys=daemon
    level=info msg="  --endpoint-queue-size='25'" subsys=daemon
    level=info msg="  --endpoint-status=''" subsys=daemon
    level=info msg="  --envoy-log=''" subsys=daemon
    level=info msg="  --exclude-local-address=''" subsys=daemon
    level=info msg="  --fixed-identity-mapping='map[]'" subsys=daemon
    level=info msg="  --flannel-master-device=''" subsys=daemon
    level=info msg="  --flannel-uninstall-on-exit='false'" subsys=daemon
    level=info msg="  --force-local-policy-eval-at-source='true'" subsys=daemon
    level=info msg="  --gops-port='9890'" subsys=daemon
    level=info msg="  --host-reachable-services-protos='tcp,udp'" subsys=daemon
    level=info msg="  --http-403-msg=''" subsys=daemon
    level=info msg="  --http-idle-timeout='0'" subsys=daemon
    level=info msg="  --http-max-grpc-timeout='0'" subsys=daemon
    level=info msg="  --http-normalize-path='true'" subsys=daemon
    level=info msg="  --http-request-timeout='3600'" subsys=daemon
    level=info msg="  --http-retry-count='3'" subsys=daemon
    level=info msg="  --http-retry-timeout='0'" subsys=daemon
    level=info msg="  --hubble-disable-tls='false'" subsys=daemon
    level=info msg="  --hubble-event-queue-size='0'" subsys=daemon
    level=info msg="  --hubble-flow-buffer-size='4095'" subsys=daemon
    level=info msg="  --hubble-listen-address=':4244'" subsys=daemon
    level=info msg="  --hubble-metrics=''" subsys=daemon
    level=info msg="  --hubble-metrics-server=''" subsys=daemon
    level=info msg="  --hubble-socket-path='/var/run/cilium/hubble.sock'" subsys=daemon
    level=info msg="  --hubble-tls-cert-file='/var/lib/cilium/tls/hubble/server.crt'" subsys=daemon
    level=info msg="  --hubble-tls-client-ca-files='/var/lib/cilium/tls/hubble/client-ca.crt'" subsys=daemon
    level=info msg="  --hubble-tls-key-file='/var/lib/cilium/tls/hubble/server.key'" subsys=daemon
    level=info msg="  --identity-allocation-mode='crd'" subsys=daemon
    level=info msg="  --identity-change-grace-period='5s'" subsys=daemon
    level=info msg="  --install-iptables-rules='true'" subsys=daemon
    level=info msg="  --ip-allocation-timeout='2m0s'" subsys=daemon
    level=info msg="  --ip-masq-agent-config-path='/etc/config/ip-masq-agent'" subsys=daemon
    level=info msg="  --ipam='kubernetes'" subsys=daemon
    level=info msg="  --ipsec-key-file=''" subsys=daemon
    level=info msg="  --iptables-lock-timeout='5s'" subsys=daemon
    level=info msg="  --iptables-random-fully='false'" subsys=daemon
    level=info msg="  --ipv4-node='auto'" subsys=daemon
    level=info msg="  --ipv4-pod-subnets=''" subsys=daemon
    level=info msg="  --ipv4-range='auto'" subsys=daemon
    level=info msg="  --ipv4-service-loopback-address='169.254.42.1'" subsys=daemon
    level=info msg="  --ipv4-service-range='auto'" subsys=daemon
    level=info msg="  --ipv6-cluster-alloc-cidr='f00d::/64'" subsys=daemon
    level=info msg="  --ipv6-mcast-device=''" subsys=daemon
    level=info msg="  --ipv6-node='auto'" subsys=daemon
    level=info msg="  --ipv6-pod-subnets=''" subsys=daemon
    level=info msg="  --ipv6-range='auto'" subsys=daemon
    level=info msg="  --ipv6-service-range='auto'" subsys=daemon
    level=info msg="  --ipvlan-master-device='undefined'" subsys=daemon
    level=info msg="  --join-cluster='false'" subsys=daemon
    level=info msg="  --k8s-api-server=''" subsys=daemon
    level=info msg="  --k8s-force-json-patch='false'" subsys=daemon
    level=info msg="  --k8s-heartbeat-timeout='30s'" subsys=daemon
    level=info msg="  --k8s-kubeconfig-path=''" subsys=daemon
    level=info msg="  --k8s-namespace='kube-system'" subsys=daemon
    level=info msg="  --k8s-require-ipv4-pod-cidr='false'" subsys=daemon
    level=info msg="  --k8s-require-ipv6-pod-cidr='false'" subsys=daemon
    level=info msg="  --k8s-service-cache-size='128'" subsys=daemon
    level=info msg="  --k8s-service-proxy-name=''" subsys=daemon
    level=info msg="  --k8s-sync-timeout='3m0s'" subsys=daemon
    level=info msg="  --k8s-watcher-endpoint-selector='metadata.name!=kube-scheduler,metadata.name!=kube-controller-manager,metadata.name!=etcd-operator,metadata.name!=gcp-controller-manager'" subsys=daemon
    level=info msg="  --k8s-watcher-queue-size='1024'" subsys=daemon
    level=info msg="  --keep-config='false'" subsys=daemon
    level=info msg="  --kube-proxy-replacement='strict'" subsys=daemon
    level=info msg="  --kube-proxy-replacement-healthz-bind-address=''" subsys=daemon
    level=info msg="  --kvstore=''" subsys=daemon
    level=info msg="  --kvstore-connectivity-timeout='2m0s'" subsys=daemon
    level=info msg="  --kvstore-lease-ttl='15m0s'" subsys=daemon
    level=info msg="  --kvstore-opt='map[]'" subsys=daemon
    level=info msg="  --kvstore-periodic-sync='5m0s'" subsys=daemon
    level=info msg="  --label-prefix-file=''" subsys=daemon
    level=info msg="  --labels=''" subsys=daemon
    level=info msg="  --lib-dir='/var/lib/cilium'" subsys=daemon
    level=info msg="  --log-driver=''" subsys=daemon
    level=info msg="  --log-opt='map[]'" subsys=daemon
    level=info msg="  --log-system-load='false'" subsys=daemon
    level=info msg="  --masquerade='true'" subsys=daemon
    level=info msg="  --max-controller-interval='0'" subsys=daemon
    level=info msg="  --metrics=''" subsys=daemon
    level=info msg="  --monitor-aggregation='medium'" subsys=daemon
    level=info msg="  --monitor-aggregation-flags='all'" subsys=daemon
    level=info msg="  --monitor-aggregation-interval='5s'" subsys=daemon
    level=info msg="  --monitor-queue-size='0'" subsys=daemon
    level=info msg="  --mtu='0'" subsys=daemon
    level=info msg="  --nat46-range='0:0:0:0:0:FFFF::/96'" subsys=daemon
    level=info msg="  --native-routing-cidr='172.21.0.0/20'" subsys=daemon
    level=info msg="  --node-port-acceleration='disabled'" subsys=daemon
    level=info msg="  --node-port-algorithm='random'" subsys=daemon
    level=info msg="  --node-port-bind-protection='true'" subsys=daemon
    level=info msg="  --node-port-mode='snat'" subsys=daemon NodePort的模式
    level=info msg="  --node-port-range='30000,32767'" subsys=daemon
    level=info msg="  --policy-audit-mode='false'" subsys=daemon
    level=info msg="  --policy-queue-size='100'" subsys=daemon
    level=info msg="  --policy-trigger-interval='1s'" subsys=daemon
    level=info msg="  --pprof='false'" subsys=daemon
    level=info msg="  --preallocate-bpf-maps='false'" subsys=daemon
    level=info msg="  --prefilter-device='undefined'" subsys=daemon
    level=info msg="  --prefilter-mode='native'" subsys=daemon
    level=info msg="  --prepend-iptables-chains='true'" subsys=daemon
    level=info msg="  --prometheus-serve-addr=''" subsys=daemon
    level=info msg="  --proxy-connect-timeout='1'" subsys=daemon
    level=info msg="  --proxy-prometheus-port='0'" subsys=daemon
    level=info msg="  --read-cni-conf=''" subsys=daemon
    level=info msg="  --restore='true'" subsys=daemon
    level=info msg="  --sidecar-istio-proxy-image='cilium/istio_proxy'" subsys=daemon
    level=info msg="  --single-cluster-route='false'" subsys=daemon
    level=info msg="  --skip-crd-creation='false'" subsys=daemon
    level=info msg="  --socket-path='/var/run/cilium/cilium.sock'" subsys=daemon
    level=info msg="  --sockops-enable='false'" subsys=daemon
    level=info msg="  --state-dir='/var/run/cilium'" subsys=daemon
    level=info msg="  --tofqdns-dns-reject-response-code='refused'" subsys=daemon
    level=info msg="  --tofqdns-enable-dns-compression='true'" subsys=daemon
    level=info msg="  --tofqdns-endpoint-max-ip-per-hostname='50'" subsys=daemon
    level=info msg="  --tofqdns-idle-connection-grace-period='0s'" subsys=daemon
    level=info msg="  --tofqdns-max-deferred-connection-deletes='10000'" subsys=daemon
    level=info msg="  --tofqdns-min-ttl='0'" subsys=daemon
    level=info msg="  --tofqdns-pre-cache=''" subsys=daemon
    level=info msg="  --tofqdns-proxy-port='0'" subsys=daemon
    level=info msg="  --tofqdns-proxy-response-max-delay='100ms'" subsys=daemon
    level=info msg="  --trace-payloadlen='128'" subsys=daemon
    level=info msg="  --tunnel='disabled'" subsys=daemon 关闭默认tunnel功能,即走路由模式
    level=info msg="  --version='false'" subsys=daemon
    level=info msg="  --write-cni-conf-when-ready=''" subsys=daemon
    level=info msg="     _ _ _" subsys=daemon
    level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
    level=info msg="|  _| | | | | |     |" subsys=daemon
    level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
    level=info msg="Cilium 1.9.9 5bcf83c 2021-07-19T16:45:00-07:00 go version go1.15.14 linux/amd64" subsys=daemon
    level=info msg="cilium-envoy  version: 82a70d56bf324287ced3129300db609eceb21d10/1.17.3/Distribution/RELEASE/BoringSSL" subsys=daemon
    level=info msg="clang (10.0.0) and kernel (5.11.1) versions: OK!" subsys=linux-datapath
    level=info msg="linking environment: OK!" subsys=linux-datapath
    level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf
    level=info msg="Mounted cgroupv2 filesystem at /run/cilium/cgroupv2" subsys=cgroups
    level=info msg="Parsing base label prefixes from default label list" subsys=labels-filter
    level=info msg="Parsing additional label prefixes from user inputs: []" subsys=labels-filter
    level=info msg="Final label prefixes to be used for identity evaluation:" subsys=labels-filter
    level=info msg=" - reserved:.*" subsys=labels-filter
    level=info msg=" - :io.kubernetes.pod.namespace" subsys=labels-filter
    level=info msg=" - :io.cilium.k8s.namespace.labels" subsys=labels-filter
    level=info msg=" - :app.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:io.kubernetes" subsys=labels-filter
    level=info msg=" - !:kubernetes.io" subsys=labels-filter
    level=info msg=" - !:.*beta.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:k8s.io" subsys=labels-filter
    level=info msg=" - !:pod-template-generation" subsys=labels-filter
    level=info msg=" - !:pod-template-hash" subsys=labels-filter
    level=info msg=" - !:controller-revision-hash" subsys=labels-filter
    level=info msg=" - !:annotation.*" subsys=labels-filter
    level=info msg=" - !:etcd_node" subsys=labels-filter
    level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
    level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.5.0.0/16
    level=info msg="Initializing daemon" subsys=daemon
    level=info msg="Establishing connection to apiserver" host="https://apiserver.qiangyun.com:6443" subsys=k8s
    level=info msg="Connected to apiserver" subsys=k8s
    level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
    level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=10.1.0.5 mtu=1500 subsys=mtu
    level=info msg="Restored services from maps" failed=0 restored=11 subsys=service
    level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
    level=info msg="Reading old endpoints..." subsys=daemon
    level=info msg="Reusing previous DNS proxy port: 39451" subsys=daemon
    level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
    level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
    level=info msg="Retrieved node information from kubernetes node" nodeName=prod-k8s-cp1 subsys=k8s
    level=info msg="Received own node information from API server" ipAddr.ipv4=10.1.0.5 ipAddr.ipv6="<nil>" k8sNodeIP=10.1.0.5 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:prod-k8s-cp1 kubernetes.io/os:linux node-role.kubernetes.io/master: topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h]" nodeName=prod-k8s-cp1 subsys=k8s v4Prefix=172.21.0.0/24 v6Prefix="<nil>"
    level=info msg="Restored router IPs from node information" ipv4=172.21.0.85 ipv6="<nil>" subsys=k8s
    level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
    level=info msg="Using auto-derived devices to attach Loadbalancer, Host Firewall or Bandwidth Manager program" devices="[eth0]" directRoutingDevice=eth0 subsys=daemon
    level=info msg="Enabling k8s event listener" subsys=k8s-watcher
    level=info msg="Removing stale endpoint interfaces" subsys=daemon
    level=info msg="Skipping kvstore configuration" subsys=daemon
    level=info msg="Restored router address from node_config" file=/var/run/cilium/state/globals/node_config.h ipv4=172.21.0.85 ipv6="<nil>" subsys=node
    level=info msg="Initializing node addressing" subsys=daemon
    level=info msg="Initializing kubernetes IPAM" subsys=ipam v4Prefix=172.21.0.0/24 v6Prefix="<nil>"
    level=info msg="Restoring endpoints..." subsys=daemon
    level=info msg="Endpoints restored" failed=0 restored=1 subsys=daemon
    level=info msg="Addressing information:" subsys=daemon
    level=info msg="  Cluster-Name: default" subsys=daemon
    level=info msg="  Cluster-ID: 0" subsys=daemon
    level=info msg="  Local node-name: prod-k8s-cp1" subsys=daemon 本地节点名称
    level=info msg="  Node-IPv6: <nil>" subsys=daemon
    level=info msg="  External-Node IPv4: 10.1.0.5" subsys=daemon 节点地址
    level=info msg="  Internal-Node IPv4: 172.21.0.85" subsys=daemon 这里就是cilium_host设备接口的地址,也可叫网关地址或者是路由器的地址
    level=info msg="  IPv4 allocation prefix: 172.21.0.0/24" subsys=daemon 本节点可以分配的PodCIDR地址范围
    level=info msg="  IPv4 native routing prefix: 172.21.0.0/20" subsys=daemon 整个集群的PodCIDRs地址范围
    level=info msg="  Loopback IPv4: 169.254.42.1" subsys=daemon
    level=info msg="  Local IPv4 addresses:" subsys=daemon
    level=info msg="  - 10.1.0.5" subsys=daemon
    level=info msg="  - 172.21.0.85" subsys=daemon
    level=info msg="Creating or updating CiliumNode resource" node=prod-k8s-cp1 subsys=nodediscovery
    level=info msg="Waiting until all pre-existing resources related to policy have been received" subsys=k8s-watcher
    level=info msg="Adding local node to cluster" node="{prod-k8s-cp1 default [{InternalIP 10.1.0.5} {CiliumInternalIP 172.21.0.85}] 172.21.0.0/24 <nil> 172.21.0.171 <nil> 0 local 0 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:prod-k8s-cp1 kubernetes.io/os:linux node-role.kubernetes.io/master: topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h] 6}" subsys=nodediscovery
    level=info msg="Successfully created CiliumNode resource" subsys=nodediscovery
    level=info msg="Annotating k8s node" subsys=daemon v4CiliumHostIP.IPv4=172.21.0.85 v4Prefix=172.21.0.0/24 v4healthIP.IPv4=172.21.0.171 v6CiliumHostIP.IPv6="<nil>" v6Prefix="<nil>" v6healthIP.IPv6="<nil>"
    level=info msg="Initializing identity allocator" subsys=identity-cache
    level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
    level=info msg="Setting up BPF datapath" bpfClockSource=ktime bpfInsnSet=v3 subsys=datapath-loader
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.ipv4.conf.all.rp_filter sysParamValue=0
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.unprivileged_bpf_disabled sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.timer_migration sysParamValue=0
    level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
    level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
    level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
    level=info msg="Adding new proxy port rules for cilium-dns-egress:39451" proxy port name=cilium-dns-egress subsys=proxy
    level=info msg="Serving cilium node monitor v1.2 API at unix:///var/run/cilium/monitor1_2.sock" subsys=monitor-agent
    level=info msg="Validating configured node address ranges" subsys=daemon
    level=info msg="Starting connection tracking garbage collector" subsys=daemon
    level=info msg="Starting IP identity watcher" subsys=ipcache
    level=info msg="Initial scan of connection tracking completed" subsys=ct-gc
    level=info msg="Regenerating restored endpoints" numRestored=1 subsys=daemon
    level=info msg="Datapath signal listener running" subsys=signal
    level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3912 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Successfully restored endpoint. Scheduling regeneration" endpointID=3912 subsys=daemon
    level=info msg="Removed endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2275 identity=4 ipv4=172.21.0.2 ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Launching Cilium health daemon" subsys=daemon
    level=info msg="Launching Cilium health endpoint" subsys=daemon
    level=info msg="Started healthz status API server" address="127.0.0.1:9876" subsys=daemon
    level=info msg="Initializing Cilium API" subsys=daemon
    level=info msg="Daemon initialization completed" bootstrapTime=7.030950659s subsys=daemon
    level=info msg="Serving cilium API at unix:///var/run/cilium/cilium.sock" subsys=daemon
    level=info msg="Configuring Hubble server" eventQueueSize=4096 maxFlows=4095 subsys=hubble
    level=info msg="Starting local Hubble server" address="unix:///var/run/cilium/hubble.sock" subsys=hubble
    level=info msg="Beginning to read perf buffer" startTime="2021-08-28 07:30:34.868191244 +0000 UTC m=+7.098570357" subsys=monitor-agent
    level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=739 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=739 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=739 identity=4 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint
    level=info msg="Compiled new BPF template" BPFCompilationTime=1.661777466s file-path=/var/run/cilium/state/templates/64d3584c04c9bb7a4a5bcb47425a2a11f84f3b3c/bpf_host.o subsys=datapath-loader
    level=info msg="Compiled new BPF template" BPFCompilationTime=1.275228541s file-path=/var/run/cilium/state/templates/2ad9ace8cb85023fc28f2df51df10829d79ebbfa/bpf_lxc.o subsys=datapath-loader
    level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=739 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3912 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Restored endpoint" endpointID=3912 ipAddr="[ ]" subsys=endpoint
    level=info msg="Finished regenerating restored endpoints" regenerated=1 subsys=daemon total=1
  2. 查看非DSR模式下的cilium-agent状态
    <root@PROD-K8S-CP1 ~># dps
    1e8bef8a28ac    Up 18 minutes    k8s_cilium-agent_cilium-mnddn_kube-system_aa96f316-d435-4cc4-8fc3-26fe2bee35e3_0
    8b87a2f6fce0    Up 18 hours    k8s_kube-controller-manager_kube-controller-manager-prod-k8s-cp1_kube-system_c5548fca3d6f1bb0c7cbee586dff7327_3
    e13f8dc37637    Up 18 hours    k8s_etcd_etcd-prod-k8s-cp1_kube-system_30e073f094203874eecc5317ed3ce2f6_10
    998ebbddead1    Up 18 hours    k8s_kube-scheduler_kube-scheduler-prod-k8s-cp1_kube-system_10803dd5434c54168be1114c7d99a067_10
    85e2890ed099    Up 18 hours    k8s_kube-apiserver_kube-apiserver-prod-k8s-cp1_kube-system_e14dd2db1d7c352e9552e3944ff3b802_16
    <root@PROD-K8S-CP1 ~># docker exec -it 1e8 bash
    root@PROD-K8S-CP1:/home/cilium# cilium status --verbose
    KVStore:                Ok   Disabled
    Kubernetes:             Ok   1.18 (v1.18.5) [linux/amd64]
    Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
    KubeProxyReplacement:   Strict   [eth0 (Direct Routing)]
    Cilium:                 Ok   1.9.9 (v1.9.9-5bcf83c)
    NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
    Cilium health daemon:   Ok   
    IPAM:                   IPv4: 2/255 allocated from 172.21.0.0/24, 
    Allocated addresses:
      172.21.0.171 (health)
      172.21.0.85 (router)
    BandwidthManager:       Disabled
    Host Routing:           BPF
    Masquerading:           BPF   [eth0]   172.21.0.0/20
    Clock Source for BPF:   ktime
    Controller Status:      18/18 healthy
      Name                                  Last success   Last error   Count   Message
      cilium-health-ep                      52s ago        never        0       no error   
      dns-garbage-collector-job             1m0s ago       never        0       no error   
      endpoint-3912-regeneration-recovery   never          never        0       no error   
      endpoint-739-regeneration-recovery    never          never        0       no error   
      k8s-heartbeat                         30s ago        never        0       no error   
      mark-k8s-node-as-available            18m53s ago     never        0       no error   
      metricsmap-bpf-prom-sync              5s ago         never        0       no error   
      neighbor-table-refresh                3m53s ago      never        0       no error   
      resolve-identity-739                  3m52s ago      never        0       no error   
      restoring-ep-identity (3912)          18m53s ago     never        0       no error   
      sync-endpoints-and-host-ips           53s ago        never        0       no error   
      sync-lb-maps-with-k8s-services        18m53s ago     never        0       no error   
      sync-policymap-3912                   50s ago        never        0       no error   
      sync-policymap-739                    51s ago        never        0       no error   
      sync-to-k8s-ciliumendpoint (3912)     3s ago         never        0       no error   
      sync-to-k8s-ciliumendpoint (739)      12s ago        never        0       no error   
      template-dir-watcher                  never          never        0       no error   
      update-k8s-node-annotations           18m59s ago     never        0       no error   
    Proxy Status:   OK, ip 172.21.0.85, 0 redirects active on ports 10000-20000
    Hubble:         Ok   Current/Max Flows: 170/4096 (4.15%), Flows/s: 0.15   Metrics: Disabled
    KubeProxyReplacement Details:
      Status:              Strict
      Protocols:           TCP, UDP
      Devices:             eth0 (Direct Routing)
      Mode:                SNAT
      Backend Selection:   Random
      Session Affinity:    Enabled
      XDP Acceleration:    Disabled
      Services:
      - ClusterIP:      Enabled
      - NodePort:       Enabled (Range: 30000-32767) 
      - LoadBalancer:   Enabled 
      - externalIPs:    Enabled 
      - HostPort:       Enabled
    BPF Maps:   dynamic sizing: on (ratio: 0.002500)
      Name                          Size
      Non-TCP connection tracking   72407
      TCP connection tracking       144815
      Endpoint policy               65535
      Events                        4
      IP cache                      512000
      IP masquerading agent         16384
      IPv4 fragmentation            8192
      IPv4 service                  65536
      IPv6 service                  65536
      IPv4 service backend          65536
      IPv6 service backend          65536
      IPv4 service reverse NAT      65536
      IPv6 service reverse NAT      65536
      Metrics                       1024
      NAT                           144815
      Neighbor table                144815
      Global policy                 16384
      Per endpoint policy           65536
      Session affinity              65536
      Signal                        4
      Sockmap                       65535
      Sock reverse NAT              72407
      Tunnel                        65536
    Cluster health:              1/19 reachable   (2021-08-28T07:40:36Z)
      Name                       IP               Node      Endpoints
      prod-k8s-cp1 (localhost)   10.1.0.5         unknown   unknown
      prod-be-k8s-wn1            10.1.17.231      unknown   unreachable
      prod-be-k8s-wn2            10.1.17.232      unknown   unreachable
      prod-be-k8s-wn6            10.1.17.236      unknown   unreachable
      prod-be-k8s-wn7            10.1.17.237      unknown   unreachable
      prod-be-k8s-wn8            10.1.17.238      unknown   unreachable
      prod-data-k8s-wn1          10.1.18.50       unknown   unreachable
      prod-data-k8s-wn2          10.1.18.49       unknown   unreachable
      prod-data-k8s-wn3          10.1.18.51       unknown   unreachable
      prod-fe-k8s-wn1            10.1.16.221      unknown   unreachable
      prod-fe-k8s-wn2            10.1.16.222      unknown   unreachable
      prod-fe-k8s-wn3            10.1.16.223      unknown   unreachable
      prod-k8s-cp2               10.1.0.7         unknown   unreachable
      prod-k8s-cp3               10.1.0.6         unknown   unreachable
      prod-sys-k8s-wn1           10.1.0.8         unknown   unreachable
      prod-sys-k8s-wn2           10.1.0.9         unknown   unreachable
      prod-sys-k8s-wn3           10.1.0.11        unknown   unreachable
      prod-sys-k8s-wn4           10.1.0.10        unknown   unreachable
      prod-sys-k8s-wn5           10.1.0.12        unknown   unreachable
  3. 查看当前节点的路由
    <root@PROD-K8S-CP1 ~># netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.1.0.253      0.0.0.0         UG        0 0          0 eth0
    10.1.0.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
    172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
    172.21.0.0      172.21.0.85     255.255.255.0   UG        0 0          0 cilium_host 
    172.21.0.64     172.21.0.85     255.255.255.192 UG        0 0          0 cilium_host
    172.21.0.85     0.0.0.0         255.255.255.255 UH        0 0          0 cilium_host
    # 简单说明一下
    发往 172.21.0.0/24 默认网关设备接口地址172.21.0.85,这个地址实际就是cilium_host设备接口地址
    发往 172.21.0.85 的请求默认网关是0.0.0.0 实际的下一跳就是本机默认网关10.1.0.253
    <root@PROD-K8S-CP1 ~># netstat -in Kernel Interface table Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg cilium_host 1500 90686 0 0 0 1022 0 0 0 BMORU cilium_net 1500 1022 0 0 0 90686 0 0 0 BMORU docker0 1500 0 0 0 0 0 0 0 0 BMU eth0 1500 7686462 0 0 0 7443167 0 0 0 BMRU lo 65536 8147119 0 0 0 8147119 0 0 0 LRU lxc_health 1500 331 0 0 0 380 0 0 0 BMRU
  4. 配置阿里云自定义路由,具体配置略过,测试Pod的网络通信情况
    # 切换至work-node,如下,随便找一个tomcat测试
    <root@PROD-BE-K8S-WN6 ~># dps 64cdb3a1adfc Up About an hour k8s_cilium-agent_cilium-l9cjf_kube-system_c436f659-486e-4979-8849-3afb464ab7a8_0 b854d3384278 Up 15 hours k8s_tomcat_tomcat-cc8d8d7d9-zw6dx_default_d8919c65-acba-4dbb-a5da-3dc3b37896f8_1 344816fbdaaa Up 15 hours k8s_tomcat_tomcat-cc8d8d7d9-ln2qk_default_f53dab7b-b14b-4795-8fa7-24b5d90bfd70_1 676e012ec482 Up 15 hours k8s_tomcat_tomcat-cc8d8d7d9-fwqzg_default_0725de58-eb13-404d-aac8-75906cc0ca2f_1 <root@PROD-BE-K8S-WN6 ~># docker exec -it 344 bash root@tomcat-cc8d8d7d9-ln2qk:/usr/local/tomcat# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether c2:22:eb:3a:6e:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.21.12.109/32 scope global eth0 valid_lft forever preferred_lft forever

    # 在容器内测试ping外部域名时,发现不同,正常现象,因为DNS问题,与Kubernetes的Coredns网络不通,所以无法解析baidu root@tomcat
    -cc8d8d7d9-ln2qk:/usr/local/tomcat# ping www.baidu.com
    # 测试ping上海的DNS地址,结果可达 root@tomcat
    -cc8d8d7d9-ln2qk:/usr/local/tomcat# ping 202.96.209.5 PING 202.96.209.5 (202.96.209.5) 56(84) bytes of data. 64 bytes from 202.96.209.5: icmp_seq=1 ttl=53 time=12.8 ms 64 bytes from 202.96.209.5: icmp_seq=2 ttl=53 time=12.7 ms --- 202.96.209.5 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 3ms rtt min/avg/max/mdev = 12.685/12.752/12.820/0.131 ms

    # 测试ping同网段的生产区的机器,但是该机器不在Kubernetes平台中 root@tomcat
    -cc8d8d7d9-ln2qk:/usr/local/tomcat# ping 10.1.17.205 PING 10.1.17.205 (10.1.17.205) 56(84) bytes of data. 64 bytes from 10.1.17.205: icmp_seq=1 ttl=63 time=0.404 ms 64 bytes from 10.1.17.205: icmp_seq=2 ttl=63 time=0.245 ms 64 bytes from 10.1.17.205: icmp_seq=3 ttl=63 time=0.174 ms

    #切换到非Kubernetes平台中的生产区机器,测试与Pod的网络可达性
    <root@PROD-BE-QN-LOANWEB01 ~># ping 172.21.12.109
    PING 172.21.12.109 (172.21.12.109) 56(84) bytes of data.
    64 bytes from 172.21.12.109: icmp_seq=1 ttl=63 time=0.263 ms
    64 bytes from 172.21.12.109: icmp_seq=2 ttl=63 time=0.167 ms
    64 bytes from 172.21.12.109: icmp_seq=3 ttl=63 time=0.152 ms
    查看该节点的路由,发现其实并没有真正去Pod的路由,这是因为走的阿里云ECS网络提供的路由
    <root@PROD-BE-QN-LOANWEB01 ~># netstat -rn
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    0.0.0.0 10.1.17.253 0.0.0.0 UG 0 0 0 eth0
    10.1.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0DSR模式

DSR模式

个人理解,在云厂商搭建自建的Kubernetes网络还依赖各自云平台的underlay network(如果云厂商的underlay network不支持需要借助于开源的网络组件支持跨网段通信,比如kube-router等其他

  1. 初始化
    # DSR
    helm install cilium cilium/cilium --version 1.9.9 \
        --namespace kube-system \
        --set tunnel=disabled \
        --set autoDirectNodeRoutes=true \
        --set kubeProxyReplacement=strict \
        --set loadBalancer.mode=hybrid \
        --set nativeRoutingCIDR=172.21.0.0/20 \
        --set ipam.mode=kubernetes \
        --set ipam.operator.clusterPoolIPv4PodCIDR=172.21.0.0/20 \
        --set ipam.operator.clusterPoolIPv4MaskSize=26 \
        --set k8sServiceHost=apiserver.qiangyun.com \
        --set k8sServicePort=6443
    
    <root@PROD-K8S-CP1 ~># helm install cilium cilium/cilium --version 1.9.9 \
    >     --namespace kube-system \
    >     --set tunnel=disabled \
    >     --set autoDirectNodeRoutes=true \
    >     --set kubeProxyReplacement=strict \
    >     --set loadBalancer.mode=hybrid \
    >     --set nativeRoutingCIDR=172.21.0.0/20 \
    >     --set ipam.mode=kubernetes \
    >     --set ipam.operator.clusterPoolIPv4PodCIDR=172.21.0.0/20 \
    >     --set ipam.operator.clusterPoolIPv4MaskSize=26 \
    >     --set k8sServiceHost=apiserver.qiangyun.com \
    >     --set k8sServicePort=6443
    NAME: cilium
    LAST DEPLOYED: Sat Aug 28 16:59:25 2021
    NAMESPACE: kube-system
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    You have successfully installed Cilium with Hubble.
    
    Your release version is 1.9.9.
    
    For any further help, visit https://docs.cilium.io/en/v1.9/gettinghelp
    <root@PROD-K8S-CP1 ~># docker logs -f a16
    level=info msg="Skipped reading configuration file" reason="Config File \"ciliumd\" Not Found in \"[/root]\"" subsys=config
    level=info msg="Started gops server" address="127.0.0.1:9890" subsys=daemon
    level=info msg="Memory available for map entries (0.003% of 16508948480B): 41272371B" subsys=config
    level=info msg="option bpf-ct-global-tcp-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-ct-global-any-max set by dynamic sizing to 72407" subsys=config
    level=info msg="option bpf-nat-global-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-neigh-global-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-sock-rev-map-max set by dynamic sizing to 72407" subsys=config
    level=info msg="  --agent-health-port='9876'" subsys=daemon
    level=info msg="  --agent-labels=''" subsys=daemon
    level=info msg="  --allow-icmp-frag-needed='true'" subsys=daemon
    level=info msg="  --allow-localhost='auto'" subsys=daemon
    level=info msg="  --annotate-k8s-node='true'" subsys=daemon
    level=info msg="  --api-rate-limit='map[]'" subsys=daemon
    level=info msg="  --arping-refresh-period='5m0s'" subsys=daemon
    level=info msg="  --auto-create-cilium-node-resource='true'" subsys=daemon
    level=info msg="  --auto-direct-node-routes='true'" subsys=daemon  开启DSR模式,路由直接返回真实的后端
    level=info msg="  --blacklist-conflicting-routes='false'" subsys=daemon
    level=info msg="  --bpf-compile-debug='false'" subsys=daemon
    level=info msg="  --bpf-ct-global-any-max='262144'" subsys=daemon
    level=info msg="  --bpf-ct-global-tcp-max='524288'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-any='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp='6h0m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-fin='10s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-syn='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-any='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-tcp='6h0m0s'" subsys=daemon
    level=info msg="  --bpf-fragments-map-max='8192'" subsys=daemon
    level=info msg="  --bpf-lb-acceleration='disabled'" subsys=daemon
    level=info msg="  --bpf-lb-algorithm='random'" subsys=daemon
    level=info msg="  --bpf-lb-maglev-hash-seed='JLfvgnHc2kaSUFaI'" subsys=daemon
    level=info msg="  --bpf-lb-maglev-table-size='16381'" subsys=daemon
    level=info msg="  --bpf-lb-map-max='65536'" subsys=daemon
    level=info msg="  --bpf-lb-mode='snat'" subsys=daemon loadbalance的模式SNAT
    level=info msg="  --bpf-map-dynamic-size-ratio='0.0025'" subsys=daemon
    level=info msg="  --bpf-nat-global-max='524288'" subsys=daemon
    level=info msg="  --bpf-neigh-global-max='524288'" subsys=daemon
    level=info msg="  --bpf-policy-map-max='16384'" subsys=daemon
    level=info msg="  --bpf-root=''" subsys=daemon
    level=info msg="  --bpf-sock-rev-map-max='262144'" subsys=daemon
    level=info msg="  --certificates-directory='/var/run/cilium/certs'" subsys=daemon
    level=info msg="  --cgroup-root='/run/cilium/cgroupv2'" subsys=daemon
    level=info msg="  --cluster-id=''" subsys=daemon
    level=info msg="  --cluster-name='default'" subsys=daemon
    level=info msg="  --clustermesh-config='/var/lib/cilium/clustermesh/'" subsys=daemon
    level=info msg="  --cmdref=''" subsys=daemon
    level=info msg="  --config=''" subsys=daemon
    level=info msg="  --config-dir='/tmp/cilium/config-map'" subsys=daemon
    level=info msg="  --conntrack-gc-interval='0s'" subsys=daemon
    level=info msg="  --crd-wait-timeout='5m0s'" subsys=daemon
    level=info msg="  --datapath-mode='veth'" subsys=daemon
    level=info msg="  --debug='false'" subsys=daemon
    level=info msg="  --debug-verbose=''" subsys=daemon
    level=info msg="  --device=''" subsys=daemon
    level=info msg="  --devices=''" subsys=daemon
    level=info msg="  --direct-routing-device=''" subsys=daemon
    level=info msg="  --disable-cnp-status-updates='true'" subsys=daemon
    level=info msg="  --disable-conntrack='false'" subsys=daemon
    level=info msg="  --disable-endpoint-crd='false'" subsys=daemon
    level=info msg="  --disable-envoy-version-check='false'" subsys=daemon
    level=info msg="  --disable-iptables-feeder-rules=''" subsys=daemon
    level=info msg="  --dns-max-ips-per-restored-rule='1000'" subsys=daemon
    level=info msg="  --egress-masquerade-interfaces=''" subsys=daemon
    level=info msg="  --egress-multi-home-ip-rule-compat='false'" subsys=daemon
    level=info msg="  --enable-auto-protect-node-port-range='true'" subsys=daemon
    level=info msg="  --enable-bandwidth-manager='false'" subsys=daemon
    level=info msg="  --enable-bpf-clock-probe='true'" subsys=daemon
    level=info msg="  --enable-bpf-masquerade='true'" subsys=daemon
    level=info msg="  --enable-bpf-tproxy='false'" subsys=daemon
    level=info msg="  --enable-endpoint-health-checking='true'" subsys=daemon
    level=info msg="  --enable-endpoint-routes='false'" subsys=daemon 关闭以endpoint为路由单位的模式
    level=info msg="  --enable-external-ips='true'" subsys=daemon
    level=info msg="  --enable-health-check-nodeport='true'" subsys=daemon
    level=info msg="  --enable-health-checking='true'" subsys=daemon
    level=info msg="  --enable-host-firewall='false'" subsys=daemon
    level=info msg="  --enable-host-legacy-routing='false'" subsys=daemon 关闭传统路由模式,数据包接受eBPF处理
    level=info msg="  --enable-host-port='true'" subsys=daemon
    level=info msg="  --enable-host-reachable-services='false'" subsys=daemon
    level=info msg="  --enable-hubble='true'" subsys=daemon
    level=info msg="  --enable-identity-mark='true'" subsys=daemon
    level=info msg="  --enable-ip-masq-agent='false'" subsys=daemon
    level=info msg="  --enable-ipsec='false'" subsys=daemon
    level=info msg="  --enable-ipv4='true'" subsys=daemon
    level=info msg="  --enable-ipv4-fragment-tracking='true'" subsys=daemon
    level=info msg="  --enable-ipv6='false'" subsys=daemon
    level=info msg="  --enable-ipv6-ndp='false'" subsys=daemon
    level=info msg="  --enable-k8s-api-discovery='false'" subsys=daemon
    level=info msg="  --enable-k8s-endpoint-slice='true'" subsys=daemon
    level=info msg="  --enable-k8s-event-handover='false'" subsys=daemon
    level=info msg="  --enable-l7-proxy='true'" subsys=daemon
    level=info msg="  --enable-local-node-route='true'" subsys=daemon
    level=info msg="  --enable-local-redirect-policy='false'" subsys=daemon
    level=info msg="  --enable-monitor='true'" subsys=daemon
    level=info msg="  --enable-node-port='false'" subsys=daemon
    level=info msg="  --enable-policy='default'" subsys=daemon
    level=info msg="  --enable-remote-node-identity='true'" subsys=daemon
    level=info msg="  --enable-selective-regeneration='true'" subsys=daemon
    level=info msg="  --enable-session-affinity='true'" subsys=daemon
    level=info msg="  --enable-svc-source-range-check='true'" subsys=daemon
    level=info msg="  --enable-tracing='false'" subsys=daemon
    level=info msg="  --enable-well-known-identities='false'" subsys=daemon
    level=info msg="  --enable-xt-socket-fallback='true'" subsys=daemon
    level=info msg="  --encrypt-interface=''" subsys=daemon
    level=info msg="  --encrypt-node='false'" subsys=daemon
    level=info msg="  --endpoint-interface-name-prefix='lxc+'" subsys=daemon
    level=info msg="  --endpoint-queue-size='25'" subsys=daemon
    level=info msg="  --endpoint-status=''" subsys=daemon
    level=info msg="  --envoy-log=''" subsys=daemon
    level=info msg="  --exclude-local-address=''" subsys=daemon
    level=info msg="  --fixed-identity-mapping='map[]'" subsys=daemon
    level=info msg="  --flannel-master-device=''" subsys=daemon
    level=info msg="  --flannel-uninstall-on-exit='false'" subsys=daemon
    level=info msg="  --force-local-policy-eval-at-source='true'" subsys=daemon
    level=info msg="  --gops-port='9890'" subsys=daemon
    level=info msg="  --host-reachable-services-protos='tcp,udp'" subsys=daemon
    level=info msg="  --http-403-msg=''" subsys=daemon
    level=info msg="  --http-idle-timeout='0'" subsys=daemon
    level=info msg="  --http-max-grpc-timeout='0'" subsys=daemon
    level=info msg="  --http-normalize-path='true'" subsys=daemon
    level=info msg="  --http-request-timeout='3600'" subsys=daemon
    level=info msg="  --http-retry-count='3'" subsys=daemon
    level=info msg="  --http-retry-timeout='0'" subsys=daemon
    level=info msg="  --hubble-disable-tls='false'" subsys=daemon
    level=info msg="  --hubble-event-queue-size='0'" subsys=daemon
    level=info msg="  --hubble-flow-buffer-size='4095'" subsys=daemon
    level=info msg="  --hubble-listen-address=':4244'" subsys=daemon
    level=info msg="  --hubble-metrics=''" subsys=daemon
    level=info msg="  --hubble-metrics-server=''" subsys=daemon
    level=info msg="  --hubble-socket-path='/var/run/cilium/hubble.sock'" subsys=daemon
    level=info msg="  --hubble-tls-cert-file='/var/lib/cilium/tls/hubble/server.crt'" subsys=daemon
    level=info msg="  --hubble-tls-client-ca-files='/var/lib/cilium/tls/hubble/client-ca.crt'" subsys=daemon
    level=info msg="  --hubble-tls-key-file='/var/lib/cilium/tls/hubble/server.key'" subsys=daemon
    level=info msg="  --identity-allocation-mode='crd'" subsys=daemon
    level=info msg="  --identity-change-grace-period='5s'" subsys=daemon
    level=info msg="  --install-iptables-rules='true'" subsys=daemon
    level=info msg="  --ip-allocation-timeout='2m0s'" subsys=daemon
    level=info msg="  --ip-masq-agent-config-path='/etc/config/ip-masq-agent'" subsys=daemon
    level=info msg="  --ipam='kubernetes'" subsys=daemon
    level=info msg="  --ipsec-key-file=''" subsys=daemon
    level=info msg="  --iptables-lock-timeout='5s'" subsys=daemon
    level=info msg="  --iptables-random-fully='false'" subsys=daemon
    level=info msg="  --ipv4-node='auto'" subsys=daemon
    level=info msg="  --ipv4-pod-subnets=''" subsys=daemon
    level=info msg="  --ipv4-range='auto'" subsys=daemon
    level=info msg="  --ipv4-service-loopback-address='169.254.42.1'" subsys=daemon
    level=info msg="  --ipv4-service-range='auto'" subsys=daemon
    level=info msg="  --ipv6-cluster-alloc-cidr='f00d::/64'" subsys=daemon
    level=info msg="  --ipv6-mcast-device=''" subsys=daemon
    level=info msg="  --ipv6-node='auto'" subsys=daemon
    level=info msg="  --ipv6-pod-subnets=''" subsys=daemon
    level=info msg="  --ipv6-range='auto'" subsys=daemon
    level=info msg="  --ipv6-service-range='auto'" subsys=daemon
    level=info msg="  --ipvlan-master-device='undefined'" subsys=daemon
    level=info msg="  --join-cluster='false'" subsys=daemon
    level=info msg="  --k8s-api-server=''" subsys=daemon
    level=info msg="  --k8s-force-json-patch='false'" subsys=daemon
    level=info msg="  --k8s-heartbeat-timeout='30s'" subsys=daemon
    level=info msg="  --k8s-kubeconfig-path=''" subsys=daemon
    level=info msg="  --k8s-namespace='kube-system'" subsys=daemon
    level=info msg="  --k8s-require-ipv4-pod-cidr='false'" subsys=daemon
    level=info msg="  --k8s-require-ipv6-pod-cidr='false'" subsys=daemon
    level=info msg="  --k8s-service-cache-size='128'" subsys=daemon
    level=info msg="  --k8s-service-proxy-name=''" subsys=daemon
    level=info msg="  --k8s-sync-timeout='3m0s'" subsys=daemon
    level=info msg="  --k8s-watcher-endpoint-selector='metadata.name!=kube-scheduler,metadata.name!=kube-controller-manager,metadata.name!=etcd-operator,metadata.name!=gcp-controller-manager'" subsys=daemon
    level=info msg="  --k8s-watcher-queue-size='1024'" subsys=daemon
    level=info msg="  --keep-config='false'" subsys=daemon
    level=info msg="  --kube-proxy-replacement='strict'" subsys=daemon
    level=info msg="  --kube-proxy-replacement-healthz-bind-address=''" subsys=daemon
    level=info msg="  --kvstore=''" subsys=daemon
    level=info msg="  --kvstore-connectivity-timeout='2m0s'" subsys=daemon
    level=info msg="  --kvstore-lease-ttl='15m0s'" subsys=daemon
    level=info msg="  --kvstore-opt='map[]'" subsys=daemon
    level=info msg="  --kvstore-periodic-sync='5m0s'" subsys=daemon
    level=info msg="  --label-prefix-file=''" subsys=daemon
    level=info msg="  --labels=''" subsys=daemon
    level=info msg="  --lib-dir='/var/lib/cilium'" subsys=daemon
    level=info msg="  --log-driver=''" subsys=daemon
    level=info msg="  --log-opt='map[]'" subsys=daemon
    level=info msg="  --log-system-load='false'" subsys=daemon
    level=info msg="  --masquerade='true'" subsys=daemon 伪装模式默认开启
    level=info msg="  --max-controller-interval='0'" subsys=daemon
    level=info msg="  --metrics=''" subsys=daemon
    level=info msg="  --monitor-aggregation='medium'" subsys=daemon
    level=info msg="  --monitor-aggregation-flags='all'" subsys=daemon
    level=info msg="  --monitor-aggregation-interval='5s'" subsys=daemon
    level=info msg="  --monitor-queue-size='0'" subsys=daemon
    level=info msg="  --mtu='0'" subsys=daemon
    level=info msg="  --nat46-range='0:0:0:0:0:FFFF::/96'" subsys=daemon
    level=info msg="  --native-routing-cidr='172.21.0.0/20'" subsys=daemon
    level=info msg="  --node-port-acceleration='disabled'" subsys=daemon
    level=info msg="  --node-port-algorithm='random'" subsys=daemon
    level=info msg="  --node-port-bind-protection='true'" subsys=daemon
    level=info msg="  --node-port-mode='hybrid'" subsys=daemon
    level=info msg="  --node-port-range='30000,32767'" subsys=daemon
    level=info msg="  --policy-audit-mode='false'" subsys=daemon
    level=info msg="  --policy-queue-size='100'" subsys=daemon
    level=info msg="  --policy-trigger-interval='1s'" subsys=daemon
    level=info msg="  --pprof='false'" subsys=daemon
    level=info msg="  --preallocate-bpf-maps='false'" subsys=daemon
    level=info msg="  --prefilter-device='undefined'" subsys=daemon
    level=info msg="  --prefilter-mode='native'" subsys=daemon
    level=info msg="  --prepend-iptables-chains='true'" subsys=daemon
    level=info msg="  --prometheus-serve-addr=''" subsys=daemon
    level=info msg="  --proxy-connect-timeout='1'" subsys=daemon
    level=info msg="  --proxy-prometheus-port='0'" subsys=daemon
    level=info msg="  --read-cni-conf=''" subsys=daemon
    level=info msg="  --restore='true'" subsys=daemon
    level=info msg="  --sidecar-istio-proxy-image='cilium/istio_proxy'" subsys=daemon
    level=info msg="  --single-cluster-route='false'" subsys=daemon
    level=info msg="  --skip-crd-creation='false'" subsys=daemon
    level=info msg="  --socket-path='/var/run/cilium/cilium.sock'" subsys=daemon
    level=info msg="  --sockops-enable='false'" subsys=daemon
    level=info msg="  --state-dir='/var/run/cilium'" subsys=daemon
    level=info msg="  --tofqdns-dns-reject-response-code='refused'" subsys=daemon
    level=info msg="  --tofqdns-enable-dns-compression='true'" subsys=daemon
    level=info msg="  --tofqdns-endpoint-max-ip-per-hostname='50'" subsys=daemon
    level=info msg="  --tofqdns-idle-connection-grace-period='0s'" subsys=daemon
    level=info msg="  --tofqdns-max-deferred-connection-deletes='10000'" subsys=daemon
    level=info msg="  --tofqdns-min-ttl='0'" subsys=daemon
    level=info msg="  --tofqdns-pre-cache=''" subsys=daemon
    level=info msg="  --tofqdns-proxy-port='0'" subsys=daemon
    level=info msg="  --tofqdns-proxy-response-max-delay='100ms'" subsys=daemon
    level=info msg="  --trace-payloadlen='128'" subsys=daemon
    level=info msg="  --tunnel='disabled'" subsys=daemon
    level=info msg="  --version='false'" subsys=daemon
    level=info msg="  --write-cni-conf-when-ready=''" subsys=daemon
    level=info msg="     _ _ _" subsys=daemon
    level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
    level=info msg="|  _| | | | | |     |" subsys=daemon
    level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
    level=info msg="Cilium 1.9.9 5bcf83c 2021-07-19T16:45:00-07:00 go version go1.15.14 linux/amd64" subsys=daemon
    level=info msg="cilium-envoy  version: 82a70d56bf324287ced3129300db609eceb21d10/1.17.3/Distribution/RELEASE/BoringSSL" subsys=daemon
    level=info msg="clang (10.0.0) and kernel (5.11.1) versions: OK!" subsys=linux-datapath
    level=info msg="linking environment: OK!" subsys=linux-datapath
    level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf
    level=info msg="Mounted cgroupv2 filesystem at /run/cilium/cgroupv2" subsys=cgroups
    level=info msg="Parsing base label prefixes from default label list" subsys=labels-filter
    level=info msg="Parsing additional label prefixes from user inputs: []" subsys=labels-filter
    level=info msg="Final label prefixes to be used for identity evaluation:" subsys=labels-filter
    level=info msg=" - reserved:.*" subsys=labels-filter
    level=info msg=" - :io.kubernetes.pod.namespace" subsys=labels-filter
    level=info msg=" - :io.cilium.k8s.namespace.labels" subsys=labels-filter
    level=info msg=" - :app.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:io.kubernetes" subsys=labels-filter
    level=info msg=" - !:kubernetes.io" subsys=labels-filter
    level=info msg=" - !:.*beta.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:k8s.io" subsys=labels-filter
    level=info msg=" - !:pod-template-generation" subsys=labels-filter
    level=info msg=" - !:pod-template-hash" subsys=labels-filter
    level=info msg=" - !:controller-revision-hash" subsys=labels-filter
    level=info msg=" - !:annotation.*" subsys=labels-filter
    level=info msg=" - !:etcd_node" subsys=labels-filter
    level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
    level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.5.0.0/16
    level=info msg="Initializing daemon" subsys=daemon
    level=info msg="Establishing connection to apiserver" host="https://apiserver.qiangyun.com:6443" subsys=k8s
    level=info msg="Connected to apiserver" subsys=k8s
    level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
    level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=10.1.0.5 mtu=1500 subsys=mtu
    level=info msg="Restored services from maps" failed=0 restored=11 subsys=service
    level=info msg="Reading old endpoints..." subsys=daemon
    level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
    level=info msg="Reusing previous DNS proxy port: 39451" subsys=daemon
    level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
    level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
    level=info msg="Retrieved node information from kubernetes node" nodeName=prod-k8s-cp1 subsys=k8s
    level=info msg="Received own node information from API server" ipAddr.ipv4=10.1.0.5 ipAddr.ipv6="<nil>" k8sNodeIP=10.1.0.5 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:prod-k8s-cp1 kubernetes.io/os:linux node-role.kubernetes.io/master: topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h]" nodeName=prod-k8s-cp1 subsys=k8s v4Prefix=172.21.0.0/24 v6Prefix="<nil>"
    level=info msg="Restored router IPs from node information" ipv4=172.21.0.85 ipv6="<nil>" subsys=k8s
    level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
    level=info msg="Using auto-derived devices to attach Loadbalancer, Host Firewall or Bandwidth Manager program" devices="[eth0]" directRoutingDevice=eth0 subsys=daemon
    level=info msg="Enabling k8s event listener" subsys=k8s-watcher
    level=info msg="Removing stale endpoint interfaces" subsys=daemon
    level=info msg="Skipping kvstore configuration" subsys=daemon
    level=info msg="Restored router address from node_config" file=/var/run/cilium/state/globals/node_config.h ipv4=172.21.0.85 ipv6="<nil>" subsys=node
    level=info msg="Initializing node addressing" subsys=daemon
    level=info msg="Initializing kubernetes IPAM" subsys=ipam v4Prefix=172.21.0.0/24 v6Prefix="<nil>"
    level=info msg="Restoring endpoints..." subsys=daemon
    level=info msg="Waiting until all pre-existing resources related to policy have been received" subsys=k8s-watcher
    level=info msg="Endpoints restored" failed=0 restored=1 subsys=daemon
    level=info msg="Addressing information:" subsys=daemon
    level=info msg="  Cluster-Name: default" subsys=daemon
    level=info msg="  Cluster-ID: 0" subsys=daemon
    level=info msg="  Local node-name: prod-k8s-cp1" subsys=daemon
    level=info msg="  Node-IPv6: <nil>" subsys=daemon
    level=info msg="  External-Node IPv4: 10.1.0.5" subsys=daemon
    level=info msg="  Internal-Node IPv4: 172.21.0.85" subsys=daemon
    level=info msg="  IPv4 allocation prefix: 172.21.0.0/24" subsys=daemon
    level=info msg="  IPv4 native routing prefix: 172.21.0.0/20" subsys=daemon
    level=info msg="  Loopback IPv4: 169.254.42.1" subsys=daemon
    level=info msg="  Local IPv4 addresses:" subsys=daemon
    level=info msg="  - 10.1.0.5" subsys=daemon
    level=info msg="  - 172.21.0.85" subsys=daemon
    level=info msg="Creating or updating CiliumNode resource" node=prod-k8s-cp1 subsys=nodediscovery
    level=info msg="Adding local node to cluster" node="{prod-k8s-cp1 default [{InternalIP 10.1.0.5} {CiliumInternalIP 172.21.0.85}] 172.21.0.0/24 <nil> 172.21.0.71 <nil> 0 local 0 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:prod-k8s-cp1 kubernetes.io/os:linux node-role.kubernetes.io/master: topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h] 6}" subsys=nodediscovery
    level=info msg="Successfully created CiliumNode resource" subsys=nodediscovery
    level=info msg="Annotating k8s node" subsys=daemon v4CiliumHostIP.IPv4=172.21.0.85 v4Prefix=172.21.0.0/24 v4healthIP.IPv4=172.21.0.71 v6CiliumHostIP.IPv6="<nil>" v6Prefix="<nil>" v6healthIP.IPv6="<nil>"
    level=info msg="Initializing identity allocator" subsys=identity-cache
    level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
    level=info msg="Setting up BPF datapath" bpfClockSource=ktime bpfInsnSet=v3 subsys=datapath-loader
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.ipv4.conf.all.rp_filter sysParamValue=0
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.unprivileged_bpf_disabled sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.timer_migration sysParamValue=0
    level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
    # 属于正常,因为我们生产环境网络划分的原因,DSR模式要求所有后端在同一个L2网段中,不影响通信 level
    =warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.13.0/24 Src: <nil> Gw: 10.1.18.50 Flags: [] Table: 0}" error="route to destination 10.1.18.50 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.64/26 Src: <nil> Gw: 10.1.17.236 Flags: [] Table: 0}" error="route to destination 10.1.17.236 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.9.0/24 Src: <nil> Gw: 10.1.16.221 Flags: [] Table: 0}" error="route to destination 10.1.16.221 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.5.0/24 Src: <nil> Gw: 10.1.17.231 Flags: [] Table: 0}" error="route to destination 10.1.17.231 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.15.0/24 Src: <nil> Gw: 10.1.18.51 Flags: [] Table: 0}" error="route to destination 10.1.18.51 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.0/26 Src: <nil> Gw: 10.1.17.237 Flags: [] Table: 0}" error="route to destination 10.1.17.237 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.14.0/24 Src: <nil> Gw: 10.1.18.49 Flags: [] Table: 0}" error="route to destination 10.1.18.49 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.6.0/24 Src: <nil> Gw: 10.1.17.232 Flags: [] Table: 0}" error="route to destination 10.1.17.232 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.10.0/24 Src: <nil> Gw: 10.1.16.222 Flags: [] Table: 0}" error="route to destination 10.1.16.222 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.192/26 Src: <nil> Gw: 10.1.16.223 Flags: [] Table: 0}" error="route to destination 10.1.16.223 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.128/26 Src: <nil> Gw: 10.1.17.238 Flags: [] Table: 0}" error="route to destination 10.1.17.238 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager level=info msg="Adding new proxy port rules for cilium-dns-egress:39451" proxy port name=cilium-dns-egress subsys=proxy level=info msg="Serving cilium node monitor v1.2 API at unix:///var/run/cilium/monitor1_2.sock" subsys=monitor-agent level=info msg="Validating configured node address ranges" subsys=daemon level=info msg="Starting connection tracking garbage collector" subsys=daemon level=info msg="Starting IP identity watcher" subsys=ipcache level=info msg="Initial scan of connection tracking completed" subsys=ct-gc level=info msg="Regenerating restored endpoints" numRestored=1 subsys=daemon level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.014266576435979946 newInterval=7m30s subsys=map-ct level=info msg="Datapath signal listener running" subsys=signal level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3912 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint level=info msg="Successfully restored endpoint. Scheduling regeneration" endpointID=3912 subsys=daemon level=info msg="Removed endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=739 identity=4 ipv4=172.21.0.171 ipv6= k8sPodName=/ subsys=endpoint level=info msg="Launching Cilium health daemon" subsys=daemon level=info msg="Launching Cilium health endpoint" subsys=daemon level=info msg="Started healthz status API server" address="127.0.0.1:9876" subsys=daemon level=info msg="Initializing Cilium API" subsys=daemon level=info msg="Daemon initialization completed" bootstrapTime=6.17475652s subsys=daemon level=info msg="Serving cilium API at unix:///var/run/cilium/cilium.sock" subsys=daemon level=info msg="Configuring Hubble server" eventQueueSize=4096 maxFlows=4095 subsys=hubble level=info msg="Starting local Hubble server" address="unix:///var/run/cilium/hubble.sock" subsys=hubble level=info msg="Beginning to read perf buffer" startTime="2021-08-28 08:59:34.474285821 +0000 UTC m=+6.245198613" subsys=monitor-agent level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3610 ipv4= ipv6= k8sPodName=/ subsys=endpoint level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3610 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ subsys=endpoint level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3610 identity=4 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager level=info msg="Compiled new BPF template" BPFCompilationTime=1.654455554s file-path=/var/run/cilium/state/templates/ebd8a5ff175221b719cd4ae752053c5787bcb5b2/bpf_host.o subsys=datapath-loader level=info msg="Compiled new BPF template" BPFCompilationTime=1.340506836s file-path=/var/run/cilium/state/templates/1cfa9d9a215498b4089c630b564520f2b1b80c85/bpf_lxc.o subsys=datapath-loader level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3610 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3912 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint level=info msg="Restored endpoint" endpointID=3912 ipAddr="[ ]" subsys=endpoint level=info msg="Finished regenerating restored endpoints" regenerated=1 subsys=daemon total=1 level=info msg="Waiting for Hubble server TLS certificate and key files to be created" subsys=hubble level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.192/26 Src: <nil> Gw: 10.1.16.223 Flags: [] Table: 0}" error="route to destination 10.1.16.223 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.6.0/24 Src: <nil> Gw: 10.1.17.232 Flags: [] Table: 0}" error="route to destination 10.1.17.232 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.9.0/24 Src: <nil> Gw: 10.1.16.221 Flags: [] Table: 0}" error="route to destination 10.1.16.221 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.13.0/24 Src: <nil> Gw: 10.1.18.50 Flags: [] Table: 0}" error="route to destination 10.1.18.50 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.128/26 Src: <nil> Gw: 10.1.17.238 Flags: [] Table: 0}" error="route to destination 10.1.17.238 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.5.0/24 Src: <nil> Gw: 10.1.17.231 Flags: [] Table: 0}" error="route to destination 10.1.17.231 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.10.0/24 Src: <nil> Gw: 10.1.16.222 Flags: [] Table: 0}" error="route to destination 10.1.16.222 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.14.0/24 Src: <nil> Gw: 10.1.18.49 Flags: [] Table: 0}" error="route to destination 10.1.18.49 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.64/26 Src: <nil> Gw: 10.1.17.236 Flags: [] Table: 0}" error="route to destination 10.1.17.236 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.15.0/24 Src: <nil> Gw: 10.1.18.51 Flags: [] Table: 0}" error="route to destination 10.1.18.51 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath level=warning msg="Unable to install direct node route {Ifindex: 0 Dst: 172.21.12.0/26 Src: <nil> Gw: 10.1.17.237 Flags: [] Table: 0}" error="route to destination 10.1.17.237 contains gateway 10.1.0.253, must be directly reachable" subsys=linux-datapath
  2. 查看DSR模式下的cilium-agent的状态
    <root@PROD-K8S-CP1 ~># dps
    a166d3d25ee3    Up 18 minutes    k8s_cilium-agent_cilium-zlhzc_kube-system_231baf2d-f32c-463b-88e8-faa73db507f4_0
    8b87a2f6fce0    Up 19 hours    k8s_kube-controller-manager_kube-controller-manager-prod-k8s-cp1_kube-system_c5548fca3d6f1bb0c7cbee586dff7327_3
    e13f8dc37637    Up 19 hours    k8s_etcd_etcd-prod-k8s-cp1_kube-system_30e073f094203874eecc5317ed3ce2f6_10
    998ebbddead1    Up 19 hours    k8s_kube-scheduler_kube-scheduler-prod-k8s-cp1_kube-system_10803dd5434c54168be1114c7d99a067_10
    85e2890ed099    Up 19 hours    k8s_kube-apiserver_kube-apiserver-prod-k8s-cp1_kube-system_e14dd2db1d7c352e9552e3944ff3b802_16
    <root@PROD-K8S-CP1 ~># docker exec -it a16 bash
    root@PROD-K8S-CP1:/home/cilium# cilium status --verbose
    KVStore:                Ok   Disabled
    Kubernetes:             Ok   1.18 (v1.18.5) [linux/amd64]
    Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
    KubeProxyReplacement:   Strict   [eth0 (Direct Routing)]
    Cilium:                 Ok   1.9.9 (v1.9.9-5bcf83c)
    NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
    Cilium health daemon:   Ok   
    IPAM:                   IPv4: 2/255 allocated from 172.21.0.0/24, 
    Allocated addresses:
      172.21.0.71 (health)
      172.21.0.85 (router)
    BandwidthManager:       Disabled
    Host Routing:           BPF
    Masquerading:           BPF   [eth0]   172.21.0.0/20
    Clock Source for BPF:   ktime
    Controller Status:      18/18 healthy
      Name                                  Last success   Last error   Count   Message
      cilium-health-ep                      52s ago        never        0       no error   
      dns-garbage-collector-job             59s ago        never        0       no error   
      endpoint-3610-regeneration-recovery   never          never        0       no error   
      endpoint-3912-regeneration-recovery   never          never        0       no error   
      k8s-heartbeat                         28s ago        never        0       no error   
      mark-k8s-node-as-available            18m53s ago     never        0       no error   
      metricsmap-bpf-prom-sync              3s ago         never        0       no error   
      neighbor-table-refresh                3m53s ago      never        0       no error   
      resolve-identity-3610                 3m52s ago      never        0       no error   
      restoring-ep-identity (3912)          18m53s ago     never        0       no error   
      sync-endpoints-and-host-ips           53s ago        never        0       no error   
      sync-lb-maps-with-k8s-services        18m53s ago     never        0       no error   
      sync-policymap-3610                   50s ago        never        0       no error   
      sync-policymap-3912                   50s ago        never        0       no error   
      sync-to-k8s-ciliumendpoint (3610)     12s ago        never        0       no error   
      sync-to-k8s-ciliumendpoint (3912)     3s ago         never        0       no error   
      template-dir-watcher                  never          never        0       no error   
      update-k8s-node-annotations           18m57s ago     never        0       no error   
    Proxy Status:   OK, ip 172.21.0.85, 0 redirects active on ports 10000-20000
    Hubble:         Ok   Current/Max Flows: 782/4096 (19.09%), Flows/s: 0.69   Metrics: Disabled
    KubeProxyReplacement Details:
      Status:              Strict
      Protocols:           TCP, UDP
      Devices:             eth0 (Direct Routing)
      Mode:                Hybrid
      Backend Selection:   Random
      Session Affinity:    Enabled
      XDP Acceleration:    Disabled
      Services:
      - ClusterIP:      Enabled
      - NodePort:       Enabled (Range: 30000-32767) 
      - LoadBalancer:   Enabled 
      - externalIPs:    Enabled 
      - HostPort:       Enabled
    BPF Maps:   dynamic sizing: on (ratio: 0.002500)
      Name                          Size
      Non-TCP connection tracking   72407
      TCP connection tracking       144815
      Endpoint policy               65535
      Events                        4
      IP cache                      512000
      IP masquerading agent         16384
      IPv4 fragmentation            8192
      IPv4 service                  65536
      IPv6 service                  65536
      IPv4 service backend          65536
      IPv6 service backend          65536
      IPv4 service reverse NAT      65536
      IPv6 service reverse NAT      65536
      Metrics                       1024
      NAT                           144815
      Neighbor table                144815
      Global policy                 16384
      Per endpoint policy           65536
      Session affinity              65536
      Signal                        4
      Sockmap                       65535
      Sock reverse NAT              72407
      Tunnel                        65536
    Cluster health:              2/19 reachable   (2021-08-28T09:17:36Z)
      Name                       IP               Node        Endpoints
      prod-k8s-cp1 (localhost)   10.1.0.5         reachable   reachable
      prod-be-k8s-wn1            10.1.17.231      reachable   unreachable
      prod-be-k8s-wn2            10.1.17.232      reachable   unreachable
      prod-be-k8s-wn6            10.1.17.236      reachable   unreachable
      prod-be-k8s-wn7            10.1.17.237      reachable   unreachable
      prod-be-k8s-wn8            10.1.17.238      reachable   unreachable
      prod-data-k8s-wn1          10.1.18.50       reachable   reachable
      prod-data-k8s-wn2          10.1.18.49       reachable   unreachable
      prod-data-k8s-wn3          10.1.18.51       reachable   unreachable
      prod-fe-k8s-wn1            10.1.16.221      reachable   unreachable
      prod-fe-k8s-wn2            10.1.16.222      reachable   unreachable
      prod-fe-k8s-wn3            10.1.16.223      reachable   unreachable
      prod-k8s-cp2               10.1.0.7         reachable   unreachable
      prod-k8s-cp3               10.1.0.6         reachable   unreachable
      prod-sys-k8s-wn1           10.1.0.8         reachable   unreachable
      prod-sys-k8s-wn2           10.1.0.9         reachable   unreachable
      prod-sys-k8s-wn3           10.1.0.11        reachable   unreachable
      prod-sys-k8s-wn4           10.1.0.10        reachable   unreachable
      prod-sys-k8s-wn5           10.1.0.12        reachable   unreachable
  3. 查看基于DSR模式下的路由情况
    # 不同的是DSR模式下只能侦察到同网段的路由信息,无法获取跨网段的路由,如果与不同的网段通信,则判断走本节点的默认路由,下一跳获取阿里云后端的自定义路由信息
    <root@PROD-K8S-CP1 ~># netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.1.0.253 0.0.0.0 UG 0 0 0 eth0 10.1.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.21.0.0 172.21.0.85 255.255.255.0 UG 0 0 0 cilium_host 172.21.0.64 172.21.0.85 255.255.255.192 UG 0 0 0 cilium_host 172.21.0.85 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host 172.21.1.0 10.1.0.7 255.255.255.0 UG 0 0 0 eth0 172.21.2.0 10.1.0.6 255.255.255.0 UG 0 0 0 eth0 172.21.3.0 10.1.0.8 255.255.255.0 UG 0 0 0 eth0 172.21.4.0 10.1.0.9 255.255.255.0 UG 0 0 0 eth0 172.21.7.0 10.1.0.11 255.255.255.0 UG 0 0 0 eth0 172.21.8.0 10.1.0.10 255.255.255.0 UG 0 0 0 eth0 172.21.11.0 10.1.0.12 255.255.255.0 UG 0 0 0 eth0 <root@PROD-BE-K8S-WN6 ~># netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.1.17.253 0.0.0.0 UG 0 0 0 eth0 10.1.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.21.5.0 10.1.17.231 255.255.255.0 UG 0 0 0 eth0 172.21.6.0 10.1.17.232 255.255.255.0 UG 0 0 0 eth0 172.21.12.0 10.1.17.237 255.255.255.192 UG 0 0 0 eth0 172.21.12.64 172.21.12.86 255.255.255.192 UG 0 0 0 cilium_host 172.21.12.86 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host 172.21.12.128 10.1.17.238 255.255.255.192 UG 0 0 0 eth0 <root@PROD-DATA-K8S-WN1 ~># netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.1.18.253 0.0.0.0 UG 0 0 0 eth0 10.1.18.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.21.13.0 172.21.13.25 255.255.255.0 UG 0 0 0 cilium_host 172.21.13.25 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host 172.21.14.0 10.1.18.49 255.255.255.0 UG 0 0 0 eth0 172.21.15.0 10.1.18.51 255.255.255.0 UG 0 0 0 eth0
  4. 测试Pod网络连通性跳过,路由存在则网络必达

endpoint模式

官方原文的意思

--set endpointRoutes.enabled=true

endpointRoutes:

# -- Enable use of per endpoint routes instead of routing via
# the cilium_host interface.
enabled: false

  1. 初始化
    <root@PROD-K8S-CP1 ~># helm install cilium cilium/cilium --version 1.9.9 \
    >     --namespace kube-system \
    >     --set tunnel=disabled \
    >     --set endpointRoutes.enabled=true \
    >     --set kubeProxyReplacement=strict \
    >     --set loadBalancer.mode=hybrid \
    >     --set nativeRoutingCIDR=172.21.0.0/20 \
    >     --set ipam.mode=kubernetes \
    >     --set ipam.operator.clusterPoolIPv4PodCIDR=172.21.0.0/20 \
    >     --set ipam.operator.clusterPoolIPv4MaskSize=26 \
    >     --set k8sServiceHost=apiserver.qiangyun.com \
    >     --set k8sServicePort=6443
    NAME: cilium
    LAST DEPLOYED: Sat Aug 28 18:04:09 2021
    NAMESPACE: kube-system
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    NOTES:
    You have successfully installed Cilium with Hubble.
    
    Your release version is 1.9.9.
    
    For any further help, visit https://docs.cilium.io/en/v1.9/gettinghelp
  2. 查看cilium-agent日志
    <root@PROD-K8S-CP1 ~># docker logs -f 716
    level=info msg="Skipped reading configuration file" reason="Config File \"ciliumd\" Not Found in \"[/root]\"" subsys=config
    level=info msg="Started gops server" address="127.0.0.1:9890" subsys=daemon
    level=info msg="Memory available for map entries (0.003% of 16508948480B): 41272371B" subsys=config
    level=info msg="option bpf-ct-global-tcp-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-ct-global-any-max set by dynamic sizing to 72407" subsys=config
    level=info msg="option bpf-nat-global-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-neigh-global-max set by dynamic sizing to 144815" subsys=config
    level=info msg="option bpf-sock-rev-map-max set by dynamic sizing to 72407" subsys=config
    level=info msg="  --agent-health-port='9876'" subsys=daemon
    level=info msg="  --agent-labels=''" subsys=daemon
    level=info msg="  --allow-icmp-frag-needed='true'" subsys=daemon
    level=info msg="  --allow-localhost='auto'" subsys=daemon
    level=info msg="  --annotate-k8s-node='true'" subsys=daemon
    level=info msg="  --api-rate-limit='map[]'" subsys=daemon
    level=info msg="  --arping-refresh-period='5m0s'" subsys=daemon
    level=info msg="  --auto-create-cilium-node-resource='true'" subsys=daemon
    level=info msg="  --auto-direct-node-routes='false'" subsys=daemon 关闭DSR模式
    level=info msg="  --blacklist-conflicting-routes='false'" subsys=daemon
    level=info msg="  --bpf-compile-debug='false'" subsys=daemon
    level=info msg="  --bpf-ct-global-any-max='262144'" subsys=daemon
    level=info msg="  --bpf-ct-global-tcp-max='524288'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-any='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp='6h0m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-fin='10s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-syn='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-any='1m0s'" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-tcp='6h0m0s'" subsys=daemon
    level=info msg="  --bpf-fragments-map-max='8192'" subsys=daemon
    level=info msg="  --bpf-lb-acceleration='disabled'" subsys=daemon
    level=info msg="  --bpf-lb-algorithm='random'" subsys=daemon
    level=info msg="  --bpf-lb-maglev-hash-seed='JLfvgnHc2kaSUFaI'" subsys=daemon
    level=info msg="  --bpf-lb-maglev-table-size='16381'" subsys=daemon
    level=info msg="  --bpf-lb-map-max='65536'" subsys=daemon
    level=info msg="  --bpf-lb-mode='snat'" subsys=daemon loadbalance模式SNAT
    level=info msg="  --bpf-map-dynamic-size-ratio='0.0025'" subsys=daemon
    level=info msg="  --bpf-nat-global-max='524288'" subsys=daemon
    level=info msg="  --bpf-neigh-global-max='524288'" subsys=daemon
    level=info msg="  --bpf-policy-map-max='16384'" subsys=daemon
    level=info msg="  --bpf-root=''" subsys=daemon
    level=info msg="  --bpf-sock-rev-map-max='262144'" subsys=daemon
    level=info msg="  --certificates-directory='/var/run/cilium/certs'" subsys=daemon
    level=info msg="  --cgroup-root='/run/cilium/cgroupv2'" subsys=daemon
    level=info msg="  --cluster-id=''" subsys=daemon
    level=info msg="  --cluster-name='default'" subsys=daemon
    level=info msg="  --clustermesh-config='/var/lib/cilium/clustermesh/'" subsys=daemon
    level=info msg="  --cmdref=''" subsys=daemon
    level=info msg="  --config=''" subsys=daemon
    level=info msg="  --config-dir='/tmp/cilium/config-map'" subsys=daemon
    level=info msg="  --conntrack-gc-interval='0s'" subsys=daemon
    level=info msg="  --crd-wait-timeout='5m0s'" subsys=daemon
    level=info msg="  --datapath-mode='veth'" subsys=daemon
    level=info msg="  --debug='false'" subsys=daemon
    level=info msg="  --debug-verbose=''" subsys=daemon
    level=info msg="  --device=''" subsys=daemon
    level=info msg="  --devices=''" subsys=daemon
    level=info msg="  --direct-routing-device=''" subsys=daemon
    level=info msg="  --disable-cnp-status-updates='true'" subsys=daemon
    level=info msg="  --disable-conntrack='false'" subsys=daemon
    level=info msg="  --disable-endpoint-crd='false'" subsys=daemon
    level=info msg="  --disable-envoy-version-check='false'" subsys=daemon
    level=info msg="  --disable-iptables-feeder-rules=''" subsys=daemon
    level=info msg="  --dns-max-ips-per-restored-rule='1000'" subsys=daemon
    level=info msg="  --egress-masquerade-interfaces=''" subsys=daemon
    level=info msg="  --egress-multi-home-ip-rule-compat='false'" subsys=daemon
    level=info msg="  --enable-auto-protect-node-port-range='true'" subsys=daemon
    level=info msg="  --enable-bandwidth-manager='false'" subsys=daemon
    level=info msg="  --enable-bpf-clock-probe='true'" subsys=daemon
    level=info msg="  --enable-bpf-masquerade='true'" subsys=daemon
    level=info msg="  --enable-bpf-tproxy='false'" subsys=daemon
    level=info msg="  --enable-endpoint-health-checking='true'" subsys=daemon
    level=info msg="  --enable-endpoint-routes='true'" subsys=daemon
    level=info msg="  --enable-external-ips='true'" subsys=daemon
    level=info msg="  --enable-health-check-nodeport='true'" subsys=daemon
    level=info msg="  --enable-health-checking='true'" subsys=daemon
    level=info msg="  --enable-host-firewall='false'" subsys=daemon
    level=info msg="  --enable-host-legacy-routing='false'" subsys=daemon 关闭传统主机路由模式,但endpointRoutes模式与eBPF会冲突,i dont know
    level=info msg="  --enable-host-port='true'" subsys=daemon
    level=info msg="  --enable-host-reachable-services='false'" subsys=daemon
    level=info msg="  --enable-hubble='true'" subsys=daemon
    level=info msg="  --enable-identity-mark='true'" subsys=daemon
    level=info msg="  --enable-ip-masq-agent='false'" subsys=daemon
    level=info msg="  --enable-ipsec='false'" subsys=daemon
    level=info msg="  --enable-ipv4='true'" subsys=daemon
    level=info msg="  --enable-ipv4-fragment-tracking='true'" subsys=daemon
    level=info msg="  --enable-ipv6='false'" subsys=daemon
    level=info msg="  --enable-ipv6-ndp='false'" subsys=daemon
    level=info msg="  --enable-k8s-api-discovery='false'" subsys=daemon
    level=info msg="  --enable-k8s-endpoint-slice='true'" subsys=daemon
    level=info msg="  --enable-k8s-event-handover='false'" subsys=daemon
    level=info msg="  --enable-l7-proxy='true'" subsys=daemon
    level=info msg="  --enable-local-node-route='true'" subsys=daemon
    level=info msg="  --enable-local-redirect-policy='false'" subsys=daemon
    level=info msg="  --enable-monitor='true'" subsys=daemon
    level=info msg="  --enable-node-port='false'" subsys=daemon
    level=info msg="  --enable-policy='default'" subsys=daemon
    level=info msg="  --enable-remote-node-identity='true'" subsys=daemon
    level=info msg="  --enable-selective-regeneration='true'" subsys=daemon
    level=info msg="  --enable-session-affinity='true'" subsys=daemon
    level=info msg="  --enable-svc-source-range-check='true'" subsys=daemon
    level=info msg="  --enable-tracing='false'" subsys=daemon
    level=info msg="  --enable-well-known-identities='false'" subsys=daemon
    level=info msg="  --enable-xt-socket-fallback='true'" subsys=daemon
    level=info msg="  --encrypt-interface=''" subsys=daemon
    level=info msg="  --encrypt-node='false'" subsys=daemon
    level=info msg="  --endpoint-interface-name-prefix='lxc+'" subsys=daemon
    level=info msg="  --endpoint-queue-size='25'" subsys=daemon
    level=info msg="  --endpoint-status=''" subsys=daemon
    level=info msg="  --envoy-log=''" subsys=daemon
    level=info msg="  --exclude-local-address=''" subsys=daemon
    level=info msg="  --fixed-identity-mapping='map[]'" subsys=daemon
    level=info msg="  --flannel-master-device=''" subsys=daemon
    level=info msg="  --flannel-uninstall-on-exit='false'" subsys=daemon
    level=info msg="  --force-local-policy-eval-at-source='true'" subsys=daemon
    level=info msg="  --gops-port='9890'" subsys=daemon
    level=info msg="  --host-reachable-services-protos='tcp,udp'" subsys=daemon
    level=info msg="  --http-403-msg=''" subsys=daemon
    level=info msg="  --http-idle-timeout='0'" subsys=daemon
    level=info msg="  --http-max-grpc-timeout='0'" subsys=daemon
    level=info msg="  --http-normalize-path='true'" subsys=daemon
    level=info msg="  --http-request-timeout='3600'" subsys=daemon
    level=info msg="  --http-retry-count='3'" subsys=daemon
    level=info msg="  --http-retry-timeout='0'" subsys=daemon
    level=info msg="  --hubble-disable-tls='false'" subsys=daemon
    level=info msg="  --hubble-event-queue-size='0'" subsys=daemon
    level=info msg="  --hubble-flow-buffer-size='4095'" subsys=daemon
    level=info msg="  --hubble-listen-address=':4244'" subsys=daemon
    level=info msg="  --hubble-metrics=''" subsys=daemon
    level=info msg="  --hubble-metrics-server=''" subsys=daemon
    level=info msg="  --hubble-socket-path='/var/run/cilium/hubble.sock'" subsys=daemon
    level=info msg="  --hubble-tls-cert-file='/var/lib/cilium/tls/hubble/server.crt'" subsys=daemon
    level=info msg="  --hubble-tls-client-ca-files='/var/lib/cilium/tls/hubble/client-ca.crt'" subsys=daemon
    level=info msg="  --hubble-tls-key-file='/var/lib/cilium/tls/hubble/server.key'" subsys=daemon
    level=info msg="  --identity-allocation-mode='crd'" subsys=daemon
    level=info msg="  --identity-change-grace-period='5s'" subsys=daemon
    level=info msg="  --install-iptables-rules='true'" subsys=daemon
    level=info msg="  --ip-allocation-timeout='2m0s'" subsys=daemon
    level=info msg="  --ip-masq-agent-config-path='/etc/config/ip-masq-agent'" subsys=daemon
    level=info msg="  --ipam='kubernetes'" subsys=daemon
    level=info msg="  --ipsec-key-file=''" subsys=daemon
    level=info msg="  --iptables-lock-timeout='5s'" subsys=daemon
    level=info msg="  --iptables-random-fully='false'" subsys=daemon
    level=info msg="  --ipv4-node='auto'" subsys=daemon
    level=info msg="  --ipv4-pod-subnets=''" subsys=daemon
    level=info msg="  --ipv4-range='auto'" subsys=daemon
    level=info msg="  --ipv4-service-loopback-address='169.254.42.1'" subsys=daemon
    level=info msg="  --ipv4-service-range='auto'" subsys=daemon
    level=info msg="  --ipv6-cluster-alloc-cidr='f00d::/64'" subsys=daemon
    level=info msg="  --ipv6-mcast-device=''" subsys=daemon
    level=info msg="  --ipv6-node='auto'" subsys=daemon
    level=info msg="  --ipv6-pod-subnets=''" subsys=daemon
    level=info msg="  --ipv6-range='auto'" subsys=daemon
    level=info msg="  --ipv6-service-range='auto'" subsys=daemon
    level=info msg="  --ipvlan-master-device='undefined'" subsys=daemon
    level=info msg="  --join-cluster='false'" subsys=daemon
    level=info msg="  --k8s-api-server=''" subsys=daemon
    level=info msg="  --k8s-force-json-patch='false'" subsys=daemon
    level=info msg="  --k8s-heartbeat-timeout='30s'" subsys=daemon
    level=info msg="  --k8s-kubeconfig-path=''" subsys=daemon
    level=info msg="  --k8s-namespace='kube-system'" subsys=daemon
    level=info msg="  --k8s-require-ipv4-pod-cidr='false'" subsys=daemon
    level=info msg="  --k8s-require-ipv6-pod-cidr='false'" subsys=daemon
    level=info msg="  --k8s-service-cache-size='128'" subsys=daemon
    level=info msg="  --k8s-service-proxy-name=''" subsys=daemon
    level=info msg="  --k8s-sync-timeout='3m0s'" subsys=daemon
    level=info msg="  --k8s-watcher-endpoint-selector='metadata.name!=kube-scheduler,metadata.name!=kube-controller-manager,metadata.name!=etcd-operator,metadata.name!=gcp-controller-manager'" subsys=daemon
    level=info msg="  --k8s-watcher-queue-size='1024'" subsys=daemon
    level=info msg="  --keep-config='false'" subsys=daemon
    level=info msg="  --kube-proxy-replacement='strict'" subsys=daemon
    level=info msg="  --kube-proxy-replacement-healthz-bind-address=''" subsys=daemon
    level=info msg="  --kvstore=''" subsys=daemon
    level=info msg="  --kvstore-connectivity-timeout='2m0s'" subsys=daemon
    level=info msg="  --kvstore-lease-ttl='15m0s'" subsys=daemon
    level=info msg="  --kvstore-opt='map[]'" subsys=daemon
    level=info msg="  --kvstore-periodic-sync='5m0s'" subsys=daemon
    level=info msg="  --label-prefix-file=''" subsys=daemon
    level=info msg="  --labels=''" subsys=daemon
    level=info msg="  --lib-dir='/var/lib/cilium'" subsys=daemon
    level=info msg="  --log-driver=''" subsys=daemon
    level=info msg="  --log-opt='map[]'" subsys=daemon
    level=info msg="  --log-system-load='false'" subsys=daemon
    level=info msg="  --masquerade='true'" subsys=daemon
    level=info msg="  --max-controller-interval='0'" subsys=daemon
    level=info msg="  --metrics=''" subsys=daemon
    level=info msg="  --monitor-aggregation='medium'" subsys=daemon
    level=info msg="  --monitor-aggregation-flags='all'" subsys=daemon
    level=info msg="  --monitor-aggregation-interval='5s'" subsys=daemon
    level=info msg="  --monitor-queue-size='0'" subsys=daemon
    level=info msg="  --mtu='0'" subsys=daemon
    level=info msg="  --nat46-range='0:0:0:0:0:FFFF::/96'" subsys=daemon
    level=info msg="  --native-routing-cidr='172.21.0.0/20'" subsys=daemon
    level=info msg="  --node-port-acceleration='disabled'" subsys=daemon
    level=info msg="  --node-port-algorithm='random'" subsys=daemon
    level=info msg="  --node-port-bind-protection='true'" subsys=daemon
    level=info msg="  --node-port-mode='hybrid'" subsys=daemon
    level=info msg="  --node-port-range='30000,32767'" subsys=daemon
    level=info msg="  --policy-audit-mode='false'" subsys=daemon
    level=info msg="  --policy-queue-size='100'" subsys=daemon
    level=info msg="  --policy-trigger-interval='1s'" subsys=daemon
    level=info msg="  --pprof='false'" subsys=daemon
    level=info msg="  --preallocate-bpf-maps='false'" subsys=daemon
    level=info msg="  --prefilter-device='undefined'" subsys=daemon
    level=info msg="  --prefilter-mode='native'" subsys=daemon
    level=info msg="  --prepend-iptables-chains='true'" subsys=daemon
    level=info msg="  --prometheus-serve-addr=''" subsys=daemon
    level=info msg="  --proxy-connect-timeout='1'" subsys=daemon
    level=info msg="  --proxy-prometheus-port='0'" subsys=daemon
    level=info msg="  --read-cni-conf=''" subsys=daemon
    level=info msg="  --restore='true'" subsys=daemon
    level=info msg="  --sidecar-istio-proxy-image='cilium/istio_proxy'" subsys=daemon
    level=info msg="  --single-cluster-route='false'" subsys=daemon
    level=info msg="  --skip-crd-creation='false'" subsys=daemon
    level=info msg="  --socket-path='/var/run/cilium/cilium.sock'" subsys=daemon
    level=info msg="  --sockops-enable='false'" subsys=daemon
    level=info msg="  --state-dir='/var/run/cilium'" subsys=daemon
    level=info msg="  --tofqdns-dns-reject-response-code='refused'" subsys=daemon
    level=info msg="  --tofqdns-enable-dns-compression='true'" subsys=daemon
    level=info msg="  --tofqdns-endpoint-max-ip-per-hostname='50'" subsys=daemon
    level=info msg="  --tofqdns-idle-connection-grace-period='0s'" subsys=daemon
    level=info msg="  --tofqdns-max-deferred-connection-deletes='10000'" subsys=daemon
    level=info msg="  --tofqdns-min-ttl='0'" subsys=daemon
    level=info msg="  --tofqdns-pre-cache=''" subsys=daemon
    level=info msg="  --tofqdns-proxy-port='0'" subsys=daemon
    level=info msg="  --tofqdns-proxy-response-max-delay='100ms'" subsys=daemon
    level=info msg="  --trace-payloadlen='128'" subsys=daemon
    level=info msg="  --tunnel='disabled'" subsys=daemon
    level=info msg="  --version='false'" subsys=daemon
    level=info msg="  --write-cni-conf-when-ready=''" subsys=daemon
    level=info msg="     _ _ _" subsys=daemon
    level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
    level=info msg="|  _| | | | | |     |" subsys=daemon
    level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
    level=info msg="Cilium 1.9.9 5bcf83c 2021-07-19T16:45:00-07:00 go version go1.15.14 linux/amd64" subsys=daemon
    level=info msg="cilium-envoy  version: 82a70d56bf324287ced3129300db609eceb21d10/1.17.3/Distribution/RELEASE/BoringSSL" subsys=daemon
    level=info msg="clang (10.0.0) and kernel (5.11.1) versions: OK!" subsys=linux-datapath
    level=info msg="linking environment: OK!" subsys=linux-datapath
    level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf
    level=info msg="Mounted cgroupv2 filesystem at /run/cilium/cgroupv2" subsys=cgroups
    level=info msg="Parsing base label prefixes from default label list" subsys=labels-filter
    level=info msg="Parsing additional label prefixes from user inputs: []" subsys=labels-filter
    level=info msg="Final label prefixes to be used for identity evaluation:" subsys=labels-filter
    level=info msg=" - reserved:.*" subsys=labels-filter
    level=info msg=" - :io.kubernetes.pod.namespace" subsys=labels-filter
    level=info msg=" - :io.cilium.k8s.namespace.labels" subsys=labels-filter
    level=info msg=" - :app.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:io.kubernetes" subsys=labels-filter
    level=info msg=" - !:kubernetes.io" subsys=labels-filter
    level=info msg=" - !:.*beta.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:k8s.io" subsys=labels-filter
    level=info msg=" - !:pod-template-generation" subsys=labels-filter
    level=info msg=" - !:pod-template-hash" subsys=labels-filter
    level=info msg=" - !:controller-revision-hash" subsys=labels-filter
    level=info msg=" - !:annotation.*" subsys=labels-filter
    level=info msg=" - !:etcd_node" subsys=labels-filter
    level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
    level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.5.0.0/16
    level=info msg="Initializing daemon" subsys=daemon
    level=info msg="Establishing connection to apiserver" host="https://apiserver.qiangyun.com:6443" subsys=k8s
    level=info msg="Connected to apiserver" subsys=k8s
    level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
    level=info msg="BPF host routing is incompatible with enable-endpoint-routes. Falling back to legacy host routing (enable-host-legacy-routing=true)." subsys=daemon 与eBPF冲突,在初始化是指定 --set bpf.hostRouting=true
    level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=10.1.0.5 mtu=1500 subsys=mtu
    level=info msg="Restored services from maps" failed=0 restored=11 subsys=service
    level=info msg="Reading old endpoints..." subsys=daemon
    level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
    level=info msg="Reusing previous DNS proxy port: 39451" subsys=daemon
    level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
    level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
    level=info msg="Retrieved node information from kubernetes node" nodeName=prod-k8s-cp1 subsys=k8s
    level=info msg="Received own node information from API server" ipAddr.ipv4=10.1.0.5 ipAddr.ipv6="<nil>" k8sNodeIP=10.1.0.5 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:prod-k8s-cp1 kubernetes.io/os:linux node-role.kubernetes.io/master: topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h]" nodeName=prod-k8s-cp1 subsys=k8s v4Prefix=172.21.0.0/24 v6Prefix="<nil>"
    level=info msg="Restored router IPs from node information" ipv4=172.21.0.85 ipv6="<nil>" subsys=k8s
    level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
    level=info msg="Using auto-derived devices to attach Loadbalancer, Host Firewall or Bandwidth Manager program" devices="[eth0]" directRoutingDevice=eth0 subsys=daemon
    level=info msg="Enabling k8s event listener" subsys=k8s-watcher
    level=info msg="Waiting until all pre-existing resources related to policy have been received" subsys=k8s-watcher
    level=info msg="Removing stale endpoint interfaces" subsys=daemon
    level=info msg="Skipping kvstore configuration" subsys=daemon
    level=info msg="Restored router address from node_config" file=/var/run/cilium/state/globals/node_config.h ipv4=172.21.0.85 ipv6="<nil>" subsys=node
    level=info msg="Initializing node addressing" subsys=daemon
    level=info msg="Initializing kubernetes IPAM" subsys=ipam v4Prefix=172.21.0.0/24 v6Prefix="<nil>"
    level=info msg="Restoring endpoints..." subsys=daemon
    level=info msg="Endpoints restored" failed=0 restored=1 subsys=daemon
    level=info msg="Addressing information:" subsys=daemon
    level=info msg="  Cluster-Name: default" subsys=daemon
    level=info msg="  Cluster-ID: 0" subsys=daemon
    level=info msg="  Local node-name: prod-k8s-cp1" subsys=daemon
    level=info msg="  Node-IPv6: <nil>" subsys=daemon
    level=info msg="  External-Node IPv4: 10.1.0.5" subsys=daemon
    level=info msg="  Internal-Node IPv4: 172.21.0.85" subsys=daemon
    level=info msg="  IPv4 allocation prefix: 172.21.0.0/24" subsys=daemon
    level=info msg="  IPv4 native routing prefix: 172.21.0.0/20" subsys=daemon
    level=info msg="  Loopback IPv4: 169.254.42.1" subsys=daemon
    level=info msg="  Local IPv4 addresses:" subsys=daemon
    level=info msg="  - 10.1.0.5" subsys=daemon
    level=info msg="  - 172.21.0.85" subsys=daemon
    level=info msg="Adding local node to cluster" node="{prod-k8s-cp1 default [{InternalIP 10.1.0.5} {CiliumInternalIP 172.21.0.85}] 172.21.0.0/24 <nil> 172.21.0.197 <nil> 0 local 0 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:prod-k8s-cp1 kubernetes.io/os:linux node-role.kubernetes.io/master: topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h] 6}" subsys=nodediscovery
    level=info msg="Creating or updating CiliumNode resource" node=prod-k8s-cp1 subsys=nodediscovery
    level=info msg="Successfully created CiliumNode resource" subsys=nodediscovery
    level=info msg="Annotating k8s node" subsys=daemon v4CiliumHostIP.IPv4=172.21.0.85 v4Prefix=172.21.0.0/24 v4healthIP.IPv4=172.21.0.197 v6CiliumHostIP.IPv6="<nil>" v6Prefix="<nil>" v6healthIP.IPv6="<nil>"
    level=info msg="Initializing identity allocator" subsys=identity-cache
    level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
    level=info msg="Setting up BPF datapath" bpfClockSource=ktime bpfInsnSet=v3 subsys=datapath-loader
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.ipv4.conf.all.rp_filter sysParamValue=0
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.unprivileged_bpf_disabled sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.timer_migration sysParamValue=0
    level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
    level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
    level=info msg="Adding new proxy port rules for cilium-dns-egress:39451" proxy port name=cilium-dns-egress subsys=proxy
    level=info msg="Serving cilium node monitor v1.2 API at unix:///var/run/cilium/monitor1_2.sock" subsys=monitor-agent
    level=info msg="Validating configured node address ranges" subsys=daemon
    level=info msg="Starting connection tracking garbage collector" subsys=daemon
    level=info msg="Starting IP identity watcher" subsys=ipcache
    level=info msg="Initial scan of connection tracking completed" subsys=ct-gc
    level=info msg="Regenerating restored endpoints" numRestored=1 subsys=daemon
    level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.025936718825527918 newInterval=7m30s subsys=map-ct
    level=info msg="Datapath signal listener running" subsys=signal
    level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3912 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Successfully restored endpoint. Scheduling regeneration" endpointID=3912 subsys=daemon
    level=info msg="Removed endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3610 identity=4 ipv4=172.21.0.71 ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Launching Cilium health daemon" subsys=daemon
    level=info msg="Launching Cilium health endpoint" subsys=daemon
    level=info msg="Started healthz status API server" address="127.0.0.1:9876" subsys=daemon
    level=info msg="Initializing Cilium API" subsys=daemon
    level=info msg="Daemon initialization completed" bootstrapTime=5.687347691s subsys=daemon
    level=info msg="Serving cilium API at unix:///var/run/cilium/cilium.sock" subsys=daemon
    level=info msg="Configuring Hubble server" eventQueueSize=4096 maxFlows=4095 subsys=hubble
    level=info msg="Starting local Hubble server" address="unix:///var/run/cilium/hubble.sock" subsys=hubble
    level=info msg="Beginning to read perf buffer" startTime="2021-08-28 10:04:17.337903259 +0000 UTC m=+5.762296463" subsys=monitor-agent
    level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2454 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2454 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2454 identity=4 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint
    level=info msg="Compiled new BPF template" BPFCompilationTime=1.676219511s file-path=/var/run/cilium/state/templates/07d958f5310f668aa25992c4b03f0ab71d723a11/bpf_host.o subsys=datapath-loader
    level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
    level=info msg="Compiled new BPF template" BPFCompilationTime=1.348419572s file-path=/var/run/cilium/state/templates/f7d40533d0d45d623a9ad0f1855c105aed55472e/bpf_lxc.o subsys=datapath-loader
    level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3912 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=2454 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Restored endpoint" endpointID=3912 ipAddr="[ ]" subsys=endpoint
    level=info msg="Finished regenerating restored endpoints" regenerated=1 subsys=daemon total=1
    level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
    level=info msg="Waiting for Hubble server TLS certificate and key files to be created" subsys=hubble
  3. 查看enpointRoutes模式下cilium-agent的状态
    root@PROD-K8S-CP1:/home/cilium# cilium status --verbose
    KVStore:                Ok   Disabled
    Kubernetes:             Ok   1.18 (v1.18.5) [linux/amd64]
    Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
    KubeProxyReplacement:   Strict   [eth0 (Direct Routing)]
    Cilium:                 Ok   1.9.9 (v1.9.9-5bcf83c)
    NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
    Cilium health daemon:   Ok   
    IPAM:                   IPv4: 2/255 allocated from 172.21.0.0/24, 
    Allocated addresses:
      172.21.0.197 (health)
      172.21.0.85 (router)
    BandwidthManager:       Disabled
    Host Routing:           Legacy 注意主机路由模式
    Masquerading:           BPF   [eth0]   172.21.0.0/20
    Clock Source for BPF:   ktime
    Controller Status:      18/18 healthy
      Name                                  Last success   Last error   Count   Message
      cilium-health-ep                      11s ago        never        0       no error   
      dns-garbage-collector-job             17s ago        never        0       no error   
      endpoint-2454-regeneration-recovery   never          never        0       no error   
      endpoint-3912-regeneration-recovery   never          never        0       no error   
      k8s-heartbeat                         17s ago        never        0       no error   
      mark-k8s-node-as-available            24m12s ago     never        0       no error   
      metricsmap-bpf-prom-sync              2s ago         never        0       no error   
      neighbor-table-refresh                4m12s ago      never        0       no error   
      resolve-identity-2454                 4m11s ago      never        0       no error   
      restoring-ep-identity (3912)          24m12s ago     never        0       no error   
      sync-endpoints-and-host-ips           12s ago        never        0       no error   
      sync-lb-maps-with-k8s-services        24m12s ago     never        0       no error   
      sync-policymap-2454                   58s ago        never        0       no error   
      sync-policymap-3912                   58s ago        never        0       no error   
      sync-to-k8s-ciliumendpoint (2454)     11s ago        never        0       no error   
      sync-to-k8s-ciliumendpoint (3912)     2s ago         never        0       no error   
      template-dir-watcher                  never          never        0       no error   
      update-k8s-node-annotations           24m16s ago     never        0       no error   
    Proxy Status:   OK, ip 172.21.0.85, 0 redirects active on ports 10000-20000
    Hubble:         Ok   Current/Max Flows: 224/4096 (5.47%), Flows/s: 0.15   Metrics: Disabled
    KubeProxyReplacement Details:
      Status:              Strict
      Protocols:           TCP, UDP
      Devices:             eth0 (Direct Routing)
      Mode:                Hybrid 个人理解除非开启 DSR模式,否则单独设置没啥意义
      Backend Selection:   Random
      Session Affinity:    Enabled
      XDP Acceleration:    Disabled
      Services:
      - ClusterIP:      Enabled
      - NodePort:       Enabled (Range: 30000-32767) 
      - LoadBalancer:   Enabled 
      - externalIPs:    Enabled 
      - HostPort:       Enabled
    BPF Maps:   dynamic sizing: on (ratio: 0.002500)
      Name                          Size
      Non-TCP connection tracking   72407
      TCP connection tracking       144815
      Endpoint policy               65535
      Events                        4
      IP cache                      512000
      IP masquerading agent         16384
      IPv4 fragmentation            8192
      IPv4 service                  65536
      IPv6 service                  65536
      IPv4 service backend          65536
      IPv6 service backend          65536
      IPv4 service reverse NAT      65536
      IPv6 service reverse NAT      65536
      Metrics                       1024
      NAT                           144815
      Neighbor table                144815
      Global policy                 16384
      Per endpoint policy           65536
      Session affinity              65536
      Signal                        4
      Sockmap                       65535
      Sock reverse NAT              72407
      Tunnel                        65536
    Cluster health:              3/19 reachable   (2021-08-28T10:20:49Z)
      Name                       IP               Node        Endpoints
      prod-k8s-cp1 (localhost)   10.1.0.5         reachable   reachable
      prod-be-k8s-wn1            10.1.17.231      reachable   unreachable
      prod-be-k8s-wn2            10.1.17.232      reachable   unreachable
      prod-be-k8s-wn6            10.1.17.236      reachable   reachable
      prod-be-k8s-wn7            10.1.17.237      reachable   unreachable
      prod-be-k8s-wn8            10.1.17.238      reachable   unreachable
      prod-data-k8s-wn1          10.1.18.50       reachable   reachable
      prod-data-k8s-wn2          10.1.18.49       reachable   unreachable
      prod-data-k8s-wn3          10.1.18.51       reachable   unreachable
      prod-fe-k8s-wn1            10.1.16.221      reachable   unreachable
      prod-fe-k8s-wn2            10.1.16.222      reachable   unreachable
      prod-fe-k8s-wn3            10.1.16.223      reachable   unreachable
      prod-k8s-cp2               10.1.0.7         reachable   unreachable
      prod-k8s-cp3               10.1.0.6         reachable   unreachable
      prod-sys-k8s-wn1           10.1.0.8         reachable   unreachable
      prod-sys-k8s-wn2           10.1.0.9         reachable   unreachable
      prod-sys-k8s-wn3           10.1.0.11        reachable   unreachable
      prod-sys-k8s-wn4           10.1.0.10        reachable   unreachable
      prod-sys-k8s-wn5           10.1.0.12        reachable   unreachable
  4. 查看节点路由信息
    <root@PROD-K8S-CP1 ~># netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.1.0.253      0.0.0.0         UG        0 0          0 eth0
    10.1.0.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
    172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
    172.21.0.0      172.21.0.85     255.255.255.0   UG        0 0          0 cilium_host
    172.21.0.85     0.0.0.0         255.255.255.255 UH        0 0          0 cilium_host
    172.21.0.117    0.0.0.0         255.255.255.255 UH        0 0          0 lxc_health
    
    <root@PROD-DATA-K8S-WN1 ~># netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.1.18.253     0.0.0.0         UG        0 0          0 eth0
    10.1.18.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
    172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
    172.21.13.0     172.21.13.25    255.255.255.0   UG        0 0          0 cilium_host
    172.21.13.25    0.0.0.0         255.255.255.255 UH        0 0          0 cilium_host
    172.21.13.73    0.0.0.0         255.255.255.255 UH        0 0          0 lxc_health<root@PROD-FE-K8S-WN1 ~># netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.1.16.253     0.0.0.0         UG        0 0          0 eth0
    10.1.16.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
    172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
    172.21.9.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.9.173    0.0.0.0         255.255.255.255 UH        0 0          0 lxc_health
    172.21.9.225    0.0.0.0         255.255.255.255 UH        0 0          0 cilium_host

    <root@PROD-BE-K8S-WN6 ~># netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 10.1.17.253 0.0.0.0 UG 0 0 0 eth0 10.1.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.21.12.64 172.21.12.86 255.255.255.192 UG 0 0 0 cilium_host 172.21.12.74 0.0.0.0 255.255.255.255 UH 0 0 0 lxc_health 172.21.12.80 0.0.0.0 255.255.255.255 UH 0 0 0 lxc8de3adfa749f 172.21.12.86 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host 172.21.12.88 0.0.0.0 255.255.255.255 UH 0 0 0 lxcc1a4ab58fd8d 172.21.12.125 0.0.0.0 255.255.255.255 UH 0 0 0 lxcc8ea1535db0e

    # 从上面看出都由endpoint为单位独立路由
  5. 测试Pod网络连通性(路由配置好,网络必达)
posted @ 2021-08-28 15:07  MacoPlus  阅读(4994)  评论(0编辑  收藏  举报