kubernetes-v1.20.8二进制安装(九)-kube-proxy

kube-proxy 是为了解决外部网络能够访问集群中容器提供的应用服务而设计的，Proxy 运行在每个Node 上。

每创建一个 Service，kube-proxy 就会从 API Server 获取 Services 和 Endpoints 的配置信息，然后根据其配置信息在 Node 上启动一个 Proxy 的进程并监听相应的服务端口。

当接收到外部请求时，kube-proxy 会根据 Load Balancer 将请求分发到后端正确的容器处理。

kube-proxy 不但解决了同一宿主机相同服务端口冲突的问题，还提供了 Service 转发服务端口对外提供服务的能力。

1. 部署

1.1. 创建kube-proxy证书

cat > /opt/software/kubernetes/certs/kube-proxy-csr.json<<EOF
{
    "CN": "system:kube-proxy",
    "hosts": [
      "127.0.0.1",
      "*.k8s-host.com"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "GuangZhou",
            "L": "TianHe",
            "O": "k8s",
            "OU": "ops"
        }
    ]
}

EOF

• CN：指定该证书的 User 为 system:kube-proxy；
• 预定义的 RoleBinding system:node-proxier 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；
• 该证书只会被 kube-proxy 当做 client 证书使用，所以 hosts 字段为空；

1.2. 签发证书

cd /opt/software/kubernetes/certs/

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssl-json -bare kube-proxy

\cp kube-proxy.pem kube-proxy-key.pem /opt/software/kubernetes/node/certs/

1.3. 生成kubeconfig

设置集群参数

cd /opt/software/kubernetes/certs/
KUBE_CONFIG="/opt/software/kubernetes/node/kubeconfig/kube-proxy.kubeconfig"
KUBE_APISERVER="https://islb.k8s-host.com:6443"

kubectl config set-cluster kubernetes \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=${KUBE_CONFIG}

设置客户端认证参数

kubectl config set-credentials kube-proxy \
  --client-certificate=kube-proxy.pem \
  --client-key=kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=${KUBE_CONFIG}

设置上下文参数

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=${KUBE_CONFIG}

设置默认上下文

kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

1.4. 编写kube-proxy配置文件

不同节点修改 bindAddress metricsBindAddress metricsBindAddress hostnameOverride 参数

cat > /opt/software/kubernetes/node/config/kube-proxy-config.yaml<<EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 10.0.0.10
metricsBindAddress: 10.0.0.10:10249
healthzBindAddress: 10.0.0.10:10256
clientConnection:
  kubeconfig: /opt/software/kubernetes/kubeconfig/kube-proxy.kubeconfig
hostnameOverride: node01.k8s-host.com
clusterCIDR: 
mode: "ipvs"
iptables:
  masqueradeAll: false
ipvs:
  scheduler: rr
  excludeCIDRs: []
EOF

• bindAddress: 监听地址；
• clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件；
• clusterCIDR: kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量，指定--cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
• hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
• mode: 使用 ipvs 模式；

cat > /opt/software/kubernetes/node/config/kube-proxy-config.conf<<EOF
KUBE_KUBEPROXY_OPTS="--log-dir=/opt/software/kubernetes/logs \\
  --config=/opt/software/kubernetes/config/kube-proxy-config.yaml \\
  --logtostderr=false \\
  --v=2"
EOF

1.5. 编写service启动文件

cat > /opt/software/kubernetes/node/service/kube-proxy.service<<EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=-/opt/software/kubernetes/config/kube-proxy-config.conf
ExecStart=/opt/software/kubernetes/bin/kube-proxy \$KUBE_KUBEPROXY_OPTS
Restart=on-failure
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF

1.6. 复制文件至节点

hosts=(node01 node02 node03)
domain='k8s-host.com'
config_files=('config/kube-proxy-config.yaml' 'config/kube-proxy-config.conf')
cd /opt/software/kubernetes
for host in ${hosts[*]}
do
    scp -r node/{bin,certs,kubeconfig,service} ${host}.${domain}:/opt/software/kubernetes/
done

# 不可重复执行,否则覆盖修改的配置
for host in ${hosts[*]}
do
	for file in ${config_files[*]}
	do
    	scp -r node/${file} ${host}.${domain}:/opt/software/kubernetes/${file}
    done
done

1.7. 启动服务

hosts=(node01 node02 node03)
domain='k8s-host.com'
for host in ${hosts[*]}
do
	# 软链接
    ssh root@${host}.${domain} "ln -s /opt/software/kubernetes/service/kube-proxy.service  /usr/lib/systemd/system/kube-proxy.service "
    # 开机启动并启动服务
    ssh root@${host}.${domain} "systemctl daemon-reload && systemctl enable kube-proxy --now "
done

1.8. 验证集群

$ kubectl get nodes
NAME                      STATUS   ROLES    AGE     VERSION
node01.k8s-host.com   Ready    <none>   3d19h   v1.20.8
node02.k8s-host.com   Ready    <none>   3d19h   v1.20.8

问题

问题一: can't set sysctl net/ipv4/vs/conn_reuse_mode, kernel version must be at least 4.1

kube-proxy[17625]: E0708 11:21:25.791081   17625 proxier.go:389] can't set sysctl net/ipv4/vs/conn_reuse_mode, kernel version must be at least 4.1

修复: 内核版本较低, 建议内核版本升级为4.1以上

参考链接