kube-proxy无法启用ipvs模式
1.明明kube-proxy设置了ipvs模式，确始终用的iptables模式
[root@iZcu9m4x2htrgoZ bin]# cat /etc/kubernetes/kube-proxy.config.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 10.80.178.135
clientConnection:
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.33.0.0/16
healthzBindAddress: 10.80.178.135:10256
hostnameOverride: 10.80.178.135
kind: KubeProxyConfiguration
metricsBindAddress: 10.80.178.135:10249
mode: "ipvs"
2.查看/var/log/messages发现kube-proxy启动的时候报了如下异常
can't determine whether to use ipvs proxy, error: error getting ipset version, error: exit status 127
3.解决方法
卸载ipset,重新安装ipset
[root@iZcu9m4x2htrgoZ bin]# rpm -qa | grep ipset
ipset-6.38-3.el7_6.x86_64
ipset-libs-6.38-3.el7_6.x86_64
[root@iZcu9m4x2htrgoZ bin]# rpm -e ipset-6.38-3.el7_6.x86_64 --nodeps
[root@iZcu9m4x2htrgoZ bin]# rpm -e ipset-libs-6.38-3.el7_6.x86_64 --nodeps
/sbin/ldconfig: File /lib64/libnetfilter_cttimeout.so.1.0.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_cttimeout.so.1 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_queue.so.1.3.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_cthelper.so.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libonig.so.2 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_cthelper.so.0.0.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_queue.so.1 is empty, not checked.
/sbin/ldconfig: File /lib64/libonig.so.2.0.0 is empty, not checked.
[root@iZcu9m4x2htrgoZ bin]# yum install ipset -y
Loaded plugins: aliases, fastestmirror, protectbase
openresty | 2.9 kB 00:00:00
https://openresty.org/package/centos/7/x86_64/repodata/repomd.xml: [Errno -1] pygpgme is not working so repomd.xml can not be verified for openresty
Trying other mirror.
Loading mirror speeds from cached hostfile
0 packages excluded due to repository protections
Resolving Dependencies
--> Running transaction check
---> Package ipset.x86_64 0:6.38-3.el7_6 will be installed
--> Processing Dependency: ipset-libs(x86-64) = 6.38-3.el7_6 for package: ipset-6.38-3.el7_6.x86_64
--> Processing Dependency: libipset.so.11(LIBIPSET_4.6)(64bit) for package: ipset-6.38-3.el7_6.x86_64
--> Processing Dependency: libipset.so.11(LIBIPSET_4.5)(64bit) for package: ipset-6.38-3.el7_6.x86_64
--> Processing Dependency: libipset.so.11(LIBIPSET_3.0)(64bit) for package: ipset-6.38-3.el7_6.x86_64
--> Processing Dependency: libipset.so.11(LIBIPSET_2.0)(64bit) for package: ipset-6.38-3.el7_6.x86_64
--> Processing Dependency: libipset.so.11(LIBIPSET_1.0)(64bit) for package: ipset-6.38-3.el7_6.x86_64
--> Processing Dependency: libipset.so.11()(64bit) for package: ipset-6.38-3.el7_6.x86_64
--> Running transaction check
---> Package ipset-libs.x86_64 0:6.38-3.el7_6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================================================================================================================
Package Arch Version Repository Size
==========================================================================================================================================================================================
Installing:
ipset x86_64 6.38-3.el7_6 updates 43 k
Installing for dependencies:
ipset-libs x86_64 6.38-3.el7_6 updates 50 k
Transaction Summary
==========================================================================================================================================================================================
Install 1 Package (+1 Dependent package)
Total download size: 94 k
Installed size: 251 k
Downloading packages:
(1/2): ipset-libs-6.38-3.el7_6.x86_64.rpm | 50 kB 00:00:00
(2/2): ipset-6.38-3.el7_6.x86_64.rpm | 43 kB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.3 MB/s | 94 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 2 pre-existing rpmdb problem(s), 'yum check' output follows:
firewalld-0.4.4.4-6.el7.noarch has missing requires of ipset
javapackages-tools-3.4.1-11.el7.noarch has missing requires of python-javapackages = ('0', '3.4.1', '11.el7')
Installing : ipset-libs-6.38-3.el7_6.x86_64 1/2
/sbin/ldconfig: File /lib64/libnetfilter_cttimeout.so.1.0.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_cttimeout.so.1 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_queue.so.1.3.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_cthelper.so.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libonig.so.2 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_cthelper.so.0.0.0 is empty, not checked.
/sbin/ldconfig: File /lib64/libnetfilter_queue.so.1 is empty, not checked.
/sbin/ldconfig: File /lib64/libonig.so.2.0.0 is empty, not checked.
Installing : ipset-6.38-3.el7_6.x86_64 2/2
Verifying : ipset-libs-6.38-3.el7_6.x86_64 1/2
Verifying : ipset-6.38-3.el7_6.x86_64 2/2
Installed:
ipset.x86_64 0:6.38-3.el7_6
Dependency Installed:
ipset-libs.x86_64 0:6.38-3.el7_6
Complete!
[root@iZcu9m4x2htrgoZ bin]# > /var/log/messages
[root@iZcu9m4x2htrgoZ bin]# service kube-proxy restart
Redirecting to /bin/systemctl restart kube-proxy.service
[root@iZcu9m4x2htrgoZ bin]# cat /var/log/messages
Apr 24 20:04:56 iZcu9m4x2htrgoZ systemd: Stopping Kubernetes Kube-Proxy Server...
Apr 24 20:04:56 iZcu9m4x2htrgoZ systemd: Started Kubernetes Kube-Proxy Server.
Apr 24 20:04:56 iZcu9m4x2htrgoZ systemd: Starting Kubernetes Kube-Proxy Server...
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.479876 17122 server_others.go:189] Using ipvs Proxier.
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: W0424 20:04:56.480402 17122 proxier.go:365] IPVS scheduler not specified, use rr by default
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.480689 17122 server_others.go:216] Tearing down inactive rules.
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.665941 17122 server.go:464] Version: v1.13.1
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.669974 17122 conntrack.go:52] Setting nf_conntrack_max to 131072
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.670214 17122 config.go:102] Starting endpoints config controller
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.670220 17122 config.go:202] Starting service config controller
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.670240 17122 controller_utils.go:1027] Waiting for caches to sync for service config controller
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.670230 17122 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.770452 17122 controller_utils.go:1034] Caches are synced for service config controller
Apr 24 20:04:56 iZcu9m4x2htrgoZ kube-proxy: I0424 20:04:56.770452 17122 controller_utils.go:1034] Caches are synced for endpoints config controller
Apr 24 20:04:56 iZcu9m4x2htrgoZ kernel: ip_set: protocol 6
[root@iZcu9m4x2htrgoZ bin]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 171.22.99.0:30306 rr
-> 171.22.8.3:6379 Masq 1 0 0
TCP 171.22.99.0:31161 rr
-> 171.22.46.4:webcache Masq 1 0 0
TCP 171.22.99.0:31370 rr
-> 171.22.45.7:webcache Masq 1 0 0
TCP 171.22.99.0:31459 rr
