namespace 网络故障排查命令 --nsenter + nc命令探测目标端口
nsenter
nsenter命令是一个可以在指定进程的命令空间下运行指定程序的命令，它位于util-linux包中。
[root@k8s-master ~]#
[root@k8s-master ~]# nsenter --help
用法：
nsenter [options] <program> [<argument>...]
Run a program with namespaces of other processes.
选项：
-t, --target <pid> 要获取名字空间的目标进程
-m, --mount[=<file>] enter mount namespace
-u, --uts[=<file>] enter UTS namespace (hostname etc)
-i, --ipc[=<file>] enter System V IPC namespace
-n, --net[=<file>] enter network namespace
-p, --pid[=<file>] enter pid namespace
-U, --user[=<file>] enter user namespace
-S, --setuid <uid> set uid in entered namespace
-G, --setgid <gid> set gid in entered namespace
--preserve-credentials do not touch uids or gids
-r, --root[=<dir>] set the root directory
-w, --wd[=<dir>] set the working directory
-F, --no-fork 执行 <程序> 前不 fork
-Z, --follow-context set SELinux context according to --target PID
-h, --help 显示此帮助并退出
-V, --version 输出版本信息并退出
##举例
#查询k8s_coredns容器的namespace的Pid
[root@k8s-master ~]#
[root@k8s-master ~]# docker inspect 2b0615c490b5 |grep Pid
"Pid": 4379,
"PidMode": "",
"PidsLimit": null,
[root@k8s-master ~]#
#通过nsenter命令进入namespace
[root@k8s-master ~]# nsenter -n -t 4379
nc
版本参数简介
语 法：nc [-hlnruz][-g<网关...>][-G<指向器数目>][-i<延迟秒数>][-o<输出文件>][-p<通信端口>][-s<来源地址>][-v...][-w<超时秒数>][主机名称][通信端口...]
补充说明：执行本指令可设置路由器的相关参数。
参 数：
-g<网关> 设置路由器跃程通信网关，最多可设置8个。
-G<指向器数目> 设置来源路由指向器，其数值为4的倍数。
-h 在线帮助。
-i<延迟秒数> 设置时间间隔，以便传送信息及扫描通信端口。
-l 使用监听模式，管控传入的资料。
-n 直接使用IP地址，而不通过域名服务器。
-o<输出文件> 指定文件名称，把往来传输的数据以16进制字码倾倒成该文件保存。
-p<通信端口> 设置本地主机使用的通信端口。
-r 乱数指定本地与远端主机的通信端口。
-s<来源地址> 设置本地主机送出数据包的IP地址。
-u 使用UDP传输协议。
-v 显示指令执行过程。
-w<超时秒数> 设置等待连线的时间。
-z 使用0输入/输出模式，只在扫描通信端口时使用。
#举例
for i in $(seq 1 100) ; do nc -w 0.1 -i 0.1 192.168.75.200 80 ;done
安装
yum install -y nc
#应用
for i in $(seq 1 100) ; do nc -w 0.1 -i 0.1 192.168.75.200 80 ;done
示例
#查询所有pod
[root@k8s-master ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
istio-system istio-egressgateway-7d6f9f54d7-62r5b 0/1 Running 0 16h 10.244.169.131 k8s-node2 <none> <none>
istio-system istio-ingressgateway-5d95b48945-hj9nb 0/1 Running 0 16h 10.244.169.132 k8s-node2 <none> <none>
istio-system istiod-d8576dfdf-4ktp5 1/1 Running 0 16h 10.244.36.69 k8s-node1 <none> <none>
kube-system calico-kube-controllers-cf4844b67-rzg4x 1/1 Running 5 (69d ago) 170d 10.244.235.209 k8s-master <none> <none>
kube-system calico-node-7vkgm 1/1 Running 6 (69d ago) 173d 192.168.75.200 k8s-master <none> <none>
kube-system calico-node-pl5t8 1/1 Running 3 (153d ago) 173d 192.168.75.202 k8s-node2 <none> <none>
kube-system calico-node-r2b8v 1/1 Running 4 (96d ago) 173d 192.168.75.201 k8s-node1 <none> <none>
kube-system coredns-7f6cbbb7b8-hlq65 1/1 Running 5 (69d ago) 170d 10.244.235.208 k8s-master <none> <none>
kube-system coredns-7f6cbbb7b8-mrdtr 1/1 Running 5 (69d ago) 170d 10.244.235.210 k8s-master <none> <none>
kube-system etcd-k8s-master 1/1 Running 6 (69d ago) 173d 192.168.75.200 k8s-master <none> <none>
kube-system kube-apiserver-k8s-master 1/1 Running 6 (69d ago) 173d 192.168.75.200 k8s-master <none> <none>
kube-system kube-controller-manager-k8s-master 1/1 Running 6 (69d ago) 173d 192.168.75.200 k8s-master <none> <none>
kube-system kube-proxy-7xjnw 1/1 Running 4 (96d ago) 173d 192.168.75.201 k8s-node1 <none> <none>
kube-system kube-proxy-gpz69 1/1 Running 6 (69d ago) 173d 192.168.75.200 k8s-master <none> <none>
kube-system kube-proxy-scpz6 1/1 Running 3 (153d ago) 173d 192.168.75.202 k8s-node2 <none> <none>
kube-system kube-scheduler-k8s-master 1/1 Running 6 (69d ago) 173d 192.168.75.200 k8s-master <none> <none>
#查询coredns容器
[root@k8s-master ~]# docker ps | grep coredns
2b0615c490b5 8d147537fb7d "/coredns -conf /etc…" 2 months ago Up 2 months k8s_coredns_coredns-7f6cbbb7b8-mrdtr_kube-system_21144da1-2c88-4f13-b806-609e6f91af93_5
26714d1a6fe9 registry.aliyuncs.com/google_containers/pause:3.5 "/pause" 2 months ago Up 2 months k8s_POD_coredns-7f6cbbb7b8-mrdtr_kube-system_21144da1-2c88-4f13-b806-609e6f91af93_5
d4d6b7ad96b2 8d147537fb7d "/coredns -conf /etc…" 2 months ago Up 2 months k8s_coredns_coredns-7f6cbbb7b8-hlq65_kube-system_f05043b5-3204-4afd-923c-36eba2e8aff0_5
e8843bf71b91 registry.aliyuncs.com/google_containers/pause:3.5 "/pause" 2 months ago Up 2 months k8s_POD_coredns-7f6cbbb7b8-hlq65_kube-system_f05043b5-3204-4afd-923c-36eba2e8aff0_5
[root@k8s-master ~]#
#查询k8s_coredns容器的namespace的Pid
[root@k8s-master ~]#
[root@k8s-master ~]# docker inspect 2b0615c490b5 |grep Pid
"Pid": 4379,
"PidMode": "",
"PidsLimit": null,
[root@k8s-master ~]#
#通过nsenter命令进入namespace
[root@k8s-master ~]# nsenter -n -t 4379
[root@k8s-master ~]#
[root@k8s-master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether ba:b5:a5:0f:b8:8b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.235.210/32 brd 10.244.235.210 scope global eth0
valid_lft forever preferred_lft forever
4: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
[root@k8s-master ~]#
[root@k8s-master ~]#
#排查问题
[root@k8s-master ~]# ping 10.96.0.10
PING 10.96.0.10 (10.96.0.10) 56(84) bytes of data.
^C
--- 10.96.0.10 ping statistics ---
140 packets transmitted, 0 received, 100% packet loss, time 142334ms
[root@k8s-master ~]# ping baidu.com
PING baidu.com (220.181.38.251) 56(84) bytes of data.
64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=1 ttl=127 time=28.2 ms
64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=2 ttl=127 time=181 ms
64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=3 ttl=127 time=21.2 ms
^C
--- baidu.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 21.221/76.952/181.362/73.885 ms
[root@k8s-master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether ba:b5:a5:0f:b8:8b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.235.210/32 brd 10.244.235.210 scope global eth0
valid_lft forever preferred_lft forever
4: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
[root@k8s-master ~]#
[root@k8s-master ~]#
#通过nc命令探测目标端口
[root@k8s-master ~]# for i in $(seq 1 100) ; do nc -w 0.1 -i 0.1 192.168.75.200 80 ;done
Ncat: Connection refused.
Ncat: Connection refused.
Ncat: Connection refused.
Ncat: Connection refused.
^C
[root@k8s-master ~]# for i in $(seq 1 100) ; do nc -w 0.1 -i 0.1 192.168.75.200 22 ;done
SSH-2.0-OpenSSH_7.4
Ncat: Idle timeout expired (100 ms).
SSH-2.0-OpenSSH_7.4
Ncat: Idle timeout expired (100 ms).
SSH-2.0-OpenSSH_7.4
Ncat: Idle timeout expired (100 ms).
SSH-2.0-OpenSSH_7.4
Ncat: Idle timeout expired (100 ms).
SSH-2.0-OpenSSH_7.4
Ncat: Idle timeout expired (100 ms).
SSH-2.0-OpenSSH_7.4
Ncat: Idle timeout expired (100 ms).
SSH-2.0-OpenSSH_7.4
#登出namespace
[root@k8s-master ~]# exit
登出
[root@k8s-master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:76:aa:74 brd ff:ff:ff:ff:ff:ff
inet 192.168.75.200/24 brd 192.168.75.255 scope global noprefixroute ens37
valid_lft forever preferred_lft forever
inet6 fe80::929e:100:479c:6303/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:30:88:d7:e1 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: cali6b72856f7ac@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
5: cali296c6c0494c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
6: cali2c685e579df@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
7: tunl0@NONE: <NOARP> mtu 1480 qdisc noqueue state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
[root@k8s-master ~]#
