网络监控分析

实时流量监控

工具：iftop
功能：
- 基于网卡的实时流量监控工具
- 指定网卡的实时流量、端口连接信息、反向解析IP等
- 精确显示本机网络流量情况及网络内各主机与本机相互通信的流量聚合
作用：
- 检测流量异常的主机
- 迅速定位主机流量异常根源
部署：
- centos: yum install iftop -y
- ubuntu: apt install iftop -y
常用的参数：

-i 设定监测的网卡，如：# iftop -i eth1
-B   以bytes为单位显示流量(默认是bits)，如：# iftop -B
-n   显示主机信息为IP，不进行DNS反向解析如：# iftop -n
-N   仅显示连接的端口号，不显示端口对应的服务名称，如: # iftop -N
-F 显示特定网段的进出流量，如# iftop -F 10.10.1.0/24或# iftop -F 10.10.1.0/255.255.255.0
-p 以混杂模式运行；
-b   使流量图形条默认就显示;
-P   显示主机和端口信息;
-m 设置界面最上边的刻度的最大值，刻度分五个大段显示，例：# iftop -m 100M

常用的参数组合
- 查看总流量动态：iftop -B
- 查看特定网卡流量：iftop -B -i eth0
- 查看特定IP和端口流量：iftop -P -B -n
界面说明
- 顶部显示网卡带宽流量
- 中间分为左、中、右三部分
- 中间左默认为本地主机和端口
- 中间中为外部连接主机和端口
- 中间右表示本地和外部连接在2s、10s、40s内的平均流量
- 中间的<= =>这两个左右箭头，表示的是流量的方向
- TX：发送流量
- RX：接收流量
- TOTAL：总流量
- cum：从运行iftop到目前时间的发送、接收、和总数据流量
- peak：表示发送、接收、和总数据流量峰值
- rates：分别表示过去 2s、10s、40s 的平均流量

网络性能评估

工具：iperf
功能：
- 测量TCP、UDP网络带宽
- 测量网络质量，包括：延时抖动、数据包丢失率、最大传输单元等统计
作用：
- 定位网络瓶颈
部署：
- centos: yum install iperf3 -y
- ubuntu: apt install iperf3 -y
常用参数：

-s 将iperf以服务端模式启动，例如： iperf3 -s
-p 服务端启动端口，默认为5201,例如：iperf3 -s -p 35201
-D 将服务端作为守护进程运行，例如：iperf3 -s -D
-c 将iperf以客户端模式启动，例如：iperf3 -c {Server_IP}
-u 指定使用UDP协议，默认不设置为TCP
-t 指定传输数据包的总时间，默认10s
-P 指定客户端和服务端之间使用的线程数，默认为单线程
-R 切换数据发送、接收方向，默认客户端发送，服务端接收
-B 绑定一个主机地址或接口
-f 指定带宽输出单位，k|m|g|K|M|G，分别表示：Kbits|Mbits|Gbits|KBytes|MBytes|GBytes,默认Mbits
-i 打印报告时间间隔，默认1s
-F 以文件作为数据流进行带宽测试，例如：iperf3 -c {Server_IP} -F web-test.tar.gz

应用场景
- 测试TCP吞吐量：2线程TCP带宽测试，每3s输出一次，测试时间30s
  - 服务端：iperf3 -s -i 3
  - 客户端：iperf3 -c (server_ip) -t 30 -i 3 -P 2

# 服务端
[root@node199051 ~]# iperf3 -s -i 3
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.199.52, port 21374
[  5] local 192.168.199.51 port 5201 connected to 192.168.199.52 port 21376
[  7] local 192.168.199.51 port 5201 connected to 192.168.199.52 port 21378
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-3.00   sec   183 MBytes   512 Mbits/sec
[  7]   0.00-3.00   sec   146 MBytes   407 Mbits/sec
[SUM]   0.00-3.00   sec   329 MBytes   919 Mbits/sec
... 省略若干内容
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  30.00-30.04  sec  2.50 MBytes   515 Mbits/sec
[  7]  30.00-30.04  sec  2.03 MBytes   419 Mbits/sec
[SUM]  30.00-30.04  sec  4.53 MBytes   934 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-30.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-30.04  sec  1.76 GBytes   503 Mbits/sec                  receiver
[  7]   0.00-30.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  7]   0.00-30.04  sec  1.50 GBytes   430 Mbits/sec                  receiver
[SUM]   0.00-30.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[SUM]   0.00-30.04  sec  3.26 GBytes   933 Mbits/sec                  receiver
-----------------------------------------------------------

# 客户端
[root@node199052 ~]# iperf3 -c 192.168.199.51 -t 30 -i 3 -P 2
Connecting to host 192.168.199.51, port 5201
[  4] local 192.168.199.52 port 21376 connected to 192.168.199.51 port 5201
[  6] local 192.168.199.52 port 21378 connected to 192.168.199.51 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-3.00   sec   188 MBytes   524 Mbits/sec    0    450 KBytes
[  6]   0.00-3.00   sec   149 MBytes   418 Mbits/sec    5    327 KBytes
[SUM]   0.00-3.00   sec   337 MBytes   942 Mbits/sec    5
... 省略若干内容
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  21.00-24.00  sec   151 MBytes   423 Mbits/sec    0    344 KBytes
[  6]  21.00-24.00  sec   185 MBytes   518 Mbits/sec    0    404 KBytes
[SUM]  21.00-24.00  sec   337 MBytes   941 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  24.00-27.00  sec   175 MBytes   490 Mbits/sec    0    403 KBytes
[  6]  24.00-27.00  sec   157 MBytes   438 Mbits/sec    2    338 KBytes
[SUM]  24.00-27.00  sec   332 MBytes   929 Mbits/sec    2
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  27.00-30.00  sec   187 MBytes   524 Mbits/sec    0    420 KBytes
[  6]  27.00-30.00  sec   148 MBytes   415 Mbits/sec    0    354 KBytes
[SUM]  27.00-30.00  sec   336 MBytes   938 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec  1.76 GBytes   505 Mbits/sec   30             sender
[  4]   0.00-30.00  sec  1.76 GBytes   504 Mbits/sec                  receiver
[  6]   0.00-30.00  sec  1.50 GBytes   431 Mbits/sec   39             sender
[  6]   0.00-30.00  sec  1.50 GBytes   430 Mbits/sec                  receiver
[SUM]   0.00-30.00  sec  3.27 GBytes   935 Mbits/sec   69             sender
[SUM]   0.00-30.00  sec  3.26 GBytes   934 Mbits/sec                  receiver

iperf Done.

# 可以看到总带宽930Mbits/sec左右，接近千兆虚拟网卡的性能

测试UDP丢包和延迟：通过UDP传输1G数据
- 服务端：iperf3 -s -i 3
- 客户端：iperf3 -c (server_ip) -u -b 1G -f M -i 3

# 服务端
[root@node199051 ~]# iperf3 -s -i 3
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.199.52, port 24418
[  5] local 192.168.199.51 port 5201 connected to 192.168.199.52 port 46538
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-3.00   sec   312 MBytes   873 Mbits/sec  0.015 ms  5736/231930 (2.5%)
[  5]   3.00-6.00   sec   331 MBytes   925 Mbits/sec  0.015 ms  1953/241556 (0.81%)
[  5]   6.00-9.00   sec   340 MBytes   950 Mbits/sec  0.030 ms  18/246002 (0.0073%)
[  5]   9.00-10.04  sec   118 MBytes   949 Mbits/sec  0.016 ms  0/85213 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec  0.016 ms  7707/804701 (0.96%)

# 客户端
[root@node199052 ~]# iperf3 -c 192.168.199.51 -u -b 1G -f M -i 3
Connecting to host 192.168.199.51, port 5201
[  4] local 192.168.199.52 port 46538 connected to 192.168.199.51 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-3.00   sec   325 MBytes   108 MBytes/sec  235172
[  4]   3.00-6.00   sec   334 MBytes   111 MBytes/sec  241618
[  4]   6.00-9.00   sec   340 MBytes   113 MBytes/sec  245988
[  4]   9.00-10.00  sec   113 MBytes   113 MBytes/sec  81945
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.09 GBytes   111 MBytes/sec  0.016 ms  7707/804701 (0.96%)
[  4] Sent 804701 datagrams

iperf Done.

# 测试丢包率为0.96%

网络探测审核

工具：nmap
功能：
- 网络主机发现
- 主机端口扫描
- 应用程序及版本侦测
- 操作系统及版本侦测
使用场景
- 典型用法：
  - 简单扫描：nmap
  - 详细扫描：nmap -A -T4 -v
    - -A 开启全面扫描
    - -T4 扫描等级，0~6个，等级越高扫描速度越快
    - -v 显示扫描细节

[root@rancher ~]# nmap 192.168.199.100

Starting Nmap 6.40 ( http://nmap.org ) at 2020-04-17 16:20 CST
Nmap scan report for 192.168.199.100
Host is up (0.00025s latency).
Not shown: 997 filtered ports
PORT     STATE SERVICE
53/tcp   open  domain
135/tcp  open  msrpc
3389/tcp open  ms-wbt-server
MAC Address: 00:0C:29:FA:B9:77 (VMware)

Nmap done: 1 IP address (1 host up) scanned in 4.86 seconds

# 由输出可知192.168.199.100主机处于up状态，开放了53、135、2289端口以及对应的服务，
# 还有对应ip地址的网卡mac地址信息。

[root@rancher ~]# nmap -A -T4 -v 192.168.199.100

Starting Nmap 6.40 ( http://nmap.org ) at 2020-04-17 16:25 CST
NSE: Loaded 110 scripts for scanning.
NSE: Script Pre-scanning.
Initiating ARP Ping Scan at 16:25
Scanning 192.168.199.100 [1 port]
Completed ARP Ping Scan at 16:25, 0.01s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 16:25
Completed Parallel DNS resolution of 1 host. at 16:25, 0.02s elapsed
Initiating SYN Stealth Scan at 16:25
Scanning 192.168.199.100 [1000 ports]
Discovered open port 135/tcp on 192.168.199.100
Discovered open port 3389/tcp on 192.168.199.100
Discovered open port 53/tcp on 192.168.199.100
Completed SYN Stealth Scan at 16:25, 4.94s elapsed (1000 total ports)
Initiating Service scan at 16:25
Scanning 3 services on 192.168.199.100
Completed Service scan at 16:26, 33.76s elapsed (3 services on 1 host)
Initiating OS detection (try #1) against 192.168.199.100
Retrying OS detection (try #2) against 192.168.199.100
NSE: Script scanning 192.168.199.100.
Initiating NSE at 16:26
NSOCK ERROR [43.8370s] mksock_bind_addr(): Bind to 0.0.0.0:25 failed (IOD #1): Address already in use (98)
Completed NSE at 16:26, 16.06s elapsed
Nmap scan report for 192.168.199.100
Host is up (0.00030s latency).
Not shown: 997 filtered ports
PORT     STATE SERVICE        VERSION
53/tcp   open  domain         Microsoft DNS
135/tcp  open  msrpc          Microsoft Windows RPC
3389/tcp open  ms-wbt-server?
MAC Address: 00:0C:29:FA:B9:77 (VMware)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: specialized
Running (JUST GUESSING): AVtech embedded (89%)
Aggressive OS guesses: AVtech Room Alert 26W environmental monitor (89%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop
TCP Sequence Prediction: Difficulty=254 (Good luck!)
IP ID Sequence Generation: Incremental
Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows

TRACEROUTE
HOP RTT     ADDRESS
1   0.30 ms 192.168.199.100

NSE: Script Post-scanning.
Read data files from: /usr/bin/../share/nmap
OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 59.91 seconds
           Raw packets sent: 2086 (96.904KB) | Rcvd: 20 (960B)

主机发现：nmap -sn -PE -PS22,80 -PU53 {host_ip}/
- -sn 只进行主机发现扫描，不进行端口扫描
- -PE 以发送ICMP echo报文的形式进行主机扫描
- -PS 以发送TCP SYN/ACK包的形式侦测主机信息，后面加典型服务端口
- -PU 以UDP的方式进行主机侦测，后面加典型服务端口

[root@rancher ~]# nmap -sn -PE -PS22,80 -PU53 192.168.199.100

Starting Nmap 6.40 ( http://nmap.org ) at 2020-04-17 17:00 CST
Nmap scan report for 192.168.199.100
Host is up (0.00025s latency).
MAC Address: 00:0C:29:FA:B9:77 (VMware)
Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds

# 批量侦测网络主机
[root@rancher ~]# nmap -sn -PE -PS22,80 -PU53 192.168.199.0/24

Starting Nmap 6.40 ( http://nmap.org ) at 2020-04-17 17:00 CST
Nmap scan report for 192.168.199.10
Host is up (0.00036s latency).
MAC Address: 00:50:56:9C:60:F9 (VMware)
Nmap scan report for 192.168.199.23
Host is up (0.00028s latency).
MAC Address: C8:5B:76:83:DC:93 (Unknown)
Nmap scan report for 192.168.199.34
Host is up (0.00033s latency).
MAC Address: 00:50:56:94:B3:F0 (VMware)
Nmap scan report for 192.168.199.48
Host is up (0.00016s latency).
MAC Address: 00:50:56:BD:B2:8D (VMware)
Nmap scan report for 192.168.199.49
... 中间忽略
Nmap scan report for 192.168.199.254
Host is up (0.00020s latency).
MAC Address: 00:15:17:2E:3B:24 (Intel Corporate)
Nmap scan report for rancher.gisuni.local (192.168.199.201)
Host is up.
Nmap done: 256 IP addresses (49 hosts up) scanned in 7.56 seconds

端口扫描：nmap -sU -sS -F
- -sU 使用UDP方式扫描UDP端口
- -sS 使用TCP SYN方式扫描TCP端口
- -F 使用快速扫描

[root@rancher ~]# nmap -sU -sS -F 192.168.199.211

Starting Nmap 6.40 ( http://nmap.org ) at 2020-04-17 17:06 CST
Nmap scan report for node199211 (192.168.199.211)
Host is up (0.00022s latency).
Not shown: 195 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
179/tcp  open  bgp
443/tcp  open  https
9100/tcp open  jetdirect
MAC Address: 52:54:00:6E:3C:2C (QEMU Virtual NIC)

Nmap done: 1 IP address (1 host up) scanned in 95.31 seconds

# 端口状态
open 开放
closed 关闭
filtered 防火墙屏蔽，无法进一步确定
unfiltered 没有屏蔽，是否开放需要进一步确认
open|filtered 状态不确定，可能开放，可能屏蔽
closed|filtered 状态不确定，可能关闭，可能屏蔽

软件版本侦测：nmap -sV

[centos@jumpserver ~]$ nmap -sV 192.168.198.68

Starting Nmap 6.40 ( http://nmap.org ) at 2020-04-17 17:26 CST
Nmap scan report for 192.168.198.68
Host is up (0.00039s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE     VERSION
22/tcp   open  ssh         OpenSSH 7.4 (protocol 2.0)
111/tcp  open  rpcbind     2-4 (RPC #100000)
1098/tcp open  rmiregistry Java RMI
8888/tcp open  http        Apache Tomcat/Coyote JSP engine 1.1

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.27 seconds

操作系统侦测：nmap -O --osscan-guess

[root@rancher ~]# nmap -O --osscan-guess 192.168.199.164

Starting Nmap 6.40 ( http://nmap.org ) at 2020-04-17 17:27 CST
Nmap scan report for 192.168.199.164
Host is up (0.00025s latency).
Not shown: 998 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
111/tcp open  rpcbind
MAC Address: 00:50:56:A4:09:F0 (VMware)
Aggressive OS guesses: Linux 2.6.32 - 3.9 (96%), Netgear DG834G WAP or Western Digital WD TV media player (96%), Linux 2.6.32 (95%), Linux 3.1 (95%), Linux 3.2 (95%), AXIS 210A or 211 Network Camera (Linux 2.6) (94%), Linux 2.6.32 - 2.6.35 (94%), Linux 2.6.32 -3.2 (94%), Linux 3.0 - 3.9 (93%), Linux 3.0 - 3.1 (93%)
No exact OS matches for host (If you know what OS is running on it, see http://nmap.org/submit/ ).
TCP/IP fingerprint:
OS:SCAN(V=6.40%E=4%D=4/17%OT=22%CT=1%CU=30888%PV=Y%DS=1%DC=D%G=Y%M=005056%T
OS:M=5E997695%P=x86_64-redhat-linux-gnu)SEQ(SP=100%GCD=1%ISR=107%TI=Z%TS=A)
OS:SEQ(SP=100%GCD=1%ISR=107%TI=Z%II=I%TS=A)SEQ(SP=100%GCD=1%ISR=107%TI=Z%CI
OS:=I%II=I%TS=A)OPS(O1=M5B4ST11NW7%O2=M5B4ST11NW7%O3=M5B4NNT11NW7%O4=M5B4ST
OS:11NW7%O5=M5B4ST11NW7%O6=M5B4ST11)WIN(W1=7120%W2=7120%W3=7120%W4=7120%W5=
OS:7120%W6=7120)ECN(R=Y%DF=Y%T=40%W=7210%O=M5B4NNSNW7%CC=Y%Q=)T1(R=Y%DF=Y%T
OS:=40%S=O%A=S+%F=AS%RD=0%Q=)T2(R=N)T3(R=N)T4(R=Y%DF=Y%T=40%W=0%S=A%A=Z%F=R
OS:%O=%RD=0%Q=)T5(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)T6(R=Y%DF=Y%T=
OS:40%W=0%S=A%A=Z%F=R%O=%RD=0%Q=)T7(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0
OS:%Q=)U1(R=Y%DF=N%T=40%IPL=164%UN=0%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)IE(R
OS:=Y%DFI=N%T=40%CD=S)

Network Distance: 1 hop

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 12.05 seconds

网络流量分析

工具：tcpdump
功能：
- 截获网络上的数据包并进行分析
- 支持针对网络层、协议、主机、网络或端口的过滤
- 提供and、or、not等逻辑语句来过滤信息
命令格式：

tcpdump [ -DenNqvX ] [ -c count ] [ -F file ] [ -i interface ] [ -r file ]
        [ -s snaplen ] [ -w file ] [ expression ]

命令选项：
- -i any：监听所有网络接口
- -i eth0：监听指定的网络接口eth0
- -D：列出所有可用的网络接口
- -n：不解析主机名
- -nn：不解析主机名和端口名
- -q：输出较少的信息
- -t：更便于阅读的时间戳输出
- -tttt：最便于阅读的时间戳输出
- -X：以 HEX 和 ASCII 模式输出数据包的内容
- -XX：与 -X 选项相同，同时还输出 ethernet 头
- -v, -vv, -vvv：输出更多数据包的信息
- -c：获取到指定数目的数据包后就停止
- -s：定义 snaplength (size) ，-s0 表示获取全部
- -S：输出绝对序列号
- -e：获取 ethernet 头信息
- -E：通过提供 key 来解密 IPSEC 流量
表达式类型
- Type（类型）选项包括 host 、net 和 port
- Direction（方向）选项包括 src 和 dst 以及它们的组合
- Proto（协议）包括 tcp 、udp 、ICMP 和 ah 等
使用示例
- 默认启动：tcpdump
  - 监视第一个网络接口(非lo口)上所有流通的数据包
  - 抓取的结果会非常多，滚动非常快