基于queryperf 和 perftcpdns 的DNS压力测试
最近在AWS上安装了PPTP VPN 做代理，手机, pad 也可以无缝FQ，甚是开心。最近工作不太忙，研究一下缓存加速的调优。系统已经安装的nscd文件级的缓存和dnsmasq，cpu级的dns缓存。为什么说nscd是文件级的缓存呢，事情得先从dnsmasq说起，大家都知道nscd -g是可以看到他对hosts的缓存命中率的：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
|
[root@shanker:~06:24]#nscd -gnscd configuration: 1 server debug level 3m 55s server runtime 4 current number of threads 32 maximum number of threads 0 number of times clients had to wait no paranoia mode enabled 3600 restart internal 5 reload countpasswd cache: no cache is enabled yes cache is persistent yes cache is shared 0 suggested size 0 total data pool size 0 used data pool size 600 seconds time to live for positive entries 20 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 0 cache misses on positive entries 0 cache misses on negative entries 0% cache hit rate 0 current number of cached values 0 maximum number of cached values 0 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/passwd for changesgroup cache: no cache is enabled yes cache is persistent yes cache is shared 0 suggested size 0 total data pool size 0 used data pool size 3600 seconds time to live for positive entries 60 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 0 cache misses on positive entries 0 cache misses on negative entries 0% cache hit rate 0 current number of cached values 0 maximum number of cached values 0 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/group for changeshosts cache: yes cache is enabled yes cache is persistent yes cache is shared 211 suggested size 216064 total data pool size 120 used data pool size 3600 seconds time to live for positive entries 20 seconds time to live for negative entries 361 cache hits on positive entries 0 cache hits on negative entries 239 cache misses on positive entries 151 cache misses on negative entries 48% cache hit rate 1 current number of cached values 80 maximum number of cached values 2 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/hosts for changesservices cache: yes cache is enabled yes cache is persistent yes cache is shared 211 suggested size 216064 total data pool size 416 used data pool size 28800 seconds time to live for positive entries 20 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 3 cache misses on positive entries 1 cache misses on negative entries 0% cache hit rate 3 current number of cached values 3 maximum number of cached values 1 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/services for changesnetgroup cache: no cache is enabled yes cache is persistent yes cache is shared 0 suggested size 0 total data pool size 0 used data pool size 28800 seconds time to live for positive entries 20 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 0 cache misses on positive entries 0 cache misses on negative entries 0% cache hit rate 0 current number of cached values 0 maximum number of cached values 0 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/netgroup for changes |
然后nscd的缓存文件是存放在/var/cache/nscd 目录下，也可以使用lsof 查看。当我想看看dnsmasq的缓存效果的时候却发现他缓存的文件都是socket,所以我判断他的缓存类似于varnish 都存在于内存中的。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
[root@shanker:~06:25]#cd /proc/`pidof dnsmasq`/[root@shanker:/proc/2662406:26]#cd fd[root@shanker:/proc/26624/fd06:26]#lltotal 0lrwx------ 1 root root 64 Jan 14 12:59 9 -> socket:[211546]lrwx------ 1 root root 64 Jan 14 12:59 8 -> socket:[211545]lrwx------ 1 root root 64 Jan 14 12:59 7 -> socket:[211544]lrwx------ 1 root root 64 Jan 14 12:59 6 -> socket:[211543]lrwx------ 1 root root 64 Jan 14 12:59 5 -> socket:[211542]lrwx------ 1 root root 64 Jan 14 12:59 4 -> socket:[211541]lrwx------ 1 root root 64 Jan 14 12:59 3 -> socket:[211539]lrwx------ 1 root root 64 Jan 14 12:59 2 -> /dev/nulll-wx------ 1 root root 64 Jan 14 12:59 14 -> /var/log/dnsmasq.logl-wx------ 1 root root 64 Jan 14 12:59 13 -> pipe:[211553]lr-x------ 1 root root 64 Jan 14 12:59 12 -> pipe:[211553]lrwx------ 1 root root 64 Jan 14 12:59 11 -> socket:[211548]lrwx------ 1 root root 64 Jan 14 12:59 10 -> socket:[211547]lrwx------ 1 root root 64 Jan 14 12:59 1 -> /dev/nulllrwx------ 1 root root 64 Jan 14 12:59 0 -> /dev/null |
然后我们测算一下dnsmasq 查询的效率如何，在bind那套工具中有一套DNS性能测试的工具，去官网下载最新的bind
|
1
|
wget https://www.isc.org/downloads/file/bind-9-10-3-p2/?version=tar-gz |
解压完后到queryperf目录
|
1
2
3
4
|
cd bind-9.10.3-P2/contrib/queryperf/./configuremakecp queryperf /usr/bin/ |
然后perftcpdns也是同样,make 后将perftcpdns cp 到/usr/bin下面。
一 queryperf使用格式
queryperf [-d datafile] [-s server_addr] [-p port] [-q num_queries]
-d: 后面接上一个文件，文件的内容是用户对DNS的请求，一行为一条请求，所以为了测试，我们可以在里面写上几千几万条。
-s: DNS服务器地址
-p: DNS服务器端口
-q: 请求多少次
我在网上随便找了一些域名地址，保存到dnsrecord里。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
ns.bta.net.cn Ans.spt.net.cn Ans.cn.net Agjjline.bta.net.cn Alinedns.bta.net.cn Ans.guangzhou.gd.cn Adns.guangzhou.gd.cn Ans.sta.net.cn Ans-pd.online.sh.cn Ans.wuhan.net.cn Ans1.hbwhptt.net.cn Adns.zj.cninfo.net Ans.wuhan.net.cn Ans.zjnbptt.net.cn Ans.snnic.com Ans1.xaonline.com Ans.tpt.net.cn Ans.dcb.ln.cn Ans.lnpta.net.cn Adns.dl.lnpta.net.cn A |
然后复制成9999行，首先用本机来测试
|
1
|
queryperf -d query.txt -s 127.0.0.1 |
结果如下
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
Statistics: Parse input file: once Ended due to: reaching end of file Queries sent: 9999 queries Queries completed: 9999 queries Queries lost: 0 queries Queries delayed(?): 0 queries RTT max: 0.329734 sec RTT min: 0.000003 sec RTT average: 0.002490 sec RTT std deviation: 0.011374 sec RTT out of range: 0 queries Percentage completed: 100.00% Percentage lost: 0.00% Started at: Thu Jan 14 12:02:52 2016 Finished at: Thu Jan 14 12:03:18 2016 Ran for: 26.355231 seconds Queries per second: 379.393374 qps |
显示总共执行了9999次查询，每秒钟的查询是379.393374个，然后用Google的公告域名服务器测试
|
1
|
queryperf -d query.txt -s 8.8.8.8 |
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
Statistics: Parse input file: once Ended due to: reaching end of file Queries sent: 9999 queries Queries completed: 9999 queries Queries lost: 0 queries Queries delayed(?): 0 queries RTT max: 0.262356 sec RTT min: 0.001734 sec RTT average: 0.005838 sec RTT std deviation: 0.020113 sec RTT out of range: 0 queries Percentage completed: 100.00% Percentage lost: 0.00% Started at: Thu Jan 14 12:04:22 2016 Finished at: Thu Jan 14 12:16:18 2016 Ran for: 715.830576 seconds |
Queries per second: 13.968389 qps
结果每秒钟只有可怜的13.96次查询，也就是说如果本机做为一台服务器，没有dnsmasq做缓存的情况下，DNS查询效率低的可怜。
二 perftcpdns的使用
Google了半天没找到有价值的信息，还是看软件自己的使用帮助把：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
|
[root@shanker:~/bind-9.10.3-P2/contrib/perftcpdns06:56]#perftcpdns server is requiredperftcpdns [-huvX0] [-4|-6] [-r<rate>] [-t<report>] [-p<test-period>] [-n<num-request>]* [-d<lost-time>]* [-D<max-loss>]* [-T<template-file>] [-l<local-addr>] [-L<local-port>]* [-a<aggressiveness>] [-s<seed>] [-M<memory>] [-x<diagnostic-selector>] [-P<port>] serverThe server argument is the name/address of the DNS server to contact.Options:-0: Add EDNS0 option with DO flag.-4: TCP/IPv4 operation (default). This is incompatible with the -6 option.-6: TCP/IPv6 operation. This is incompatible with the -4 option.-a<aggressiveness>: When the target sending rate is not yet reached, control how many connections are initiated before the next pause.-d<lost-time>: Specify the time after which a connection or a query is treated as having been lost. The value is given in seconds and may contain a fractional component. The default is 1 second.-h: Print this help.-l<local-addr>: Specify the local hostname/address to use when communicating with the server.-L<local-port>: Specify the (minimal and maximal) local port number-M<memory>: Size of the tables (default 60000)-P<port>: Specify an alternate (i.e., not 53) port-r<rate>: Initiate <rate> TCP DNS connections per second. A periodic report is generated showing the number of exchanges which were not completed, as well as the average response latency. The program continues until interrupted, at which point a final report is generated.-s<seed>: Specify the seed for randomization, making it repeatable.-t<report>: Delay in seconds between two periodic reports.-T<template-file>: The name of a file containing the template to use as a stream of hexadecimal digits.-u: Use UDP in place of TCP.-v: Report the version number of this program.-X: change default template to get NXDOMAIN responses.-x<diagnostic-selector>: Include extended diagnostics in the output. <diagnostic-selector> is a string of single-keywords specifying the operations for which verbose output is desired. The selector keyletters are: * 'a': print the decoded command line arguments * 'e': print the exit reason * 'i': print rate processing details * 'T': when finished, print templatesStopping conditions:-D<max-loss>: Abort the test if more than <max-loss> connections or queries have been lost. If <max-loss> includes the suffix '%', it specifies a maximum percentage of losses before stopping. In this case, testing of the threshold begins after 10 connections/responses have been expected to be accepted/received.-n<num-request>: Initiate <num-request> transactions. No report is generated until all transactions have been initiated/waited-for, after which a report is generated and the program terminates.-p<test-period>: Send requests for the given test period, which is specified in the same manner as -d. This can be used as an alternative to -n, or both options can be given, in which case the testing is completed when either limit is reached.Errors:- locallimit: reached to local system limits when sending a message.- badconn: connection failed (from getsockopt(SO_ERROR))- collconn: connect() timed out- badsent: send() failed- callsent: timed out waiting from a response- recverr: recv() system call failed- tooshort: received a too short message- badid: the id mismatches between the query and the response- notresp: doesn't receive a responseRate stats:- loops: number of thread loop iterations- shortwait: no direct activity in a thread iteration- compconn: computed number of connect() calls- lateconn: connect() already dued when computing delay to the next oneExit status:The exit status is:0 on complete success.1 for a general error.2 if an error is found in the command line arguments.3 if there are no general failures in operation, but one or more exchanges are not successfully completed. |
用这个命令的前提是你的服务器是真正的DNS 服务器而不是缓存服务器，
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
perftcpdns -x aeiT -r 40000 8.8.8.8connect: 117669, sent: 117667, received: 60393embryonics: 2 (0.0%)drops: 57274 (48.7%)total losses: 57276 (48.7%)local limits: 0, bad connects: 0, connect timeouts: 2bad sends: 0, bad recvs: 0, recv timeouts: 57090too shorts: 13, bad IDs: 0, not responses: 0rcode counters: noerror: 1572, formerr: 0, servfail: 58821 nxdomain: 0, noimp: 0, refused: 0, others: 0rates: 241,241,124 (target 250)loops: 436434,117670,474085,462908shortwait: 0,356509,402606compconn: 117669, lateconn: 1badconn: 0, collconn: 2, recverr: 0, collsent: 57090memory: used(246) / allocated(60000)RTT: min/avg/max/stddev: 1.798/9.197/523.429/22.042 mslength = 0x1ccontent:00 00 01 00 00 01 00 00 00 00 00 00 05 69 63 616e 6e 04 6c 69 6e 6b 00 00 01 00 01 |
当我使用一下参数进行测试的时候发，系统提示too many files are open,ulimit 把打开文件数改成65535在继续测试，过一会发现之前打开的ssh窗口莫名的关闭，而且敲命名反应很卡，强制关掉perftcpdns，查看日志发现，很熟悉的一幕出现了：
|
1
2
3
4
5
6
|
Jan 15 03:00:59 shanker kernel: [223099.654142] nf_conntrack: table full, dropping packetJan 15 03:00:59 shanker kernel: [223099.654144] nf_conntrack: table full, dropping packetJan 15 03:01:05 shanker kernel: [223104.992102] net_ratelimit: 29728 callbacks suppressed Jan 15 03:01:05 shanker kernel: [223104.992128] nf_conntrack: table full, dropping packetJan 15 03:01:05 shanker kernel: [223104.992135] nf_conntrack: table full, dropping packetJan 15 03:01:05 shanker kernel: [223105.152351] nf_conntrack: table full, dropping packet |
想起以前处理过的紧急case，nf_conntrack table full，最直接的处理方式就是加大conntrack table 和 bucket的值：
net.netfilter.nf_conntrack_max = 196608
net.netfilter.nf_conntrack_buckets = 65534
如果你的服务器内存足够大而且只处理内网请求，扩大这个数值是理所当然的。Google了一下，有其他的治根的法式是用iptable raw 表，跳过记录，如果不依赖于iptables的话，我觉得更改内核参数的方式也是可行的。
参考文章：http://shanker.blog.51cto.com/1189689/1735341
