linux分析、诊断及调优必备的“杀器”之二

先说明下，之所以同类内容分成多篇文章，不是为了凑篇数，而是为了便于自己和大家阅读，下面继续：

7、sar

The sar command is used to collect, report, and save system activity information. The sar command consists of three applications: sar, which displays the data, and sa1 and sa2, which are used for collecting and storing the data. The sar tool features a wide range of options so be sure to check the man page for it. The sar utility is part of the sysstat package.

To accomplish this, add the lines to /etc/crontab.Keep in mind that a default cron job running sar daily is set up automatically after installing sar on your system.

Example of starting automatic log reporting with cron

-------------------------------------------------------------------------------------------------------------

# 8am-7pm activity reports every 10 minutes during weekdays.
*/10 8-18 * * 1-5 /usr/lib/sa/sa1 600 6 &
# 7pm-8am activity reports every an hour during weekdays.
0 19-7 * * 1-5 /usr/lib/sa/sa1 &
# Activity reports every an hour on Saturday and Sunday.
0 * * * 0,6 /usr/lib/sa/sa1 &
# Daily summary prepared at 19:05
5 19 * * * /usr/lib/sa/sa2 -A &

----------------------------------------------------------------------------------------------------------------

The raw data for the sar tool is stored under /var/log/sa/ where the various files represent the days of the respective month. To examine your results, select the weekday of the month and he requested performance data. For example, to display the network counters from the 21st, use the command sar -n DEV -f sa21 and pipe it to less as in the following example:

---------------------------------------------------------------------------------------------------------------

root# sar -n DEV -f sa21 | less
Linux 2.6.9-5.ELsmp (linux.itso.ral.ibm.com) 04/21/2005
12:00:01 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
12:10:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:10:01 AM eth0 1.80 0.00 247.89 0.00 0.00 0.00 0.00
12:10:01 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00

---------------------------------------------------------------------------------------------------------------

You can also use sar to run near-real-time reporting from the command line as the following example:
---------------------------------------------------------------------------------------------------------------

root# sar -u 3 10
Linux 2.4.21-9.0.3.EL (x232) 05/22/2004
02:10:40 PM CPU %user %nice %system %idle
02:10:43 PM all 0.00 0.00 0.00 100.00
02:10:46 PM all 0.33 0.00 0.00 99.67
02:10:49 PM all 0.00 0.00 0.00 100.00
02:10:52 PM all 7.14 0.00 18.57 74.29
02:10:55 PM all 71.43 0.00 28.57 0.00
02:10:58 PM all 0.00 0.00 100.00 0.00
02:11:01 PM all 0.00 0.00 0.00 0.00
02:11:04 PM all 0.00 0.00 100.00 0.00
02:11:07 PM all 50.00 0.00 50.00 0.00
02:11:10 PM all 0.00 0.00 100.00 0.00
Average: all 1.62 0.00 3.33 95.06

----------------------------------------------------------------------------------------------------------------

From the collected data, you see a detailed overview of CPU utilization (%user, %nice,%system, %idle), memory paging, network I/O and transfer statistics, process creation activity, activity for block devices, and interrupts/second over time.

8、mpstat

The mpstat command is used to report the activities of each of the available CPUs on a multiprocessor server. Global average activities among all CPUs are also reported. The mpstat utility is part of the sysstat package.for example:

root# mpstat -P ALL
Linux 2.6.9-5.ELsmp (linux.itso.ral.ibm.com) 04/22/2005
03:19:21 PM       CPU         %user         %nice         %system        %iowait            %irq              %soft              %idle              intr/s
03:19:21 PM       all              0.03             0.00            0.34                 0.06                  0.02               0.08                99.47             1124.22
03:19:21 PM       0                0.03             0.00            0.33                 0.03                  0.04               0.15                99.43              612.12
03:19:21 PM       1                0.03             0.00            0.36                 0.10                  0.01               0.01                99.51               512.09

 

root# mpstat -P ALL 1 2
Linux 2.6.9-5.ELsmp (linux.itso.ral.ibm.com) 04/22/2005
03:31:51 PM           CPU           %user             %nice            %system             %iowait                 %irq                   %soft                 %idle                      intr/s
03:31:52 PM           all                0.00                 0.00               0.00                      0.00                       0.00                    0.00                   100.00                   1018.81
03:31:52 PM           0                  0.00                 0.00               0.00                      0.00                       0.00                    0.00                   100.00                   991.09
03:31:52 PM           1                  0.00                 0.00               0.00                      0.00                       0.00                    0.00                    99.01                     27.72

Average:                  CPU           %user             %nice            %system             %iowait                 %irq                   %soft                 %idle                       intr/s
Average:                  all                0.00                 0.00                0.00                     0.00                        0.00                   0.00                   100.00                    1031.89
Average:                  0                  0.00                 0.00                0.00                     0.00                        0.00                   0.00                   100.00                    795.68
Average:                  1                  0.00                 0.00                0.00                     0.00                        0.00                   0.00                    99.67                      236.54

For the complete syntax of the mpstat command, issue:
mpstat -?

9、numastat

With Non-Uniform Memory Architecture (NUMA) systems such as the IBM System x 3950,NUMA architectures have become mainstream in enterprise data centers. However, NUMA systems introduce new challenges to the performance tuning process. Topics such as memory locality were of no interest until NUMA systems arrived. Luckily, Enterprise Linux distributions provide a tool for monitoring the behavior of NUMA architectures. The numastat command provides information about the ratio of local versus remote memory usage and the overall memory configuration of all nodes. Failed allocations of local memory, as displayed in the numa_miss column and allocations of remote memory (slower memory), as displayed in the numa_foreign column should be investigated. Excessive allocation of remote memory will increase system latency and likely decrease overall performance. Binding processes to a node with the memory map in the local RAM will most likely improve performance.for example:

root# numastat
                                node1                  node0
numa_hit              76557759           92126519
numa_miss         30772308           30827638
numa_foreign      30827638          30772308
interleave_hit       106507               103832
local_node           76502227           92086995
other_node          30827840            30867162

10、pmap

The pmap command reports the amount of memory that one or more processes are using. You can use this tool to determine which processes on the server are being allocated memory and whether this amount of memory is a cause of memory bottlenecks. For detailed information,use pmap -d option.

pmap -d <pid>

for example,

root# pmap -d 1
1: init [3]
Address                                Kbytes          Mode         Offset                                      Device                                  Mapping
0000000000400000          36                  r-x--            0000000000000000           0fd:00000                            init
0000000000508000          8                    rw---           0000000000008000           0fd:00000                            init
000000000050a000         132                rwx--           000000000050a000           000:00000                           [ anon ]
0000002a95556000          4                    rw---           0000002a95556000           000:00000                           [ anon ]
0000002a95574000          8                    rw---           0000002a95574000           000:00000                           [ anon ]
00000030c3000000          84                  r-x--            0000000000000000            0fd:00000                            ld-2.3.4.so
00000030c3114000          8                    rw---           0000000000014000            0fd:00000                            ld-2.3.4.so
00000030c3200000          1196             r-x--            0000000000000000            0fd:00000                            libc-2.3.4.so
00000030c332b000          1024             -----             000000000012b000            0fd:00000                           libc-2.3.4.so
00000030c342b000          8                    r----             000000000012b000            0fd:00000                           libc-2.3.4.so
00000030c342d000         12                  rw---            000000000012d000            0fd:00000                           libc-2.3.4.so
00000030c3430000         16                  rw---            00000030c3430000            000:00000                          [ anon ]
00000030c3700000          56                 r-x--             0000000000000000            0fd:00000                           libsepol.so.1
00000030c370e000         1020             -----              000000000000e000            0fd:00000                           libsepol.so.1
00000030c380d000          4                   rw---             000000000000d000           0fd:00000                           libsepol.so.1
00000030c380e000          32                 rw---             00000030c380e000           000:00000                          [ anon ]
00000030c4500000          56                 r-x--              0000000000000000           0fd:00000                           libselinux.so.1
00000030c450e000          1024            -----               000000000000e000           0fd:00000                           libselinux.so.1
00000030c460e000          4                   rw---              000000000000e000          0fd:00000                           libselinux.so.1
00000030c460f000           4                   rw---              00000030c460f000            000:00000                          [ anon ]
0000007fbfffc000               16                 rw---              0000007fbfffc000                000:00000                          [ stack ]
ffffffffff600000                       8192            -----                0000000000000000          000:00000                          [ anon ]
mapped: 12944K               writeable/private: 248K                  shared: 0K

For the complete syntax of the pmap command, issue:
pmap -?

11、netstat

常用选项：

-a Show all socket information
-r Show routing information
-i Show network interface statistics
-s Show network protocol statistics

例子：

root# netstat -natuw
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:2207 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:36285 127.0.0.1:12865 TIME_WAIT
tcp 0 0 10.0.0.5:37322 10.0.0.4:33932 TIME_WAIT
tcp 0 1 10.0.0.5:55351 10.0.0.4:33932 SYN_SENT
tcp 0 1 10.0.0.5:55350 10.0.0.4:33932 LAST_ACK
tcp 0 0 10.0.0.5:64093 10.0.0.4:33932 TIME_WAIT
tcp 0 0 10.0.0.5:35122 10.0.0.4:12865 ESTABLISHED
tcp 0 0 10.0.0.5:17318 10.0.0.4:33932 TIME_WAIT
tcp 0 0 :::22 :::* LISTEN
tcp 0 2056 ::ffff:192.168.0.254:22 ::ffff:192.168.0.1:3020 ESTABLISHED
udp 0 0 0.0.0.0:111 0.0.0.0:*
udp 0 0 0.0.0.0:631 0.0.0.0:*
udp 0 0 :::5353 :::*

输出列说明：

Proto:The protocol (tcp, udp, raw) used by the socket.
Recv-Q:The count of bytes not copied by the user program connected to this socket.
Send-Q:The count of bytes not acknowledged by the remote host.
Local Address:Address and port number of the local end of the socket. Unless the --numeric (-n) option is specified, the socket address is resolved to its canonical host name (FQDN), and the port number is translated into the corresponding service name.
Foreign Address:Address and port number of the remote end of the socket.
State:The state of the socket. Since there are no states in raw mode and usually no states used in UDP, this column may be left blank. 

至此，已经介绍了linux上常用的工具，此外，还有一些使用频率较低或者针对某个方面更专业的工具，例如：iptraf,tcpdump / ethereal,nmon,strace等，这了不再一一详述，仅供大家参考学习，禁止转载。