linux分析、诊断及调优的必备“杀器”之一

下面分别列出linux分析、诊断及调优时用到的工具，并分别进行说明，以方便自己和其他同学参考学习，禁止转载。

1、top

top - 02:06:59 up 4 days, 17:14, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 62 total, 1 running, 61 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2% us, 0.3% sy, 0.0% ni, 97.8% id, 1.7% wa, 0.0% hi, 0.0% si
Mem: 515144k total, 317624k used, 197520k free, 66068k buffers
Swap: 1048120k total, 12k used, 1048108k free, 179632k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13737 root 17 0 1760 896 1540 R 0.7 0.2 0:00.05 top
238 root 5 -10 0 0 0 S 0.3 0.0 0:01.56 reiserfs/0
1 root 16 0 588 240 444 S 0.0 0.0 0:05.70 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
6 root 5 -10 0 0 0 S 0.0 0.0 0:00.02 events/0

输出列说明：
PID：Process identification.
USER：Name of the user who owns (and perhaps started) the process.
PRI：Priority of the process. (See 1.1.4, “Process priority and nice level” on page 5 for details.)
SIZE：Amount of memory (code+data+stack) used by the process in kilobytes.
RSS：Amount of physical RAM used, in kilobytes.
SHARE：Amount of memory shared with other processes, in kilobytes.
STAT：State of the process: S=sleeping, R=running, T=stopped or traced,D=interruptible sleep, Z=zombie. 
%CPU：Share of the CPU usage (since the last screen update).
%MEM：Share of physical memory.
TIME：Total CPU time used by the process (since it was started).
COMMAND：Command line used to start the task (including parameters).

有用的热键：

t ：Displays summary information off and on.
m ：Displays memory information off and on.
A ：Sorts the display by top consumers of various system resources. Useful for quick identification of performance-hungry tasks on a system.
f ：Enters an interactive configuration screen for top. Helpful for setting up top for a specific task.
o ：Enables you to interactively select the ordering within top.
r ：Issues renice command.
k ：Issues kill command.

O or F：enters an interactive configuration screen for processes order.

h：displays help screen.

2、vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 0 1742264 112116 1999864 0 0 1 4 3 3 0 0 99 0
0 1 0 1742072 112208 1999772 0 0 0 2536 1258 1146 0 1 75 24
0 1 0 1741880 112260 1999720 0 0 0 2668 1235 1002 0 1 75 24
0 1 0 1741560 112308 1999932 0 0 0 2930 1240 1015 0 1 75 24
1 1 0 1741304 112344 2000416 0 0 0 2980 1238 925 0 1 75 24
0 1 0 1741176 112384 2000636 0 0 0 2968 1233 929 0 1 75 24
0 1 0 1741304 112420 2000600 0 0 0 3024 1247 925 0 1 75 24

输出列说明：
Process (procs) r:The number of processes waiting for runtime
                              b: The number of processes in uninterruptable sleep
Memory                swpd: The amount of virtual memory used (KB)
                              free: The amount of idle memory (KB)
                              buff: The amount of memory used as buffers (KB)
                             cache: The amount of memory used as cache (KB)
Swap                   si: Amount of memory swapped from the disk (KBps)
                             so: Amount of memory swapped to the disk (KBps)
IO                         bi: Blocks sent to a block device (blocks/s)
                             bo: Blocks received from a block device (blocks/s)
System               in: The number of interrupts per second, including the clock
                             cs: The number of context switches per second
CPU (% of total CPU time)
                             us: Time spent running non-kernel code (user time, including nice time).
                             sy: Time spent running kernel code (system time).
                             id: Time spent idle. Prior to Linux 2.5.41, this included I/O-wait time.
                             wa: Time spent waiting for IO. Prior to Linux 2.5.41, this appeared as zero.

有用的命令行参数：
-m:displays the memory utilization of the kernel (slabs)
-a:provides information about active and inactive memory pages
-n:displays only one header line, useful if running vmstat in sampling mode and piping the output to a file. (For example, root#vmstat –n 2 10 generates vmstat 10 times with a sampling rate of two seconds.)When used with the –p {partition} flag, vmstat also provides I/O statistics.

3、uptime

10:12:37 up 244 days, 26 min,  1 user,  load average: 10.92, 23.68, 23.64

说明：

The uptime command can be used to see how long the server has been running and how many users are logged on, as well as for a quick overview of the average load of the server.The system load average is displayed for the past 1minute, 5 minute, and 15 minute intervals.

4、ps

有用的命令选项：

-e:All processes. Identical to -A
-l:Show long format
-F:Extra full mode
-H:Forest
-L:Show threads, possibly with LWP and NLWP columns
-m:Show threads after processes

root# ps -elFL
F S UID PID PPID LWP C NLWP PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
4 S root 1 0 1 0 1 76 0 - 457 - 552 0 Mar08 ? 00:00:01 init [3]
1 S root 2 1 2 0 1 -40 - - 0 migrat 0 0 Mar08 ? 00:00:36 [migration/0]
1 S root 3 1 3 0 1 94 19 - 0 ksofti 0 0 Mar08 ? 00:00:00 [ksoftirqd/0]

root# ps -eLF| grep -E "LWP|/usr/sbin/httpd"
UID PID PPID LWP C NLWP SZ RSS PSR STIME TTY TIME CMD
root 4504 1 4504 0 1 4313 8600 2 08:33 ? 00:00:00 /usr/sbin/httpd
apache 4507 4504 4507 0 1 4313 4236 1 08:33 ? 00:00:00 /usr/sbin/httpd
apache 4508 4504 4508 0 1 4313 4228 1 08:33 ? 00:00:00 /usr/sbin/httpd

输出列说明：

F:Process flag
S:State of the process: S=sleeping, R=running, T=stopped or traced,D=interruptable sleep, Z=zombie. 
UID:Name of the user who owns (and perhaps started) the process.
PID:Process ID number
PPID:Parent process ID number
LWP:LWP(light weight process, or thread) ID of the lwp being reported.
C:Integer value of the processor utilization percentage.(CPU usage)
NLWP:Number of lwps (threads) in the process. (alias thcount).
PRI:Priority of the process. 
NI:Niceness level (whether the process tries to be nice by adjusting the priority by the number given; see below for details).
ADDR:Process Address space (not displayed)
SZ:Amount of memory (code+data+stack) used by the process in kilobytes.
WCHAN:Name of the kernel function in which the process is sleeping, a “-” if the process is running, or a “*” if the process is multi-threaded and ps is not displaying threads.
RSS:Resident set size, the non-swapped physical memory that a task has used (in kiloBytes).
PSR:Processor that process is currently assigned to.
STIME:Time the command started.
TTY:Terminal
TIME:Total CPU time used by the process (since it was started).
CMD:Command line used to start the task (including parameters).

5、free

有用的命令参数：

-b, -k, -m, -g:Display values in bytes, kilobytes, megabytes, and gigabytes
-l:Distinguishes between low and high memory 
-c <count>:Displays the free output <count> number of times

[root@edam ~]# free -l
                 total                used               free            shared                buffers                   cached
Mem:       4154484       2381500       1772984    0                          108256                  1974344
Low:         877828        199436           678392
High:        3276656      2182064        1094592
-/+ buffers/cache:        298900           3855584
Swap:      4194296      0                       4194296

mem:total:total physical memory;

            used:used physical memory;

            free:free physical memory;

            shared:shared physical memory;

            buffers:physical memory  for device buffers;

            cached:physical memory for  fs cache;

low:total:total physical memory for low zone;

       used:used physical memory for low zone;

       free:free physical memory for low zone;

high:total:total physical memory for high zone;

         used:used physical memory for high zone;

         free:free physical memory for high zone;

-/+ buffers/cache: used:mem:used-mem:buffers-mem:cache;

                                 free:mem:free+mem:buffers+mem:cached;

swap:total:total swap space;

           used:used swap space;

           free:free swap space;

6、iostat

iostat 3

avg-cpu: %user     %nice     %sys      %idle
                 0.03         0.00        0.02        99.95
Device:          tps          Blk_read/s            Blk_wrtn/s            Blk_read             Blk_wrtn
dev2-0           0.00        0.00                       0.04                       203                       2880
dev8-0           0.45        2.18                       2.21                       166464               168268My Oracle Support

输出列说明：

The CPU utilization report has four sections:

%user:Shows the percentage of CPU utilization that was taken up while executing at the user level (applications).
%nice:Shows the percentage of CPU utilization that was taken up while executing at the user level with a nice priority. 
%sys:Shows the percentage of CPU utilization that was taken up while executing at the system level (kernel).
%idle:Shows the percentage of time the CPU was idle.

The device utilization report has these sections:
Device:The name of the block device.
tps:The number of transfers per second (I/O requests per second) to the device.Multiple single I/O requests can be combined in a transfer request, because a transfer request can have different sizes.
Blk_read/s, Blk_wrtn/s:Blocks read and written per second indicate data read from or written to the device in seconds. Blocks can also have different sizes. Typical sizes are 1024, 2048, and 4048 bytes, depending on the partition size. 

For example,the block size of /dev/sda1 can be found with:
dumpe2fs -h /dev/sda1 |grep -F "Block size"
dumpe2fs 1.34 (25-Jul-2003)
Block size: 1024

Blk_read, Blk_wrtn:Indicates the total number of blocks read and written since the boot.

root# iostat -d -x sdb 1
Linux 2.6.9-42.ELsmp (lnxsu4.itso.ral.ibm.com) 03/18/2007
Device:        rrqm/s        wrqm/s          r/s          w/s            rsec/s             wsec/s            rkB/s             wkB/s             avgrq-sz         avgqu-sz         await          svctm            %util
sdb              0.15             0.00              0.02        0.00          0.46                0.00                  0.23              0.00                29.02              0.00                 2.60            1.05               0.00

输出列说明：

rrqm/s, wrqm/s:The number of read/write requests merged per second that were issued to the device. Multiple single I/O requests can be merged in a transfer request,
because a transfer request can have different sizes.
r/s, w/s:The number of read/write requests that were issued to the device per second.
rsec/s, wsec/s:The number of sectors read/write from the device per second.
rkB/s, wkB/s:The number of kilobytes read/write from the device per second.
avgrq-sz:The average size of the requests that were issued to the device. This value is displayed in sectors.
avgqu-sz:The average queue length of the requests that were issued to the device.
await:Shows the percentage of CPU utilization that was used while executing at the system level (kernel).
svctm:The average service time (in milliseconds) for I/O requests that were issued to the device.
%util:Percentage of CPU time during which I/O requests were issued to the device
(bandwidth utilization for the device). Device saturation occurs when this
value is close to 100%.