什么是SystemTap以及如何使用它?
环境
- Red Hat Enterprise Linux (all versions)
问题
- What is SystemTap and how to use it?
决议
SystemTap is an innovative tool which allows for simplified information gathering on the running Linux kernel. The purpose of using SystemTap is to obtain information on either performance issues or functional problems (bugs). By using SystemTap, developers and system administrators can debug problems and gather profiling and performance data without having to create and install instrumented kernels or other packages.
In essence, SystemTap provides the infrastructure (a command line interface and scripting language) needed to gather information. The actual job that SystemTap performs relies on user-developed scripts tailored to a specific purpose. Currently, there are a number of example SystemTap scripts pre-made for general use.
The operation of SystemTap is quite simple. The stap
command takes as an argument the name of a SystemTap file (called a probe). There may be additional command line arguments passed depending on the probe. SystemTap translates the probe into C and compiles the C source as a kernel module. It then inserts the resulting kernel module into the running kernel to perform the probe functions defined in the script. The output is printed to the console, or can be redirected to file.
Requirements
Because SystemTap compiles scripts from C code and launches probes for kernel instrumentation, it requires several packages in order to function. See the Installing SystemTap chapter of the SystemTap Beginners Guide for detailed installation instructions.
The easiest way to satisfy the requirements is to simply subscribe the system to the relevant debuginfo channels in RHN, then run the following commands which should set up the environment for SystemTap:
yum install systemtap
stap-prep
To set the environment up manually, in addition to the systemtap
package, the following packages must also be installed:
kernel-devel
for the running kernelkernel-debuginfo
for the running kernelkernel-debuginfo-common
for the running kernelgcc
systemtap
For example, for a 2.6.32-71.18.2.el6.x86_64
kernel you'll need:
kernel-debuginfo-2.6.32-71.18.2.el6.x86_64
kernel-debuginfo-common-x86_64-2.6.32-71.18.2.el6.x86_64
kernel-devel-2.6.32-71.18.2.el6.x86_64
As well as the gcc
and systemtap
packages.
The gcc
and kernel-devel
packages are available on Red Hat Network and can be installed using yum
(RHEL 5 onwards).
Legacy
On RHEL 4, use up2date
. For Red Hat Enterprise Linux (RHEL) 4, the kernel-debuginfo
package is not available via up2date
and must be installed from the ISO image available on Red Hat Network or from ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4AS/en/os/Debuginfo/ (substitute 4AS
in the URL with 4WS
, 4ES
, or 4Desktop
, depending on the variant of of RHEL you have installed. Note: Red Hat Enterprise Linux 4 does not have kernel-debuginfo-common
package.
For RHEL 5, you can download the kernel-debuginfo
and kernel-debuginfo-common
packages from ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/.
Using Systemtap
You can verify the SystemTap environment with the stap-report
command.
An example script is shown below:
#! /usr/bin/env stap
# Using statistics and maps to examine kernel memory allocations
global kmalloc
probe kernel.function("__kmalloc") {
kmalloc[execname()] <<< $size
}
# Exit after 10 seconds
probe timer.ms(10000) { exit () }
probe end {
foreach ([name] in kmalloc) {
printf("Allocations for %s\\n", name)
printf("Count: %d allocations\\n", @count(kmalloc[name]))
printf("Sum: %d Kbytes\\n", @sum(kmalloc[name])/1000)
printf("Average: %d bytes\\n", @avg(kmalloc[name]))
printf("Min: %d bytes\\n", @min(kmalloc[name]))
printf("Max: %d bytes\\n", @max(kmalloc[name]))
print("\\nAllocations by size in bytes\\n")
print(@hist_log(kmalloc[name]))
printf("-------------------------------------------------------\\n\\n");
}
}
This script, drawn from the SystemTap project wiki, can be used to print information kernel memory allocations of the system. The script can be invoked as follows:
stap kmalloc2.stp
For issues during the compile or loading of the module within the stap
command, append the parameter -vv
to show more verbose output.
SystemTap will then translate the probe into C, compile the C program, and insert the probe into the running kernel. Truncated output is below:
-------------------------------------------------------
Allocations for httpd
Count: 10 allocations
Sum: 0 Kbytes
Average: 0 bytes
Min: 0 bytes
Max: 0 bytes
Allocations by size in bytes
value |-------------------------------------------------- count
0 |@@@@@@@@@@ 10
1 | 0
2 | 0
-------------------------------------------------------
Allocations for sendmail
Count: 2 allocations
Sum: 0 Kbytes
Average: 24 bytes
Min: 24 bytes
Max: 24 bytes
Allocations by size in bytes
value |-------------------------------------------------- count
4 | 0
8 | 0
16 |@@ 2
32 | 0
64 | 0
-------------------------------------------------------
This is a simple example that just touches the surface of the capabilities offered by SystemTap. System Administrators could use this information to better understand kernel memory allocation on the running system and adjust kernel tuning parameters accordingly. Application developers can use this information as an overview of which applications are receiving more kernel memory allocations, which can be used as a starting point for deeper application profiling.
Additional Information about SystemTap
- Complement to this guide
- SystemTap Beginners Guide
- The SystemTap project Wiki: http://sourceware.org/systemtap/wiki/HomePage
- The SystemTap project Homepage: http://sourceware.org/systemtap/
- Red Hat Magazine overview of SystemTap: http://www.redhat.com/magazine/011sep05/features/systemtap/