BCC观测工具的使用

观测工具下载地址:https://github.com/iovisor/bcc

bcc观测工具的安装方式:https://github.com/iovisor/bcc/blob/master/INSTALL.md

 

工具简介:

BCC makes BPF programs easier to write, with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua. It is suited for many tasks, including performance analysis and network traffic control.

BCC 是用于创建高效内核跟踪和操作程序的工具包,包括一些有用的工具和示例。它利用了扩展的BPF(Berkeley Packet Filters),正式名称为 eBPF。

 

使用要求:

1)Linux 内核版本最低 4.1
2)kernel-devel 版本必须和内核版本一致

[root@yang-01 ~]# rpm -qa|grep -i kernel | grep -i devel
kernel-devel-3.10.0-1127.18.2.el7.x86_64
[root@yang-01 ~]# uname -r
3.10.0-1127.el7.x86_64

3)kernel 配置文件相关参数必须按照下面的要求

[root@yang-01 ~]# grep -i BPF /boot/config-`uname -r`
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m 
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y

4)对于 CentOS 建议升级到 3.6 及以上版本
5)使用默认的 Python 2.7 即可,不需要升级 Python 3

 

 

yum安装
[root@yang-01 ~]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
[root@yang-01 ~]# uname -r
3.10.0-1127.el7.x86_64
  
[root@yang-01 ~]# yum update kernel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * epel: mirrors.yun-idc.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:3.10.0-1127.18.2.el7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
===========================================================================================================================================================================
 Package                              Arch                                 Version                                             Repository                             Size
===========================================================================================================================================================================
Installing:
 kernel                               x86_64                               3.10.0-1127.18.2.el7                                updates                                50 M
 
Transaction Summary
===========================================================================================================================================================================
Install  1 Package
 
Total download size: 50 M
Installed size: 64 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-3.10.0-1127.18.2.el7.x86_64.rpm                                                                                                              |  50 MB  00:00:08    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-3.10.0-1127.18.2.el7.x86_64                                                                                                                      1/1
  Verifying  : kernel-3.10.0-1127.18.2.el7.x86_64                                                                                                                      1/1
 
Installed:
  kernel.x86_64 0:3.10.0-1127.18.2.el7                                                                                                                                    
 
Complete!
[root@yang-01 ~]# yum install bcc -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * epel: mirrors.yun-idc.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package bcc.x86_64 0:0.10.0-1.el7 will be installed
--> Processing Dependency: bcc-tools = 0.10.0-1.el7 for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: llvm-private >= 6.0.1-0.3 for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libLLVM-7-rhel.so(LLVM_7)(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangSerialization.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangSema.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangRewrite.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangParse.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangLex.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangFrontend.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangEdit.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangDriver.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangCodeGen.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangBasic.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangAnalysis.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangAST.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libLLVM-7-rhel.so()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Running transaction check
---> Package bcc-tools.x86_64 0:0.10.0-1.el7 will be installed
--> Processing Dependency: python-bcc = 0.10.0-1.el7 for package: bcc-tools-0.10.0-1.el7.x86_64
--> Processing Dependency: python-netaddr for package: bcc-tools-0.10.0-1.el7.x86_64
--> Processing Dependency: kernel-devel for package: bcc-tools-0.10.0-1.el7.x86_64
---> Package llvm-private.x86_64 0:7.0.1-1.el7 will be installed
--> Running transaction check
---> Package kernel-devel.x86_64 0:3.10.0-1127.18.2.el7 will be installed
---> Package python-bcc.x86_64 0:0.10.0-1.el7 will be installed
---> Package python-netaddr.noarch 0:0.7.5-9.el7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
===========================================================================================================================================================================
 Package                                    Arch                               Version                                           Repository                           Size
===========================================================================================================================================================================
Installing:
 bcc                                        x86_64                             0.10.0-1.el7                                      base                                567 k
Installing for dependencies:
 bcc-tools                                  x86_64                             0.10.0-1.el7                                      base                                394 k
 kernel-devel                               x86_64                             3.10.0-1127.18.2.el7                              updates                              18 M
 llvm-private                               x86_64                             7.0.1-1.el7                                       base                                 23 M
 python-bcc                                 x86_64                             0.10.0-1.el7                                      base                                 74 k
 python-netaddr                             noarch                             0.7.5-9.el7                                       base                                983 k
 
Transaction Summary
===========================================================================================================================================================================
Install  1 Package (+5 Dependent packages)
 
Total download size: 42 M
Installed size: 126 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/6): bcc-tools-0.10.0-1.el7.x86_64.rpm                                                                                                            | 394 kB  00:00:05    
(2/6): bcc-0.10.0-1.el7.x86_64.rpm                                                                                                                  | 567 kB  00:00:05    
(3/6): python-bcc-0.10.0-1.el7.x86_64.rpm                                                                                                           |  74 kB  00:00:00    
(4/6): python-netaddr-0.7.5-9.el7.noarch.rpm                                                                                                        | 983 kB  00:00:00    
(5/6): kernel-devel-3.10.0-1127.18.2.el7.x86_64.rpm                                                                                                 |  18 MB  00:00:06    
(6/6): llvm-private-7.0.1-1.el7.x86_64.rpm                                                                                                          |  23 MB  00:00:30    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                      1.4 MB/s |  42 MB  00:00:30    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : llvm-private-7.0.1-1.el7.x86_64                                                                                                                         1/6
  Installing : kernel-devel-3.10.0-1127.18.2.el7.x86_64                                                                                                                2/6
  Installing : python-netaddr-0.7.5-9.el7.noarch                                                                                                                       3/6
  Installing : python-bcc-0.10.0-1.el7.x86_64                                                                                                                          4/6
  Installing : bcc-tools-0.10.0-1.el7.x86_64                                                                                                                           5/6
  Installing : bcc-0.10.0-1.el7.x86_64                                                                                                                                 6/6
  Verifying  : python-netaddr-0.7.5-9.el7.noarch                                                                                                                       1/6
  Verifying  : python-bcc-0.10.0-1.el7.x86_64                                                                                                                          2/6
  Verifying  : bcc-0.10.0-1.el7.x86_64                                                                                                                                 3/6
  Verifying  : kernel-devel-3.10.0-1127.18.2.el7.x86_64                                                                                                                4/6
  Verifying  : bcc-tools-0.10.0-1.el7.x86_64                                                                                                                           5/6
  Verifying  : llvm-private-7.0.1-1.el7.x86_64                                                                                                                         6/6
 
Installed:
  bcc.x86_64 0:0.10.0-1.el7                                                                                                                                               
 
Dependency Installed:
  bcc-tools.x86_64 0:0.10.0-1.el7           kernel-devel.x86_64 0:3.10.0-1127.18.2.el7       llvm-private.x86_64 0:7.0.1-1.el7       python-bcc.x86_64 0:0.10.0-1.el7     
  python-netaddr.noarch 0:0.7.5-9.el7     
 
Complete!
 
[root@yang-01 tools]# pwd
/usr/share/bcc/tools
[root@yang-01 tools]# ls
argdist       cachestat     dcstat      fileslower      javagc       mountsnoop      opensnoop  pythoncalls  rubystat     sslsniff    tcpaccept   tplist
bashreadline  cachetop      deadlock    filetop         javaobjnew   mysqld_qslower  perlcalls  pythonflow   runqlat      stackcount  tcpconnect  trace
biolatency    capable       deadlock.c  funccount       javastat     nfsdist         perlflow   pythongc     runqlen      statsnoop   tcpconnlat  ttysnoop
biosnoop      cobjnew       doc         funclatency     javathreads  nfsslower       perlstat   pythonstat   runqslower   syncsnoop   tcpdrop     vfscount
biotop        cpudist       drsnoop     funcslower      killsnoop    nodegc          phpcalls   reset-trace  shmsnoop     syscount    tcplife     vfsstat
bitesize      cpuunclaimed  execsnoop   gethostlatency  lib          nodestat        phpflow    rubycalls    slabratetop  tclcalls    tcpretrans  wakeuptime
bpflist       dbslower      ext4dist    hardirqs        llcstat      offcputime      phpstat    rubyflow     sofdsnoop    tclflow     tcpsubnet   xfsdist
btrfsdist     dbstat        ext4slower  javacalls       mdflush      offwaketime     pidpersec  rubygc       softirqs     tclobjnew   tcptop      xfsslower
btrfsslower   dcsnoop       filelife    javaflow        memleak      oomkill         profile    rubyobjnew   solisten     tclstat     tcptracer
 点击此处展开...

 

 

 

报错举例1
# 报错举例
[root@yang-01 ~]# cachestat 1 3
modprobe: FATAL: Module kheaders not found.
chdir(/lib/modules/3.10.0-1127.el7.x86_64/build): No such file or directory
Traceback (most recent call last):
  File "/usr/share/bcc/tools/cachestat", line 96, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 325, in __init__
    raise Exception("Failed to compile BPF text")
Exception: Failed to compile BPF text
 
# 发现报错,这时需要进行内核版本的查看,确保 kernel-devel 与系统上当前安装的内核版本匹配
[root@yang-01 tools]# rpm -qa|grep -i kernel | grep -i devel
kernel-devel-3.10.0-1127.18.2.el7.x86_64
[root@yang-01 tools]# uname -r
3.10.0-1127.el7.x86_64
[root@yang-01 tools]# yum install "kernel-devel-uname-r == $(uname -r)"
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.cn99.com
 * epel: mirrors.yun-idc.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package kernel-devel.x86_64 0:3.10.0-1127.el7 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
===========================================================================================================================================================================
 Package                                    Arch                                 Version                                          Repository                          Size
===========================================================================================================================================================================
Installing:
 kernel-devel                               x86_64                               3.10.0-1127.el7                                  base                                18 M
 
Transaction Summary
===========================================================================================================================================================================
Install  1 Package
 
Total download size: 18 M
Installed size: 38 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-devel-3.10.0-1127.el7.x86_64.rpm                                                                                                             |  18 MB  00:00:06    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-devel-3.10.0-1127.el7.x86_64                                                                                                                     1/1
  Verifying  : kernel-devel-3.10.0-1127.el7.x86_64                                                                                                                     1/1
 
Installed:
  kernel-devel.x86_64 0:3.10.0-1127.el7                                                                                                                                   
 
Complete!

 

报错举例2
报错: 在安装kernel-devel显示 No package
       No package kernel-devel-uname-r == 5.6.2-1.el7.elrepo.x86_64 available.
  
解决办法: 启用[elrepo-kerne]存储库,
# vim /etc/yum.repos.d/elrepo.repo
[elrepo-kernel]
name=ELRepo.org Community Enterprise Linux Kernel Repository - el7
baseurl=http://elrepo.org/linux/kernel/el7/$basearch/
        http://mirrors.coreix.net/elrepo/kernel/el7/$basearch/
        http://mirror.rackspace.com/elrepo/kernel/el7/$basearch/
        http://repos.lax-noc.com/elrepo/kernel/el7/$basearch/
        http://mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-kernel.el7
# 修改这一行将0改为1
enabled=1
 
然后重新执行:
# sudo yum install "kernel-devel-uname-r == $(uname -r)"
缓存命中查询 命令使用举例
[root@yang-01 tools]# cachestat 1 5
    HITS   MISSES  DIRTIES HITRATIO   BUFFERS_MB  CACHED_MB
    1409        0       77  100.00%            1       2440
    1438        0      101  100.00%            1       2440
    1413        0       68  100.00%            1       2440
    1412        0      117  100.00%            1       2440
    1411        0       72  100.00%            1       2440
第一个参数为间隔秒,第二个参数为输出次数
TOTAL ,表示总的 I/O 次数
MISSES ,表示缓存未命中的次数
HITS ,表示缓存命中的次数
DIRTIES, 表示新增到缓存中的脏页数
BUFFERS_MB 表示 Buffers 的大小,以 MB 为单位
CACHED_MB 表示 Cache 的大小,以 MB 为单位
 
 
  
[root@yang-01 tools]# cachetop
16:24:15 Buffers MB: 1 / Cached MB: 2443 / Sort: HITS / Order: ascending
PID      UID      CMD              HITS     MISSES   DIRTIES  READ_HIT%  WRITE_HIT%
    1653 actionte ustats                  1        0        0     100.0%       0.0%
    1778 actionte umon-mgr                1        0        0     100.0%       0.0%
     903 mysql    jsvc                    1        0        0     100.0%       0.0%
   10909 actionte mysqld                  2        0        0     100.0%       0.0%
   22506 actionte uproxy                  2        0        1      50.0%       0.0%
     869 mysql    jsvc                    2        2        2       0.0%       0.0%
   10788 actionte mysqld                  3        0        0     100.0%       0.0%
    1724 actionte urman-agent             4        2        2      33.3%       0.0%
   21780 actionte uguard-agent            4        2        2      33.3%       0.0%
   12798 actionte ustats                  6        0        0     100.0%       0.0%
   12799 actionte ustats                  6        0        0     100.0%       0.0%
   12800 actionte ustats                  6        0        0     100.0%       0.0%
这个命令的输出和top类似,默认按照缓存的命中次数(HITS)排序,展示了每个进程的缓存命中情况。
这里的 HITS、MISSES 和 DIRTIES ,跟 cachestat 里的含义一样,分别代表间隔时间内的缓存命中次数、未命中次数以及新增到缓存中的脏页数。
而 READ_HIT 和 WRITE_HIT ,分别表示读和写的缓存命中率。

 

 

BCC工具
# 查看新的进程。那些会消耗系统资源,但很短暂的进程,它们甚至不会出现在 top(1) 命令或其它工具中的显示之中。这些新进程可以使用 execsnoop 进行检测(或使用行业术语说,可以被追踪traced)。
[root@yang-01 tools]# execsnoop -h
usage: execsnoop [-h] [-t] [-x] [-q] [-n NAME] [-l LINE] [--max-args MAX_ARGS]
 
Trace exec() syscalls
 
optional arguments:
  -h, --help            show this help message and exit
  -t, --timestamp       include timestamp on output
  -x, --fails           include failed exec()s
  -q, --quote           Add quotemarks (") around arguments.
  -n NAME, --name NAME  only print commands matching this name (regex), any
                        arg
  -l LINE, --line LINE  only print commands where arg contains this line
                        (regex)
  --max-args MAX_ARGS   maximum number of arguments parsed and displayed,
                        defaults to 20
 
examples:
    ./execsnoop           # trace all exec() syscalls
    ./execsnoop -x        # include failed exec()s
    ./execsnoop -t        # include timestamps
    ./execsnoop -q        # add "quotemarks" around arguments
    ./execsnoop -n main   # only print command lines containing "main"
    ./execsnoop -l tpkg   # only print command where arguments contains "tpkg"
[root@yang-01 tools]# execsnoop
PCOMM            PID    PPID   RET ARGS
bash             17436  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
bash             17437  1642     0
systemctl        17437  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service /usr/bin/systemctl is-active mysqld_3307.service
systemctl        17436  1642     0 /usr/bin/systemctl is-active mysqld_5690.service
bash             17439  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
systemctl        17439  1642     0 /usr/bin/systemctl is-active mysqld_3307.service
bash             17440  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
systemctl        17440  1642     0 /usr/bin/systemctl is-active mysqld_5690.service
bash             17442  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
systemctl        17442  1642     0 /usr/bin/systemctl is-active mysqld_3307.service
bash             17443  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
systemctl        17443  1642     0 /usr/bin/systemctl is-active mysqld_5690.service
bash             17445  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
bash             17446  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
##################################################################################################################################################################
  
  
  
# opensnoop 通过跟踪 open() 系统调用来工作。opensnoop 的一些优点在于它能在系统范围内工作,并且跟踪所有进程的 open() 系统调用。
[root@yang-01 tools]# opensnoop -h
usage: opensnoop [-h] [-T] [-U] [-x] [-p PID] [-t TID] [-u UID] [-d DURATION]
                 [-n NAME] [-e] [-f FLAG_FILTER]
 
Trace open() syscalls
 
optional arguments:
  -h, --help            show this help message and exit
  -T, --timestamp       include timestamp on output
  -U, --print-uid       print UID column
  -x, --failed          only show failed opens
  -p PID, --pid PID     trace this PID only
  -t TID, --tid TID     trace this TID only
  -u UID, --uid UID     trace this UID only
  -d DURATION, --duration DURATION
                        total duration of trace in seconds
  -n NAME, --name NAME  only print process names containing this name
  -e, --extended_fields
                        show extended fields
  -f FLAG_FILTER, --flag_filter FLAG_FILTER
                        filter on flags argument (e.g., O_WRONLY)
 
examples:
    ./opensnoop           # trace all open() syscalls
    ./opensnoop -T        # include timestamps
    ./opensnoop -U        # include UID
    ./opensnoop -x        # only show failed opens
    ./opensnoop -p 181    # only trace PID 181
    ./opensnoop -t 123    # only trace TID 123
    ./opensnoop -u 1000   # only trace UID 1000
    ./opensnoop -d 10     # trace for 10 seconds only
    ./opensnoop -n main   # only print process names containing "main"
    ./opensnoop -e        # show extended fields
    ./opensnoop -f O_WRONLY -f O_RDWR  # only print calls for writing
[root@yang-01 tools]# opensnoop
PID    COMM               FD ERR PATH
1642   ustats             14   0 ./logs
18699  opensnoop          -1   2 /usr/lib64/python2.7/encodings/ascii.so
18699  opensnoop          -1   2 /usr/lib64/python2.7/encodings/asciimodule.so
18699  opensnoop          16   0 /usr/lib64/python2.7/encodings/ascii.py
18699  opensnoop          17   0 /usr/lib64/python2.7/encodings/ascii.pyc
1309   uagent             16   0 /action-dmp/ustats/ustats.pid
1309   uagent             16   0 /proc/1642/cmdline
1309   uagent             16   0 /action-dmp/umon-mgr/umon-mgr.pid
1309   uagent             16   0 /proc/1684/cmdline
1309   uagent             16   0 /action-dmp/urman-agent/urman-agent.pid
1309   uagent             16   0 /proc/1672/cmdline
1309   uagent             16   0 /action-dmp/ucore/ucore.pid
1309   uagent             16   0 /proc/1538/cmdline
1309   uagent             16   0 /action-dmp/uguard-agent/uguard-agent.pid
1309   uagent             16   0 /proc/1664/cmdline
1309   uagent             16   0 /action-dmp/uguard-mgr/uguard-mgr.pid
1309   uagent             16   0 /proc/1636/cmdline
1309   uagent             16   0 /action-dmp/umon/umon.pid
1309   uagent             16   0 /proc/1734/cmdline
1309   uagent             16   0 /action-dmp/udeploy/udeploy.pid
1309   uagent             16   0 /proc/1644/cmdline
1309   uagent             16   0 /action-dmp/umc/umc.pid
1309   uagent             16   0 /proc/1638/cmdline
1309   uagent             16   0 /action-dmp/uproxy/uproxy.pid
1309   uagent             16   0 /proc/22496/cmdline
1309   uagent             16   0 /action-dmp/urman-mgr/urman-mgr.pid
1309   uagent             16   0 /proc/1640/cmdline
1664   uguard-agent       18   0 ./logs
1642   ustats             14   0 /dev/null
1642   ustats             40   0 /dev/null
18703  bash                3   0 /etc/ld.so.cache
18703  bash                3   0 /lib64/libtinfo.so.5
18703  bash                3   0 /lib64/libdl.so.2
18703  bash                3   0 /lib64/libc.so.6
18703  bash               -1   6 /dev/tty
18703  bash                3   0 /usr/lib/locale/locale-archive
18703  bash                3   0 /proc/meminfo
18703  bash                3   0 /usr/lib64/gconv/gconv-modules.cache
18703  bash                3   0 /etc/nsswitch.conf
18703  bash                3   0 /etc/ld.so.cache
18703  bash                3   0 /lib64/libnss_files.so.2
18703  bash                3   0 /etc/passwd
5448   mysqld             -1   2 ./universe/u_delay.frm
18703  systemctl           3   0 /etc/ld.so.cache
18703  systemctl           3   0 /lib64/librt.so.1
18703  systemctl           3   0 /lib64/libselinux.so.1
18703  systemctl           3   0 /lib64/liblzma.so.5
18703  systemctl           3   0 /lib64/liblz4.so.1
18703  systemctl           3   0 /lib64/libgcrypt.so.11
18703  systemctl           3   0 /lib64/libcap.so.2
18703  systemctl           3   0 /lib64/libgcc_s.so.1
18703  systemctl           3   0 /lib64/libpthread.so.0
18703  systemctl           3   0 /lib64/libc.so.6
18703  systemctl           3   0 /lib64/libpcre.so.1
18703  systemctl           3   0 /lib64/libdl.so.2
18703  systemctl           3   0 /lib64/libgpg-error.so.0
18703  systemctl           3   0 /lib64/libattr.so.1
18704  bash                3   0 /etc/ld.so.cache
18704  bash                3   0 /lib64/libtinfo.so.5
##################################################################################################################################################################
  
  
# xfsslower 工具可以跟踪大于 1 毫秒(参数)延迟的常见 XFS 文件系统操作。
# 在 xfsslower 工具运行的时候,通过在 XFS 中动态地检测内核函数实现的,并当它结束的时候解除该检测。
# 这个 bcc 工具也有其它文件系统的版本:ext4slower、btrfsslower、zfsslower 和 nfsslower。
[root@yang-01 tools]# xfsslower -h
usage: xfsslower [-h] [-j] [-p PID] [min_ms]
 
Trace common XFS file operations slower than a threshold
 
positional arguments:
  min_ms             minimum I/O duration to trace, in ms (default 10)
 
optional arguments:
  -h, --help         show this help message and exit
  -j, --csv          just print fields: comma-separated values
  -p PID, --pid PID  trace this PID only
 
examples:
    ./xfsslower             # trace operations slower than 10 ms (default)
    ./xfsslower 1           # trace operations slower than 1 ms
    ./xfsslower -j 1        # ... 1 ms, parsable output (csv)
    ./xfsslower 0           # trace all operations (warning: verbose)
    ./xfsslower -p 185      # trace PID 185 only
[root@yang-01 tools]# xfsslower 1
Tracing XFS operations slower than 1 ms
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
17:02:44 umon-mgr       1684   0       0           5.93 detail.log
17:02:44 umc            1638   0       0           1.05 detail.log
17:02:45 mysqld         10763  81920   0           8.12 temp_16.ibt
17:02:45 mysqld         10763  16384   80          2.16 temp_16.ibt
17:02:45 ustats         1642   0       0           3.12 brief.log
17:02:46 mysqld         10763  835584  1024        7.81 ibdata1
17:02:46 mysqld         10763  0       0          31.14 ib_logfile0
17:02:46 ucore          1538   0       0           5.89 raft.db
17:02:46 ustats         1642   0       0           4.04 detail.log
17:02:46 mysqld         10763  0       0           3.58 ib_logfile0
17:02:46 ustats         1642   0       0           1.49 brief.log
17:02:46 umc            1638   0       0          13.70 brief.log
17:02:46 uguard-mgr     1636   0       0           9.62 detail.log
17:02:46 ucore          1538   0       0           8.87 detail.log
17:02:48 ustats         1642   0       0           2.16 brief.log
17:02:49 ustats         1642   0       0           1.41 brief.log
17:02:49 uguard-agent   1664   0       0          26.30 brief.log
17:02:49 uguard-agent   1664   0       0           6.56 detail.log
17:02:49 mysqld         10763  0       0           1.89 ib_logfile0
17:02:50 ustats         1642   0       0           1.01 brief.log
17:02:50 mysqld         10763  0       0           1.18 ib_logfile0
##################################################################################################################################################################
  
  
# 可以用来实现高效的内核摘要统计。从内核层到用户层的数据传输仅仅是“计数”列。 用户级程序生成其余的。
[root@yang-01 tools]# biolatency -h
usage: biolatency [-h] [-T] [-Q] [-m] [-D] [-F] [interval] [count]
 
Summarize block device I/O latency as a histogram
 
positional arguments:
  interval            output interval, in seconds
  count               number of outputs
 
optional arguments:
  -h, --help          show this help message and exit
  -T, --timestamp     include timestamp on output
  -Q, --queued        include OS queued time in I/O time
  -m, --milliseconds  millisecond histogram
  -D, --disks         print a histogram per disk device
  -F, --flags         print a histogram per set of I/O flags
 
examples:
    ./biolatency            # summarize block I/O latency as a histogram
    ./biolatency 1 10       # print 1 second summaries, 10 times
    ./biolatency -mT 1      # 1s summaries, milliseconds, and timestamps
    ./biolatency -Q         # include OS queued time in I/O time
    ./biolatency -D         # show each disk device separately
    ./biolatency -F         # show I/O flags separately
 
 
[root@yang-01 tools]# biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          0        |                                        |
         2 -> 3          0        |                                        |
         4 -> 7          0        |                                        |
         8 -> 15         0        |                                        |
        16 -> 31         13       |*                                       |
        32 -> 63         306      |*********************************       |
        64 -> 127        367      |****************************************|
       128 -> 255        254      |***************************             |
       256 -> 511        12       |*                                       |
       512 -> 1023       1        |                                        |
      1024 -> 2047       1        |                                        |
      2048 -> 4095       9        |                                        |
      4096 -> 8191       10       |*                                       |
      8192 -> 16383      9        |                                        |
     16384 -> 32767      3        |                                        |
     32768 -> 65535      2        |                                        |
##################################################################################################################################################################
  
  
  
# tcplife显示 TCP 会话的生命周期和吞吐量统计
[root@yang-01 tools]# tcplife -h
usage: tcplife [-h] [-T] [-t] [-w] [-s] [-p PID] [-L LOCALPORT]
               [-D REMOTEPORT]
 
Trace the lifespan of TCP sessions and summarize
 
optional arguments:
  -h, --help            show this help message and exit
  -T, --time            include time column on output (HH:MM:SS)
  -t, --timestamp       include timestamp on output (seconds)
  -w, --wide            wide column output (fits IPv6 addresses)
  -s, --csv             comma separated values output
  -p PID, --pid PID     trace this PID only
  -L LOCALPORT, --localport LOCALPORT
                        comma-separated list of local ports to trace.
  -D REMOTEPORT, --remoteport REMOTEPORT
                        comma-separated list of remote ports to trace.
 
examples:
    ./tcplife           # trace all TCP connect()s
    ./tcplife -t        # include time column (HH:MM:SS)
    ./tcplife -w        # wider colums (fit IPv6)
    ./tcplife -stT      # csv output, with times & timestamps
    ./tcplife -p 181    # only trace PID 181
    ./tcplife -L 80     # only trace local port 80
    ./tcplife -L 80,81  # only trace local ports 80 and 81
    ./tcplife -D 80     # only trace remote port 80
 
 
[root@yang-01 tools]# tcplife
PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
854   jsvc       ::ffff:127.0.0.1 33222 ::ffff:127.0.0.1 3307      0     0 1.91
10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33222     0     0 1.91
854   jsvc       ::ffff:127.0.0.1 33224 ::ffff:127.0.0.1 3307      0     0 2.02
10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33224     0     0 2.03
1638  umc        10.186.60.24    48697 10.186.60.24    5704      0     0 3.01
1636  uguard-mgr ::ffff:10.186.60.24 5704  ::ffff:10.186.60.24 48697     0     0 3.03
854   jsvc       ::ffff:127.0.0.1 33228 ::ffff:127.0.0.1 3307      0     0 1.76
10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33228     0     0 1.77
1684  umon-mgr   ::1             37504 ::1             8200      0     0 0.03
1684  umon-mgr   127.0.0.1       48056 127.0.0.1       8200      0     0 0.02
854   jsvc       ::ffff:127.0.0.1 33234 ::ffff:127.0.0.1 3307      0     0 2.02
10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33234     0     0 2.04
1642  ustats     127.0.0.1       33236 127.0.0.1       3307      0     0 3.13
10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33236     0     0 3.15
1642  ustats     127.0.0.1       45738 127.0.0.1       5690      0     0 3.33
5448  mysqld     ::ffff:127.0.0.1 5690  ::ffff:127.0.0.1 45738     0     0 3.35
854   jsvc       ::ffff:127.0.0.1 33240 ::ffff:127.0.0.1 3307      0     0 1.65
10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33240     0     0 1.66
####################################################################################################################################################################
  
  
  
# 检测用于名称解析的 gethostbyname(3) 和相关的库调用
[root@yang-01 tools]# gethostlatency -h
usage: gethostlatency [-h] [-p PID]
 
Show latency for getaddrinfo/gethostbyname[2] calls
 
optional arguments:
  -h, --help         show this help message and exit
  -p PID, --pid PID  trace this PID only
 
examples:
    ./gethostlatency           # trace all TCP accept()s
    ./gethostlatency -p 181    # only trace PID 181
  
[root@yang-01 tools]# gethostlatency
TIME      PID    COMM                  LATms HOST
17:37:55  903    jsvc                   0.28 localhost
17:38:03  22079  umon                   0.28 localhost
17:38:08  14308  umon-mgr               0.23 localhost
17:38:16  919    jsvc                   0.17 yang-01
17:38:25  903    jsvc                   0.28 localhost
17:38:37  24347  umon                   0.25 localhost
17:38:38  1780   umon-mgr               0.34 localhost
17:38:38  22085  umon                   0.22 localhost
17:38:43  1780   umon-mgr               0.22 localhost
17:38:45  893    jsvc                   0.08 yang-01
#####################################################################################################################################################################
  
  
  
#  trace 工具由 Sasha Goldshtein 提供,并提供了一些基本的 printf(1) 功能和自定义探针。
[root@yang-01 tools]# trace -h
usage: trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE]
             [-S] [-M MAX_EVENTS] [-t] [-T] [-C] [-B] [-s SYM_FILE_LIST] [-K]
             [-U] [-a] [-I header]
             probe [probe ...]
 
Attach to functions and print trace messages.
 
positional arguments:
  probe                 probe specifier (see examples)
 
optional arguments:
  -h, --help            show this help message and exit
  -b BUFFER_PAGES, --buffer-pages BUFFER_PAGES
                        number of pages to use for perf_events ring buffer
                        (default64)
  -p PID, --pid PID     id of the process to trace (optional)
  -L TID, --tid TID     id of the thread to trace (optional)
  -v, --verbose         print resulting BPF program code before executing
  -Z STRING_SIZE, --string-size STRING_SIZE
                        maximum size to read from strings
  -S, --include-self    do not filter trace's own pid from the trace
  -M MAX_EVENTS, --max-events MAX_EVENTS
                        number of events to print before quitting
  -t, --timestamp       print timestamp column (offset from trace start)
  -T, --time            print time column
  -C, --print_cpu       print CPU id
  -B, --bin_cmp         allow to use STRCMP with binary values
  -s SYM_FILE_LIST, --sym_file_list SYM_FILE_LIST
                        coma separated list of symbol files to use for symbol
                        resolution
  -K, --kernel-stack    output kernel stack trace
  -U, --user-stack      output user stack trace
  -a, --address         print virtual address in stacks
  -I header, --include header
                        additional header files to include in the BPF program
                        as either full path, or relative to current working
                        directory, or relative to default kernel header search
                        path
 
EXAMPLES:
 
trace do_sys_open
        Trace the open syscall and print a default trace message when entered
trace 'do_sys_open "%s", arg2'
        Trace the open syscall and print the filename being opened
trace 'sys_read (arg3 > 20000) "read %d bytes", arg3'
        Trace the read syscall and print a message for reads >20000 bytes
trace 'r::do_sys_open "%llx", retval'
        Trace the return from the open syscall and print the return value
trace 'c:open (arg2 == 42) "%s %d", arg1, arg2'
        Trace the open() call from libc only if the flags (arg2) argument is 42
trace 'c:malloc "size = %d", arg1'
        Trace malloc calls and print the size being allocated
trace 'p:c:write (arg1 == 1) "writing %d bytes to STDOUT", arg3'
        Trace the write() call from libc to monitor writes to STDOUT
trace 'r::__kmalloc (retval == 0) "kmalloc failed!"'
        Trace returns from __kmalloc which returned a null pointer
trace 'r:c:malloc (retval) "allocated = %x", retval'
        Trace returns from malloc and print non-NULL allocated buffers
trace 't:block:block_rq_complete "sectors=%d", args->nr_sector'
        Trace the block_rq_complete kernel tracepoint and print # of tx sectors
trace 'u:pthread:pthread_create (arg4 != 0)'
        Trace the USDT probe pthread_create when its 4th argument is non-zero
trace 'p::SyS_nanosleep(struct timespec *ts) "sleep for %lld ns", ts->tv_nsec'
        Trace the nanosleep syscall and print the sleep duration in ns
trace -I 'linux/fs.h' \
      'p::uprobe_register(struct inode *inode) "a_ops = %llx", inode->i_mapping->a_ops'
        Trace the uprobe_register inode mapping ops, and the symbol can be found
        in /proc/kallsyms
trace -I 'kernel/sched/sched.h' \
      'p::__account_cfs_rq_runtime(struct cfs_rq *cfs_rq) "%d", cfs_rq->runtime_remaining'
        Trace the cfs scheduling runqueue remaining runtime. The struct cfs_rq is defined
        in kernel/sched/sched.h which is in kernel source tree and not in kernel-devel
        package.  So this command needs to run at the kernel source tree root directory
        so that the added header file can be found by the compiler.
trace -I 'net/sock.h' \
      'udpv6_sendmsg(struct sock *sk) (sk->sk_dport == 13568)'
        Trace udpv6 sendmsg calls only if socket's destination port is equal
        to 53 (DNS; 13568 in big endian order)
trace -I 'linux/fs_struct.h' 'mntns_install "users = %d", $task->fs->users'
        Trace the number of users accessing the file system of the current task
  
  
  

 

 

内核版本低的限制
# bcc 基于 eBPF 开发(需要 Linux 3.15 及更高版本)。bcc 使用的大部分内容都需要 Linux 4.1 及更高版本。
  
[root@yang-01 tools]# dbstat mysql -p 10763 -u
Traceback (most recent call last):
  File "/usr/share/bcc/tools/dbstat", line 88, in <module>
    usdt.enable_probe("query__start""probe_start")
  File "/usr/lib/python2.7/site-packages/bcc/usdt.py", line 154, in enable_probe
    probe
bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable
 
[root@yang-01 tools]# dbslower mysql -p 10763 -m 30
Traceback (most recent call last):
  File "/usr/share/bcc/tools/dbslower", line 193, in <module>
    usdt.enable_probe("query__start""query_start")
  File "/usr/lib/python2.7/site-packages/bcc/usdt.py", line 154, in enable_probe
    probe
bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable
 
  
# OS内核版本, >= 4.4 存在统计Bug, 部分功能需要>= 4.13
# "bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable” 需要有Dtrace tracepoint的MySQL, 需要重新编译. https://dev.mysql.com/doc/refman/5.7/en/dba-dtrace-server.html
# 使用bcc需要root权限

 

[root@yang-01 tools]# biolatency -D 2
Tracing block device I/O... Hit Ctrl-C to end.
 
disk = 'vda'
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 63       |***********************************     |
        64 -> 127        : 71       |****************************************|
       128 -> 255        : 46       |*************************               |
       256 -> 511        : 4        |**                                      |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 3        |*                                       |
      4096 -> 8191       : 3        |*                                       |
 
disk = ''
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 0        |                                        |
       128 -> 255        : 0        |                                        |
       256 -> 511        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 1        |****************************************|
 
 
disk = 'vda'
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 10       |***                                     |
        32 -> 63         : 84       |********************************        |
        64 -> 127        : 103      |****************************************|
       128 -> 255        : 43       |****************                        |
       256 -> 511        : 4        |*                                       |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 7        |**                                      |
      4096 -> 8191       : 17       |******                                  |
      8192 -> 16383      : 1        |                                        |
     16384 -> 32767      : 1        |                                        |
     32768 -> 65535      : 1        |                                        |
     65536 -> 131071     : 1        |                                        |
    131072 -> 262143     : 44       |*****************                       |
 
disk = ''
     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 0        |                                        |
       128 -> 255        : 0        |                                        |
       256 -> 511        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 1        |****************************************|
 
  
  
[root@yang-01 tools]# ps aux | grep mysql | grep opt
actiont+ 10763  3.3 18.0 4570952 1070628 ?     Sl   Jul16 1887:06 /opt/mysql/base/8.0.19/bin/mysqld --defaults-file=/opt/mysql/etc/3307/my.cnf --daemonize --pid-file=/opt/mysql/data/3307/mysqld.pid --user=actiontech-mysql --socket=/opt/mysql/data/3307/mysqld.sock --port=3307
[root@yang-01 tools]# filetop -p 10763 -C 5
Tracing... Output every 5 secs. Hit Ctrl-C to end
 
18:01:40 loadavg: 0.14 0.15 0.20 2/1119 31195
 
TID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
10788  mysqld           0      1      0       832     R ibdata1
10909  mysqld           19     0      117     0       R mysql-relay.000044
10794  mysqld           0      99     0       67      R ib_logfile0
10977  mysqld           0      60     0       4       R mysql-relay.000044
10910  mysqld           0      10     0       4       R mysql-bin.000015
10789  mysqld           0      1      0       0       R ib_logfile0
 
18:01:45 loadavg: 0.21 0.16 0.21 2/1119 31212
 
TID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
10909  mysqld           19     0      116     0       R mysql-relay.000044
10794  mysqld           0      98     0       64      R ib_logfile0
10977  mysqld           0      60     0       4       R mysql-relay.000044
10910  mysqld           0      10     0       4       R mysql-bin.000015
 
18:01:50 loadavg: 0.19 0.16 0.20 2/1119 31226
 
TID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
10909  mysqld           15     0      88      0       R mysql-relay.000044
10794  mysqld           0      106    0       70      R ib_logfile0
19780  mysqld           2      2      8       7       R ibzMa2hy
10977  mysqld           0      60     0       4       R mysql-relay.000044
10910  mysqld           0      10     0       4       R mysql-bin.000015
19780  mysqld           1      0      0       0       R mysql-bin.000009
19780  mysqld           1      0      0       0       R mysql-bin.000012
19780  mysqld           1      0      0       0       R mysql-bin.000014
19780  mysqld           1      0      0       0       R mysql-bin.000010
19780  mysqld           1      0      0       0       R mysql-bin.000013
19780  mysqld           1      0      0       0       R mysql-bin.000011
 
 
 
[root@yang-01 tools]# filelife
TIME     PID    COMM             AGE(s)  FILE
18:03:26 1823   ustats           0.00    disk-read-write-test.log
18:03:26 1830   ustats           0.00    disk-read-write-test.log
18:03:29 1736   umon-mgr         0.01    grafana.db-journal
18:03:41 1823   ustats           0.00    disk-read-write-test.log
18:03:41 1784   ustats           0.01    disk-read-write-test.log

 

posted @ 2020-09-09 11:43  Tech_Shrimp  阅读(4654)  评论(2编辑  收藏  举报