【调试】ftrace(三)trace-cmd和kernelshark

之前使用ftrace的时候需要一系列的配置,使用起来有点繁琐,这里推荐一个ftrace的一个前端工具,它就是trace-cmd

trace-cmd安装教程

安装trace-cmd及其依赖库

git clone https://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git/
cd libtraceevent
make
sudo make install

git clone https://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git/
cd libtracefs
make
sudo make install

git clone https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
make
make libs
sudo make install
sudo make install_libs

安装 kernelshark 依赖

sudo apt-get install build-essential git cmake libjson-c-dev -y
sudo apt-get install freeglut3-dev libxmu-dev libxi-dev -y
sudo apt-get install flex bison -y
sudo apt-get install fonts-freefont-ttf -y
sudo apt-get install qtbase5-dev -y

下载kernelshark并安装

git clone git://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
cd kernel-shark/build
cmake ../
make
sudo ./install_gui.sh

注意:禁止使用 sudo apt-get install kernelshark安装!

cmake 版本必须在3.12以上。cmake版本更新方法参考:https://blog.csdn.net/qq_27350133/article/details/121994229

trace-cmd使用方法

可以使用trace-cmd -h 看下支持那些命令。

trace-cmd COMMAND清单

trace-cmd version 3.1.6 ()

usage:
  trace-cmd [COMMAND] ...

  commands:
     record - record a trace into a trace.dat file
     set - set a ftrace configuration parameter
     start - start tracing without recording into a file
     extract - extract a trace from the kernel
     stop - stop the kernel from recording trace data
     restart - restart the kernel trace data recording
     show - show the contents of the kernel tracing buffer
     reset - disable all kernel tracing and clear the trace buffers
     clear - clear the trace buffers
     report - read out the trace stored in a trace.dat file
     stream - Start tracing and read the output directly
     profile - Start profiling and read the output directly
     hist - show a histogram of the trace.dat information
     stat - show the status of the running tracing (ftrace) system
     split - parse a trace.dat file into smaller file(s)
     options - list the plugin options available for trace-cmd report
     listen - listen on a network socket for trace clients
     agent - listen on a vsocket for trace clients
     setup-guest - create FIFOs for tracing guest VMs
     list - list the available events, plugins or options
     restore - restore a crashed record
     snapshot - take snapshot of running trace
     stack - output, enable or disable kernel stack tracing
     check-events - parse trace event formats
     dump - read out the meta data from a trace file
     convert - convert trace file to different version

record

记录trace信息到trace.dat 。

trace-cmd record ['OPTIONS'] ['command']

跟踪ls进程函数调用过程

 # trace-cmd record -p function -e sched_switch ls > /dev/null
 # trace-cmd report
              ls-13587 [002] 106467.860310: function: hrtick_start_fair <-- pick_next_task_fair
              ls-13587 [002] 106467.860313: sched_switch: prev_comm=trace-cmd prev_pid=13587 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=13583 next_prio=120
       trace-cmd-13585 [001] 106467.860314: function: native_set_pte_at <-- __do_fault
       trace-cmd-13586 [003] 106467.860314: function:             up_read <-- do_page_fault
              ls-13587 [002] 106467.860317: function:             __phys_addr <-- schedule
       trace-cmd-13585 [001] 106467.860318: function: _raw_spin_unlock <-- __do_fault
              ls-13587 [002] 106467.860320: function: native_load_sp0 <-- __switch_to
       trace-cmd-13586 [003] 106467.860322: function: down_read_trylock <-- do_page_fault

使用function_graph跟踪irq_handler_entry跟踪点的do_IRQ函数。

 # trace-cmd record -p function_graph -e irq_handler_entry  -l do_IRQ sleep 10
 # trace-cmd report
          <idle>-0     [000] 157412.933969: funcgraph_entry:                  |  do_IRQ() {
          <idle>-0     [000] 157412.933974: irq_handler_entry:    irq=48 name=eth0
          <idle>-0     [000] 157412.934004: funcgraph_exit:       + 36.358 us |  }
          <idle>-0     [000] 157413.895004: funcgraph_entry:                  |  do_IRQ() {
          <idle>-0     [000] 157413.895011: irq_handler_entry:    irq=48 name=eth0
          <idle>-0     [000] 157413.895026: funcgraph_exit:                        + 24.014 us |  }
          <idle>-0     [000] 157415.891762: funcgraph_entry:                  |  do_IRQ() {
          <idle>-0     [000] 157415.891769: irq_handler_entry:    irq=48 name=eth0
          <idle>-0     [000] 157415.891784: funcgraph_exit:       + 22.928 us |  }
          <idle>-0     [000] 157415.934869: funcgraph_entry:                  |  do_IRQ() {
          <idle>-0     [000] 157415.934874: irq_handler_entry:    irq=48 name=eth0
          <idle>-0     [000] 157415.934906: funcgraph_exit:       + 37.512 us |  }
          <idle>-0     [000] 157417.888373: funcgraph_entry:                  |  do_IRQ() {
          <idle>-0     [000] 157417.888381: irq_handler_entry:    irq=48 name=eth0
          <idle>-0     [000] 157417.888398: funcgraph_exit:       + 25.943 us |  }

OPTIONScommand 可以使用 trace-cmd record --h列出。

set

设置ftrace参数配置。

trace-cmd set ['OPTIONS'] ['command']

使能所有的跟踪事件

trace-cmd set -e all

设置function tracer

trace-cmd set -p function

start

开始跟踪,但并不记录信息到文件。

trace-cmd start ['OPTIONS']

extract

从tracer中提取跟踪信息,一般用在start 和stop之后。也可以将跟踪信息从ring buffer 提取到trace.dat。

trace-cmd extract ['OPTIONS']

stop

停止向ring buffer写入跟踪信息。

restart

重新开始记录跟踪信息

show

显示tracing buffer中的信息(trace, snapshot, trace_pipe)。类似于 cat /sys/kernel/debug/tracing/trace

trace-cmd show ['OPTIONS']

reset

对ftrace的设置和ring buffer复位。

trace-cmd  reset ['OPTIONS']

设置每个cpu的buffer大小为1024

 trace-cmd reset -b 1024

删除实例buffer,并重置顶层实例,

  trace-cmd reset -t -a -d

clear

清空ring buffer。

trace-cmd clear ['OPTIONS']

清空所有buffer,包括top level

trace-cmd clear -a

清空指定buffer

trace-cmd clear -B 	buffer-name

report

读取trace的信息,并存储在trace.dat。

trace-cmd report ['OPTIONS'] ['input-file']
 # trace-cmd report
       trace-cmd-16129 [002] 158126.498411: function: __mutex_unlock_slowpath <-- mutex_unlock
       trace-cmd-16131 [000] 158126.498411: kmem_cache_alloc: call_site=811223c5 ptr=0xffff88003ecf2b40 bytes_req=272 bytes_alloc=320 gfp_flags=GFP_KERNEL|GFP_ZERO
       trace-cmd-16130 [003] 158126.498411: function:             do_splice_to <-- sys_splice
           sleep-16133 [001] 158126.498412: function: inotify_inode_queue_event <-- vfs_write
       trace-cmd-16129 [002] 158126.498420: lock_release: 0xffff88003f1fa4f8 &sb->s_type->i_mutex_key
       trace-cmd-16131 [000] 158126.498421: function: security_file_alloc <-- get_empty_filp
           sleep-16133 [001] 158126.498422: function: __fsnotify_parent <-- vfs_write
       trace-cmd-16130 [003] 158126.498422: function: rw_verify_area <-- do_splice_to
       trace-cmd-16131 [000] 158126.498424: function: cap_file_alloc_security <-- security_file_alloc
       trace-cmd-16129 [002] 158126.498425: function: syscall_trace_leave <-- int_check_syscall_exit_work
           sleep-16133 [001] 158126.498426: function: inotify_dentry_parent_queue_event <-- vfs_write
       trace-cmd-16130 [003] 158126.498426: function: security_file_permission <-- rw_verify_area
       trace-cmd-16129 [002] 158126.498428: function: audit_syscall_exit <-- syscall_trace_leave

记录除function traces 之外的所有跟踪信息

 # trace-cmd report -v -F 'function'
       trace-cmd-16131 [000] 158126.498411: kmem_cache_alloc: call_site=811223c5 ptr=0xffff88003ecf2b40 bytes_req=272 bytes_alloc=320 gfp_flags=GFP_KERNEL|GFP_ZERO
       trace-cmd-16129 [002] 158126.498420: lock_release: 0xffff88003f1fa4f8 &sb->s_type->i_mutex_key
       trace-cmd-16130 [003] 158126.498436: lock_acquire: 0xffffffff8166bf78 read all_cpu_access_lock
       trace-cmd-16131 [000] 158126.498438: lock_acquire: 0xffff88003df5b520 read &fs->lock
       trace-cmd-16129 [002] 158126.498446: kfree: call_site=810a7abb ptr=0x0
       trace-cmd-16130 [003] 158126.498448: lock_acquire: 0xffff880002250a80 &per_cpu(cpu_access_lock, cpu)
       trace-cmd-16129 [002] 158126.498450: sys_exit_splice:      0xfffffff5
       trace-cmd-16131 [000] 158126.498454: lock_release: 0xffff88003df5b520 &fs->lock
           sleep-16133 [001] 158126.498456: kfree: call_site=810a7abb ptr=0x0
           sleep-16133 [001] 158126.498460: sys_exit_write:       0x1
       trace-cmd-16130 [003] 158126.498462: kmalloc: call_site=810bf95b ptr=0xffff88003dedc040 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL|GFP_ZERO

只记录kmalloc系统调用中大于1000 bytes 的信息

#trace-cmd report -F 'kmalloc: bytes_req > 1000'
          <idle>-0     [000] 158128.126641: kmalloc: call_site=81330635 ptr=0xffff88003c2fd000 bytes_req=2096 bytes_alloc=4096 gfp_flags=GFP_ATOMIC

stream

实时在shell中显式ftrace结果。

trace-cmd stream ['OPTIONS'] ['command']

profile

类似于trace-cmd record - profile命令,但profile不会将跟踪内容写入文件,可以节省磁盘空间,适用于长时间跟踪。

trace-cmd profile ['OPTIONS'] ['command']

hist

对trace.dat显式统计信息。

trace-cmd hist ['OPTIONS']['input-file']

stat

显示当前ftrace的events、ring buffer等情况。

*Instances:* List all configured ftrace instances.

*Tracer:* if one of the tracers (like function_graph) is active. Otherwise
  nothing is displayed.

*Events:* Lists the events that are enable.

*Event filters:* Shows any filters that are set for any events

*Function filters:* Shows any filters for the function tracers

*Graph functions:* Shows any functions that the function graph tracer should graph

*Buffers:* Shows the trace buffer size if they have been expanded.
   By default, tracing buffers are in a compressed format until they are used.
   If they are compressed, the buffer display will not be shown.

*Trace clock:* If the tracing clock is anything other than the default "local"
   it will be displayed.

*Trace CPU mask:* If not all available CPUs are in the tracing CPU mask, then
   the tracing CPU mask will be displayed.

*Trace max latency:* Shows the value of the trace max latency if it is other than zero.

*Kprobes:* Shows any kprobes that are defined for tracing.

*Uprobes:* Shows any uprobes that are defined for tracing.

*Error log:* Dump the content of ftrace error_log file.
trace-cmd stat ['OPTIONS']

split

以start-time和end-time将trace.dat分为更小的文件。

trace-cmd split ['OPTIONS'] ['start-time' ['end-time']]

options

列出trace-cmd可用的插件。

listen

设置一个端口等待远程连接。连接上后,如果远程主机发来信息,则创建一个trace.HOST:PORT.dat文件记录信息。

trace-cmd listen -p 'port' ['OPTIONS']

agent

trace-cmd agent通过vsocket或TCP端口侦听连接,以控制对计算机的跟踪。然后,agent将在本地计算机上开始跟踪,并将数据传递给控制连接。

trace-cmd agent ['OPTIONS']

setup-guest

创建用于跟踪虚拟机的fifos

list

显示当前ftrace支持的events、tracers、options。

trace-cmd list ['OPTIONS']

restore

恢复崩溃的trace-cmd-record 文件。

trace-cmd restore ['OPTIONS'] ['command'] cpu-file [cpu-file ...]

snapshot

抓取tracing过程中的快照。

*trace-cmd snapshot* ['OPTIONS']

stack

内核的堆栈跟踪

trace-cmd stack [--start][--stop][--reset]

check-events

解析所有events的格式字符串,有助于检查任何跟踪事件格式字符串

trace-cmd check-events ['OPTIONS']

dump

展示 trace-cmd record记录的元数据

# trace-cmd dump --summary -i trace.dat

 Tracing meta data in file trace.dat:
	[Initial format]
		6	[Version]
		0	[Little endian]
		8	[Bytes in a long]
		4096	[Page size, bytes]
	[Header info, 205 bytes]
	[Header event, 205 bytes]
	[Ftrace format, 15 events]
	[Events format, 2 systems]
	[Kallsyms, 7144493 bytes]
	[Trace printk, 2131 bytes]
	[Saved command lines, 117 bytes]
	8 [CPUs with tracing data]
	[12 options]
	[Flyrecord tracing data]
# trace-cmd dump --summary --systems -i trace.dat

 Tracing meta data in file trace.dat:
	[Initial format]
		6	[Version]
		0	[Little endian]
		8	[Bytes in a long]
		4096	[Page size, bytes]
	[Header info, 205 bytes]
	[Header event, 205 bytes]
	[Ftrace format, 15 events]
	[Events format, 3 systems]
		sched 23 [system, events]
		irq 5 [system, events]
		kvm 70 [system, events]
	[Kallsyms, 7144493 bytes]
	[Trace printk, 2131 bytes]
	[Saved command lines, 157 bytes]
	8 [CPUs with tracing data]
	[11 options]
	[Flyrecord tracing data]

convert

trace file转换为不同格式。

trace-cmd convert ['OPTIONS'] ['output-file']

trace file选择合适的压缩算法,压缩为trace_compress.dat

trace-cmd convert --compression any trace_compress.dat

更详细的使用方法详见 https://github.com/rostedt/trace-cmd/tree/master/Documentation/trace-cmd

trace-cmd使用举例

function && function_graph跟踪

列出可用的追踪器

root@firefly:~# trace-cmd list -t
blk function_graph wakeup_dl wakeup_rt wakeup irqsoff function nop

查看可被追踪的函数

root@firefly:~# trace-cmd list -f | grep do_sys_open
do_sys_open

调整函数调用中追踪深度:

root@firefly:~# trace-cmd start -p function_graph --max-graph-depth 5

启用function跟踪并查看输出

root@firefly:~# trace-cmd start -p function
  plugin 'function'
root@firefly:~# trace-cmd show | head -20
# tracer: function
#
# entries-in-buffer/entries-written: 231316/448381   #P:6
#
#                              _-----=> irq                        / _----=       | / _---=> hardirq/softir    || / _--=> preempt-depth
#            ||| /     delay
#      TIMESTAMP  FUNCTION
#       |       |   ||||       |         |
          <idle>-0     [005]607.424217: hrtimer_start_range_ick_nohz_restart
          <idl-0     [005] dn..   607.424218: rtimer_base.isra.1 <-hrtimer_start_range_ns
          <idle>-0 [005] dn..   607.424219: _raw_spck_irqsave <-lock_hrtimer_base.ia.1
          <idle>-0     [005n..   607.424219: do_raw_spin_lock <-_raw_spin_lock_irqsave
          <idle>-0     [005] dn..   607.424220: ktime_add_safe <-hrtimer_start_range_ns
          <idle>-0     [005] dn..   607.424222: enqueue_hrtimer <-hrtimer_start_range_ns
          <idle>-0     [005] dn..   607.424223: tick_program_event <-hrtimer_start_range_ns
          <idle>-0     [005] dn..   607.424224: clockevents_program_event <-tick_program_event
          <idle>-0     [005] dn..   607.424224: ktime_get <-clockevents_program_event
root@firefly:~# 

停止跟踪并清空缓冲区

trace-cmd stop
trace-cmd clear

追踪特定函数

root@firefly:~# trace-cmd start  -p function -l do_sys_open
  plugin 'function'
root@firefly:~# trace-cmd show | head -20
# tracer: function
#
# entries-in-buffer/entries-written: 49/49   #P:6
#
#                  off
#                                                  | / _---                   || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    IMESTAMP  FUNCTION
#              | |       |   ||||       |         |
      lxqt-panel-1100  [002] ....  1342.237668: do_sys_open <-SyS_openat
         systemd-1     [000] ....  1343.172830: do_sys_open <-SyS_openat
      sd-resolve-683   [002] ....  1343.173117: do_sys_open <-SyS_openat
      lxqt-panel-1100  [002] ....  1343.237235: do_sys_open <-SyS_openat
      lxqt-panel-1100  [002] ....  1344.237580: do_sys_open <-SyS_openat
      lxqt-panel-1100  [002] ....  1345.237314: do_sys_open <-SyS_openat
      lxqt-panel-1100  [002] ....  1346.237657: do_sys_open <-SyS_openat
      lxqt-panel-1100  [002] ....  1347.237311: do_sys_open <-SyS_openat
            head-1289  [004] ....  1348.034549: do_sys_open <-SyS_openat

不追踪某个函数

root@firefly:~# trace-cmd start  -p function -n do_sys_open

追踪特定PID

trace-cmd record -P 10885 -p function_graph

trace-cmd starttrace-cmd record参数类似,trace-cmd record 会记录追踪内容到trace.dat。

追踪指定的进程

root@firefly:~# trace-cmd record -p function -F ls
root@firefly:~# trace-cmd report trace.dat | head -20
cpus=6
              ls-1286  [005]   650.095974: function:             mutex_unlock
              ls-1286  [005]   650.095976 function:                __mutewpath
              ls-1286  [005]   650.095977: function:            _raw_spin_lock
              ls-1286  [005]   650.095977: function:                     n_lock
              ls-1286  [   650.095978: function:                  wake_q_add
          s-1286  [005]   650.095979: function:                   _raw_spin_unlock
              ls-1286  [005]   650.095979: function:                      do_raw_spin_unlock
              ls-1286  [005]   650.095980: function:                   wake_up_q
              ls-1286  [005]   650.095980: function:                      try_to_wake_up
              ls-1286  [005]   650.095981: function:                         _raw_spin_lock_irqsave
              ls-1286  [005]   650.095981: function:                            do_raw_spin_lock
              ls-1286  [005]   650.095982: function:                         _raw_spin_lock
              ls-1286  [005]   650.095983: function:                            do_raw_spin_lock
              ls-1286  [005]   650.095984: function:                         _raw_spin_unlock
              ls-1286  [005]   650.095984: function:                            do_raw_spin_unlock
              ls-1286  [005]   650.095984: function:                         task_waking_fair
              ls-1286  [005]   650.095985: function:                            record_wakee
              ls-1286  [005]   650.095986: function:                         select_task_rq_fair
              ls-1286  [005]   650.095986: function:                            record_wakee

event跟踪

查看所有这些事件的列表:

cat /sys/kernel/debug/tracing/available_events

事件跟踪模板

trace-cmd record -e 'subsystem:event-name'

追踪sched子系统的sched_switch事件

root@firefly:~# trace-cmd record -e sched:sched_switch
Hit Ctrl^C to stop recording
^C^C^C^C^CCPU0 data recorded at offset=0x117000
    214 bytes in size (4096 uncompressed)
CPU1 data recoed at offset=0x118000
    293 bytes in size (4096 uncompressed)rded at offset=0x119000
    260096 uncompressed)
CPU3 data recorded at offset=0x11a000
    11 uncompressed)
CPU4 data recordset=0x11b000
    260 bytes in sncompressed)
CPU5 data recordedffset=0x11c000
    914 bytes in size (12288 uncompressed)
root@firefly:~# trace-cmd report trace.dat | head -20
cpus=6
       trace-cmd-1298  [000]   857.010582: sched_switch:         trace-cmd:1298 [120] S 004]   857.010631: sched_switch::1299 [120]
          <idle>-0 pper/5:0 [120] R ==> trace-cmd:1857.010704: sched_switch:       race-cmd:1304 [120]
       trac sched_switch:         trace-cmd==> swapper/4:0 [120]
       tr  [005]   857.010744: sched_switrace-cmd:1304 [120] S ==> swappe0]
          <idle>-0     [001]        swapper/1:0 [120] R ==>  [120]
          <idle>-0     [005]   857.011293: sched_switch:         swapper/5:0 [120] R ==> trace-cmd:1300 [120]
       trace-cmd-1300  [005]   857.011342     trace-cmd:1300 [120] S ==> 120]
       rcu_sched-7     [00  857.011358: sched_switch:         rcu_sched:7 [120] S ==> swapper/1:0 [120]
          <idle>-0     [001]   857.014074: sched_switch:         swapper/1:0 [120] R ==> rcu_sched:7 [120]
       rcu_sched-7     [001]   857.014150: sched_switch:         rcu_sched:7 [120] S ==> swapper/1:0 [120]
          <idle>-0     [005]   857.014948: sched_switch:         swapper/5:0 [120] R ==> kworker/5:2:136 [120]
     kworker/5:2-136   [005]   857.014971: sched_switch:         kworker/5:2:136 [120] S ==> swapper/5:0 [120]
          <idle>-0     [001]   857.017159: sched_switch:         swapper/1:0 [120] R ==> rcu_sched:7 [120]
       rcu_sched-7     [001]   857.017267: sched_switch:         rcu_sched:7 [120] S ==> swapper/1:0 [120]
          <idle>-0     [001]   857.020239: sched_switch:         swapper/1:0 [120] R ==> rcu_sched:7 [120]
       rcu_sched-7     [001]   857.020336: sched_switch:         rcu_sched:7 [120] S ==> swapper/1:0 [120]
          <idle>-0     [005]   857.058091: sched_switch:         swapper/5:0 [120] R ==> kworker/5:2:136 [120]

只打印下一个进程ID大于800的跟踪结果

root@firefly:~# trace-cmd report -F 'sched_switch: next_pid > 800'
cpus=6
       trace-cmd-1375  [005]  1891.087860: sched_switch:         trace-cmd:1375 [120] R   1891.087913: sched_switch:    5 [120]
       trace-cmd-1376           trace-cmd:1376 [120] R idle>-0     [005]  1891.088014: apper/5:0 [120] R ==> trace-cmd:371  [004]  1891.088048: sched_sh:         trace-cmd:1371 [120] 376 [120]
          <idle>-0   91.090580: sched_switch:        r/4:0 [120] R ==> trace-cmd:1373    <idle>-0     [000]  1891.371      swapper/0:0 [120] R ==> kw1342 [120]
          <idle>-0  1]  1891.403968: sched_switch:         swapper/1:0 [120] R ==> gvfs-afc-volume:1161 [120]
      <idle>-0     [000]  1891.586038: sched_switch:         swapper/0:0 [120] R ==> kworker/u12:0:1342 [120]
          <idle>-0     [001]  1891.586262: sched_switch:         swapper/1:0 [120] R ==> trace-cmd:1370 [120]

事件打印格式可参考format

root@firefly:~# cat /sys/kernel/debug/tracing/events/sched/sched_switch/format
name: sched_switch
ID: 250
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:char prev_comm[16];       offset:8;       size:16;        signed:0;
        field:pid_t prev_pid;   offset:24;      size:4; signed:1;
        field:int prev_prio;    offset:28;      size:4; signed:1;
        field:long prev_state;  offset:32;      size:8; signed:1;
        field:char next_comm[16];       offset:40;      size:16;        signed:0;
        field:pid_t next_pid;   offset:56;      size:4; signed:1;
        field:int next_prio;    offset:60;      size:4; signed:1;

print fmt: "prev_comm=%s prev_pid=%d prev_prio=%d prev_state=%s%s ==> next_comm=%s next_pid=%d next_prio=%d", REC->prev_comm, REC->prev_pid, REC->prev_prio, REC->prev_state & (4096-1) ? __print_flags(REC->prev_state & (4096-1), "|", { 1, "S"} , { 2, "D" }, { 4, "T" }, { 8, "t" }, { 16, "Z" }, { 32, "X" }, { 64, "x" }, { 128, "K" }, { 256, "W" }, { 512, "P" }, { 1024, "N" }) : "R", REC->prev_state & 4096 ? "+" : "", REC->next_comm, REC->next_pid, REC->next_prio

kernelshark的使用

kernelshark记录事件

通过在命令行输出kernelshark可以启动kernelshark。通过菜单栏执行Tools->Record,会弹出Capture对话框。

常用的设置有Events和Plugin,比如这里设置了sched_switch和sched_wakeup两个事件,选择了function_graph插件。

然后点击apply应用配置,capture开始捕获。ctrl+c可以停止捕获。

Close关闭后,可以在Kenelshark中查看细节。

可以看出这里不光显示了sched_switch和sched_wakeup事件,同时还显示了function_graph函数调用。

数据过滤

通常获取的数据需要进行一些过滤,才能发现问题。

filter提供了丰富的过滤功能,比如过滤事件,CPU,Tasks等。

也可以使用高级过滤功能,可以

如果想清除所有过滤信息,点击clear all filters 即可。

图表过滤

Plots提供了CPU和Tasks的过滤,如下所示,我们选择只展示CPU0的nautilus进程。

最后筛选结果

其它功能

还有一些其它有用的功能,比如搜索功能,选定要搜索的Column,文本框中输入字符串即可。

如下,我们筛选出PDI为3038的进程,总共找到44831个,当前是第1937个。

本文参考

https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git/about/

https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git/about/

https://github.com/rostedt/trace-cmd/tree/master/Documentation/trace-cmd

https://www.cnblogs.com/arnoldlu/p/9014365.html

https://cloud.tencent.com/developer/article/1876479

https://simpleiot.blog.csdn.net/article/details/103587609

posted @ 2023-04-07 22:27  学习,积累,成长  阅读(869)  评论(0编辑  收藏  举报