随笔-cpu profiling-perf record on-cpu off-cpu

perf on-cpu

xpid=$(cat /var/run/xx.pid); perf record -F 99 -p $xpid --call-graph dwarf -- sleep 60

直接在控制台上查看：perf report
或者生成火焰图：
xdata=...; perf script --header -i $xdata > $xdata.stacks
xfile=...;/opt/FlameGraph/stackcollapse-perf.pl < $xfile | /opt/FlameGraph/flamegraph.pl --hash > $xfile.svg

FlameGraph: git clone https://github.com/brendangregg/FlameGraph

perf off-cpu

# test in centos7

echo 1 | sudo tee /proc/sys/kernel/sched_schedstats
xpid=$(cat /var/run/xx.pid); perf record -e sched:sched_stat_sleep -e sched:sched_switch -e sched:sched_process_exit -p $xpid -g -o perf-offcpu.data.raw sleep 30
perf inject -v -s -i perf-offcpu.data.raw -o perf-offcpu.data # The step is merging sched_start and sched_switch events

perf script -i perf-offcpu.data -F comm,pid,tid,cpu,time,period,event,ip,sym,dso,trace | awk '
    NF > 4 { exec = $1; period_ms = int($5 / 1000000) }
    NF > 1 && NF <= 4 && period_ms > 0 { print $2 }
    NF < 2 && period_ms > 0 { printf "%s\n%d\n\n", exec, period_ms }' | \
    /opt/FlameGraph/stackcollapse.pl | \
    /opt/FlameGraph/flamegraph.pl --countname=ms --title="Off-CPU Time Flame Graph" --colors=io > offcpu.svg

使用bcc tool offcputime

# /usr/share/bcc/tools/offcputime -df -p `pgrep -nx mysqld` 30 > out.stacks
[...copy out.stacks to your local system if desired...]
# git clone https://github.com/brendangregg/FlameGraph
# cd FlameGraph
# ./flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" --countname=us < out.stacks > out.svg

参考

1. Linux perf_events Off-CPU Time Flame Graph
2. https://perf.wiki.kernel.org/index.php/Tutorial
3. PingCAP 工欲性能调优，必先利其器（2）- 火焰图
4. 透过Tracepoint理解内核 - 调度器框架和性能 - J.FW的文章 - 知乎