linux性能分析工具之火焰图

一.环境

1.1 jello@jello:~$ uname -a

Linux jello 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

1.2 jello@jello-Inspiron-N4050:~$ lsb_release -a

Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

二.准备工作.

2.1安装systemtap (火焰图依赖于此工具)

sudo apt-get install systemtap

2.2 查找内核对应的debug包

jello@jello$ uname -r
4.4.0-98-generic

那么接下来就是去http://ddebs.ubuntu.com/pool/main/l/linux/下载对应的debug包,找4.4.0-98-generic一致的

2.3 下载对应的debug包

wget http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb  (这是笔者自己找到的对应下载路径)

2.4 安装debug包

sudo dpkg -i linux-image-4.4.0-98-generic-dbgsym_4.4.0-98.121_amd64.ddeb

2.5 安装nginx

sudo apt-get install nginx

此时在浏览器中输入localhost即可出现以下信息表明安装ok

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

 

2.5 编写systemtap脚本nginx.systemtap,内容如下:

global s;
global quit=0;
probe timer.profile {
if (pid() == target()){
if (quit) {
foreach (i in s-) {
print_ustack(i);
printf("\t%d\n",@count(s[i]));
}
exit();
}
else {
s[ubacktrace()] <<< 1;
}
}
}
probe timer.s(20){
quit = 1
}

2.6 使用systemtap

sudo stap --ldd -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out

各参数解析:

--ldd，添加通过ldd解析出来的所有共享库符号表信息以便为probe到的用户空间二进制提供信息或者以-d选项列出来，注意：这会使得probe模块相当的大

-d /usr/sbin/nginx，为给定的模块(这里是nginx)添加符号表信息到内核对象模块,这可能使能这些模块或者程序的象征性traceback，即使他们没有显式probe到他们里面

--all-modules,相当于为所有当前被加载的模块指定'-dkernel'和‘-d'选项

-D MAXMAPENTRIES=256 ,添加给定的c预处理直接到模块makefile中,这些能被用来限制以下描述的参数,MAXMAPENTRIES=256,设定默认全局数组的最大默认长度为256，通过了解，是用来指定堆栈大小的  

 -D MAXACTION=20000 设定单个probe hit包含执行的最大语句数目

-D MAXTRACE=100 

-D MAXSTRINGLEN=4096 设定字符串最大长度

-D MAXBACKTRACE=100 设定栈帧的最大数目

-x 2082 设定target()的pid为2082

--vp 0001

 

笔者运行之后出现以下错误信息：

jello@jello:~$ sudo stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out
In file included from /usr/share/systemtap/runtime/transport/control.c:14:0,
from /usr/share/systemtap/runtime/transport/transport.c:76,
from /usr/share/systemtap/runtime/linux/print.c:17,
from /usr/share/systemtap/runtime/print.c:17,
from /usr/share/systemtap/runtime/runtime_context.h:22,
from /tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.c:221:
/usr/share/systemtap/runtime/transport/symbols.c: In function ‘_stp_module_update_self’:
/usr/share/systemtap/runtime/transport/symbols.c:243:44: error: ‘struct module’ has no member named ‘symtab’
if (attr->address == (unsigned long) mod->symtab)
^
/usr/share/systemtap/runtime/transport/symbols.c:245:9: error: ‘struct module’ has no member named ‘num_symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
/usr/share/systemtap/runtime/transport/symbols.c:245:34: error: ‘struct module’ has no member named ‘symtab’
mod->num_symtab * sizeof(mod->symtab[0]);
^
make[1]: *** [/tmp/stapiPADvo/stap_20923465b7e2011fc7e903a669485b1f_9417_src.o] Error 1
make: *** [_module_/tmp/stapiPADvo] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_20923465b7e2011fc7e903a669485b1f_9417.ko" in 9530usr/380sys/12191real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started

 

从错误信息来看是某个mod的某个字段不存在，所以断定是内核版本过高，但是systemtap版本过低导致的，因此升级systemtap版本

2.6.1 升级systemtap版本

2.6.1.1获取最新systemtap代码

方案一.git clone git://sourceware.org/git/systemtap.git  (下载速度比较慢,虽然配置FQ了，仍然慢，推荐方案二)

方案二.tsocks wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.2.tar.gz (tsocks使能wget通过FQ下载源码，需要配置vpn进行FQ)

以上方案均需FQ，否则下载会非常慢!

2.6.1.2解压systemtap源码

tar xvf systemtap-3.2.tar.gz

2.6.1.3 切换到解压后的systemtap目录

cd systemtap-3.2

2.6.1.4 进行编译

2.6.1.4.1 根据README得知：

需要安装带有libdwfl库的elfutils (版本号需要大于或等于0.151)才能正常配合systemtap;

2.6.1.4.2 安装elfutils

sudo apt-get install elfutils

2.6.1.4.3 验证elfutils的版本信息

eu-objdump -V

2.6.1.4.4 安装必不可少的编译工具

sudo apt-get build-dep systemtap

2.6.1.4.5 创建编译目录

mkdir build

2.6.1.4.6 切换到build目录

cd build

2.6.1.4.7 进行编译前的配置

../configure

2.6.1.4.8 进行正式编译

make all

2.6.1.4.8.1 编译过程中出现以下问题

../bpf-translate.cxx:37:29: fatal error: elfutils/libebl.h: 没有那个文件或目录

2.6.1.4.8.2 查看bpf-translate.cxx

#if _ELFUTILS_PREREQ (0, 167)
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif

2.6.1.4.8.2 修改以上内容为:

#if 1
#include <elfutils/libdwelf.h>
typedef Dwelf_Strent Stap_Strent;
typedef Dwelf_Strtab Stap_Strtab;
#define stap_strtab_init dwelf_strtab_init
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
#define stap_strtab_free dwelf_strtab_free
#define stap_strtab_finalize dwelf_strtab_finalize
#define stap_strent_offset dwelf_strent_off
#else
#include <elfutils/libebl.h>
typedef Ebl_Strent Stap_Strent;
typedef Ebl_Strtab Stap_Strtab;
#define stap_strtab_init ebl_strtabinit
#define stap_strtab_add(X,Y) ebl_strtabadd(X,Y,0)
#define stap_strtab_free ebl_strtabfree
#define stap_strtab_finalize ebl_strtabfinalize
#define stap_strent_offset ebl_strtaboffset
#endif

2.6.1.4.8.3 通过以上修改出现以下错误信息:

make all-recursive
make[1]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
Making all in .
make[2]: Entering directory '/home/jello/data/development/systemtap-3.2/build'
CXX stap-bpf-translate.o
../bpf-translate.cxx:29:9: error: ‘Dwelf_Strent’ does not name a type
typedef Dwelf_Strent Stap_Strent;
^
../bpf-translate.cxx:30:9: error: ‘Dwelf_Strtab’ does not name a type
typedef Dwelf_Strtab Stap_Strtab;
^
../bpf-translate.cxx:1773:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Section::BPF_Section(const string&)’:
../bpf-translate.cxx:1782:22: error: class ‘bpf::BPF_Section’ does not have any field named ‘name_ent’
: scn(0), name(n), name_ent(0), data(0), free_data(false)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1794:3: error: ‘Stap_Strent’ does not name a type
Stap_Strent *name_ent;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Symbol::BPF_Symbol(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1801:14: error: class ‘bpf::BPF_Symbol’ does not have any field named ‘name_ent’
: name(n), name_ent(0)
^
../bpf-translate.cxx: At global scope:
../bpf-translate.cxx:1812:3: error: ‘Stap_Strtab’ does not name a type
Stap_Strtab *str_tab;
^
../bpf-translate.cxx: In constructor ‘bpf::BPF_Output::BPF_Output(int)’:
../bpf-translate.cxx:1827:5: error: class ‘bpf::BPF_Output’ does not have any field named ‘str_tab’
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx:1827:34: error: ‘dwelf_strtab_init’ was not declared in this scope
str_tab(stap_strtab_init(true))
^
../bpf-translate.cxx: In destructor ‘bpf::BPF_Output::~BPF_Output()’:
../bpf-translate.cxx:1835:20: error: ‘str_tab’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx:1835:27: error: ‘dwelf_strtab_free’ was not declared in this scope
stap_strtab_free(str_tab);
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Section* bpf::BPF_Output::new_scn(const string&)’:
../bpf-translate.cxx:1854:6: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:1854:33: error: ‘str_tab’ was not declared in this scope
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1854:17: note: in expansion of macro ‘stap_strtab_add’
n->name_ent = stap_strtab_add(str_tab, n->name.c_str());
^
../bpf-translate.cxx: In member function ‘bpf::BPF_Symbol* bpf::BPF_Output::new_sym(const string&, bpf::BPF_Section*, long int)’:
../bpf-translate.cxx:1864:6: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:1864:33: error: ‘str_tab’ was not declared in this scope
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx:32:48: note: in definition of macro ‘stap_strtab_add’
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:32:51: error: ‘dwelf_strtab_add’ was not declared in this scope
#define stap_strtab_add(X,Y) dwelf_strtab_add(X,Y)
^
../bpf-translate.cxx:1864:17: note: in expansion of macro ‘stap_strtab_add’
s->name_ent = stap_strtab_add(str_tab, s->name.c_str());
^
../bpf-translate.cxx: In function ‘void bpf::output_symbols_sections(bpf::BPF_Output&)’:
../bpf-translate.cxx:2183:31: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2183:49: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2190:39: error: ‘struct bpf::BPF_Symbol’ has no member named ‘name_ent’
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2190:47: error: ‘dwelf_strent_off’ was not declared in this scope
b->st_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2196:29: error: ‘struct bpf::BPF_Output’ has no member named ‘str_tab’
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2196:47: error: ‘dwelf_strtab_finalize’ was not declared in this scope
stap_strtab_finalize(eo.str_tab, str->data);
^
../bpf-translate.cxx:2203:48: error: ‘struct bpf::BPF_Section’ has no member named ‘name_ent’
s->shdr->sh_name = stap_strent_offset(s->name_ent);
^
../bpf-translate.cxx:2203:56: error: ‘dwelf_strent_off’ was not declared in this scope
s->shdr->sh_name = stap_strent_offset(s->name_ent);

 从以上信息可知：以上修改反而会引入更多的问题,因此看一下代码

2.6.1.4.8.4 将sytemtap的代码还原

2.6.1.4.8.5 清掉build目录

rm -rf build

2.6.1.4.8.6 重新创建build目录

mkdir build

2.6.1.4.8.7 切换到build目录

cd build

2.6.1.4.8.8 查找elfutils目录

sudo find / |grep elfutils

笔者找到elfutils对应的头文件路径是/usr/include (默认使用apt-get install安装的软件对应的头文件在/usr/include目录下)

2.6.1.4.8.9 指定elfutils的头文件路径进行配置并且指定安装目录为/opt/

rm  * -rf (需要清一下之前在build目录下生成的文件)

../configure --with-elfutils=/usr/include --prefix=/opt/

出现以下错误信息:

checking for libvirt... no
checking for libxml2... yes
configure: WARNING: will not build systemtap virt support, cannot find libvirt headers
checking for python-config... /usr/bin/python-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for python3-config... /usr/bin/python3-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for jsonc... no
checking for ncurses... yes
checking for assembler .section "?" flags support... yes
checking linux/bpf.h usability... yes
checking linux/bpf.h presence... yes
checking for linux/bpf.h... yes
configure: error: No /usr/include/configure, forgot to run autoreconf -i?

从此信息中发现缺少configure，应该与elfutils有关,那么下载与当前elfutils对应的版本源代码

2.6.1.4.8.10 查看当前elfutils版本的方法:

eu-objdump -V

objdump (elfutils) 0.165
Copyright (C) 2012 Red Hat, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Ulrich Drepper.

版本为0.165

2.6.1.4.8.11 获取elfutils的源码

笔者找到的0.165下载地址为:https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2

2.6.1.4.8.12 下载elfutils源码

tsocks wget https://sourceware.org/elfutils/ftp/0.165/elfutils-0.165.tar.bz2 (FQ下载才快)

2.6.1.4.8.13解压elfutils

tar xvf elfutils-0.165.tar.bz2

2.6.1.4.8.14 指定elfutils源码的路径(笔者的elfutils源码路径在/home/jello/elfutils-0.165下)并指定systemtap的安装路径

../configure --with-elfutils=/home/jello/elfutils-0.165 --prefix=/opt/

2.6.1.4.8.15 开始编译

make

2.6.1.4.8.16 编译又出现错误

systemtap-3.2/build/python/../includes/sys -fPIC -I/usr/include/python2.7 -c HelperSDT/_HelperSDT.c -o /home/jello/systemtap-3.2/build/python/py2build/temp.linux-x86_64-2.7/HelperSDT/_HelperSDT.o
HelperSDT/_HelperSDT.c:10:21: fatal error: sys/sdt.h: 没有那个文件或目录
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:621: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2
从以上信息可知:没有头文件sys/sdt.h,那么就查找sys/sdt.h的路径

2.6.1.4.8.17 查找sys/sdt.h的路径

sudo find /|grep "sys/sdt.h"

发现sys/sdt.h在systemtap-3.2的includes目录下

2.6.1.4.8.18 修改build/python目录下的Makefile

修改前的内容:

AM_CPPFLAGS = -I$(srcdir)/../includes \
        -I$(abs_builddir)/../includes/sys修改后的内容:AM_CPPFLAGS = -I$(srcdir)/../includes \
        -I$(abs_builddir)/../includes/sys \ 　　-I$(abs_srcdir)/../includes
2.6.1.4.8.19 再次编译又出现以下错误(cd ../../python; CFLAGS="-I../../python/../includes -I/home/jello/systemtap-3.2/build/python/../includes/sys -I/home/jello/systemtap-3.2/build/../python/../includes" /usr/bin/python3 setup.py build \
         --build-base /home/jello/systemtap-3.2/build/python/py3build \
         --verbose)
Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    from setuptools import setup, Extension
ImportError: No module named 'setuptools'
Makefile:622: recipe for target 'all-local' failed
make[2]: *** [all-local] Error 1
make[2]: Leaving directory '/home/jello/data/development/systemtap-3.2/build/python'
Makefile:2012: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jello/data/development/systemtap-3.2/build'
Makefile:717: recipe for target 'all' failed
make: *** [all] Error 2


从以上信息可知:python缺少setuptools模块,所以安装setuptools模块,并且使用python3进行安装
2.6.1.4.8.20 安装setuptools模块
2.6.1.4.8.20.1 获取模块

wget https://bootstrap.pypa.io/ez_setup.py

2.6.1.4.8.20.2 安装模块
sudo python3 ez_setup.py2.6.1.4.8.21 继续编译
make一切ok2.6.1.4.9 安装systemtapsudo make install2.6.1.4.10 卸载原来使用apt-get install安装的systemtapsudo apt-get remove systemtap2.6.1.4.11 设置环境变量使新安装的stap生效
vi /etc/profile往文件最后加入一句:export "PATH=/opt/bin:$PATH"2.6.1.5 验证systap是否安装成功stap -VSystemtap translator/driver (version 3.2/0.165, non-git sources)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.11
enabled features: AVAHI PYTHON2 PYTHON3 LIBSQLITE3 LIBXML2 NLS NSS READLINE以上信息中发现版本为3.2/0.165，表明systemtap已经安装成功

 

2.6.2 启动stap

2.6.2.1 先切换为最高权限

sudo su -l

2.6.2.2 启动stap  (之前写过一个nginx.systemtap文件，记得指定该文件路径，笔者将nginx.systemtap放在了当前目录下，因此直接的指定文件）

stap --ld -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 nginx.systemtap --vp 0001 > nginx.out

 

 2.7重开终端进行压力测试(可选)

进行压力测试:

ab -n 900000 -c 50 http://192.168.2.323/index.php  /*192.168.2.323为笔者的本地ip地址，注意修改成自己的本地ip地址*/

 2.8 FlameFraph

2.8.1 获取FlameGraph

git clone https://github.com/brendangregg/FlameGraph.git

2.8.2

 

由于2.6.2.2一直未生成nginx.out,因此需要找一下nginx.systemtap是否有错误

未写完，待续...