系统分析与排障利器sysdig

 

 

【简介】

https://github.com/draios/sysdig

 

【安装】

安装方法：

CentOS, RHEL, Fedora, Amazon Linux

Trust the Draios GPG key, configure the yum repository
rpm --import https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public  
curl -s -o /etc/yum.repos.d/draios.repo https://s3.amazonaws.com/download.draios.com/stable/rpm/draios.repo
Install the EPEL repository
Note: The following command is required only if DKMS is not available in the distribution. You can verify if DKMS is available with yum list dkms

rpm -i https://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm
Install kernel headers
Warning: The following command might not work with any kernel. Make sure to customize the name of the package properly

yum -y install kernel-devel-$(uname -r)
Install sysdig
yum -y install sysdig

 

 

【使用】

 sysdig -cl  列出可用的目录

[loki@loki-c7t1 ~]$ sysdig -cl |more

Category: Application  # 应用
---------------------
httplog         HTTP requests log
httptop         Top HTTP requests
memcachelog     memcached requests log

Category: CPU Usage  # CPU用量
-------------------
spectrogram     Visualize OS latency in real time.
subsecoffset    Visualize subsecond offset execution time.
topcontainers_cpu
                Top containers by CPU usage
topprocs_cpu    Top processes by CPU usage

Category: Errors   # 错误信息
----------------
topcontainers_error
                Top containers by number of errors
topfiles_errors Top files by number of errors
topprocs_errors top processes by number of errors

Category: I/O    # 磁盘IO相关
-------------
echo_fds        Print the data read and written by processes.
fdbytes_by      I/O bytes, aggregated by an arbitrary filter field
fdcount_by      FD count, aggregated by an arbitrary filter field
fdtime_by       FD time group by
iobytes         Sum of I/O bytes on any type of FD
iobytes_file    Sum of file I/O bytes
spy_file        Echo any read/write made by any process to all files. Optionall
                y, you can provide the name of one file to only intercept reads
                /writes to that file.
stderr          Print stderr of processes
stdin           Print stdin of processes
stdout          Print stdout of processes
topcontainers_file
                Top containers by R+W disk bytes
topfiles_bytes  Top files by R+W bytes
topfiles_time   Top files by time
topprocs_file   Top processes by R+W disk bytes
udp_extract     extract data from UDP streams to files.

Category: Logs    # 系统日志相关
--------------
spy_logs        Echo any write made by any process to a log file. Optionally, e
                xport the events around each log message to file.
spy_syslog      Print every message written to syslog. Optionally, export the e
                vents around each syslog message to file.

Category: Misc
--------------
around          Export to file the events around the time range where the given
                 filter matches.

Category: Net    # 网络相关
-------------
iobytes_net     Show total network I/O bytes
spy_ip          Show the data exchanged with the given IP address
spy_port        Show the data exchanged using the given IP port number
topconns        Top network connections by total bytes
topcontainers_net
                Top containers by network I/O
topports_server Top TCP/UDP server ports by R+W bytes
topprocs_net    Top processes by network I/O

Category: Performance    #  性能相关
---------------------
bottlenecks     Slowest system calls
fileslower      Trace slow file I/O
netlower        Trace slow network I/0
proc_exec_time  Show process execution time
scallslower     Trace slow syscalls
topscalls       Top system calls by number of calls
topscalls_time  Top system calls by time

Category: Security    #  安全相关
------------------
list_login_shells
                List the login shell IDs
shellshock_detect
                print shellshock attacks
spy_users       Display interactive user activity

Category: System State    # 系统状态
----------------------
lscontainers    List the running containers
lsof            List (and optionally filter) the open file descriptors.
netstat         List (and optionally filter) network connections.
ps              List (and optionally filter) the machine processes.

Category: Tracers
-----------------
tracers_2_statsd
                Export spans duration as statds metrics.

Use the -i flag to get detailed information about a specific chisel

 

 

# 使用-i 查看具体的信息的介绍

sysdig -i topcontainers_cpu

 

 

 

# 监控交互用户活动

自行创建一个日志存放目录：mkdir -p /log/sysdig/ 

-z 压缩

-w 写入

-r 读取

# 开始监控当前用户正在操作的命令

sudo  sysdig -z -w /log/sysdig/loki.gz -c spy_users

 

# 读取监控信息

sysdig -r /log/sysdig/loki.gz -c spy_users  

 

# 查看R+W读写量最大的文件

sysdig -c topfiles_bytes

 

# 查看占用网络带宽最多的进程

sysdig -c topprocs_net

 

# 显示主机192.168.x.x的网络传输数据

as binary:
sysdig -s2000 -X -c echo_fds fd.cip=192.168.x.x
as ASCII:
sysdig -s2000 -A -c echo_fds fd.cip=192.168.x.x

 

# 查看服务器连接最多的端口

sysdig -c fdbytes_by fd.sport

 

# 查看服务器连接最多的IP

sysdig -c fdbytes_by fd.cip

 

# 查看CPU占用量最大的进程

sysdig -c topprocs_cpu 

 

# 列出所有不是访问apache服务的访问连接

sysdig -p"%proc.name %fd.name" "proc.name!=httpd"

 

# 查看机器所有的HTTP请求

sudo sysdig -s 2000 -A -c echo_fds fd.port=80 and evt.buffer contains GET

 

 

参考资料：

https://www.oschina.net/p/sysdig

https://github.com/draios/sysdig/wiki/sysdig-examples