随笔 - 60,  文章 - 0,  评论 - 3,  阅读 - 63968

 

为方便问题发生后,问题原因的分析排查,我们可以在服务器中事先部署如下脚本,方便故障发生后,问题原因的分析排查

 

脚本部署方法:

1.将脚本【linux_reports.sh】上传到服务器

2.登陆虚拟机,并切换到root用户

3.执行命令: chmod +x  <path>/linux-reports.sh  【<path>替换成实际脚本上传的目录】

4.执行命令:crontab -e

5.添加定时任务:* * * * * <the path of the script>/linux-reports.sh <ouputs name> <keeps>

备注:

范例:*/6 * * * * /root/linux-reports.sh performancelog 120 &

a.performancelog为输出日志名称k,可以根据实际情况任意指定

b.120为性能数据保留的最长期限,单位为分钟。比如120,就是代表所有的性能数据收集的时间在120分钟之内的予以保留,超过120分钟的会被删除。如果想保留超过一天及以上的,只有输入确定的分钟数即可,比如保留三天,那么分钟数为4320。

c.所有的性能数据都存放在目录/var/tmp/下. 命名类似:

 6.可以通过如下方法查看搜集到的性能日志:

1
2
3
4
5
6
7
8
9
10
11
12
gunzip performancelog_hlmcent69nma_070001-22-Nov-2017.tar.gz
tar xvf performancelog_hlmcent69nma_070001-22-Nov-2017.tar
cd hlmcent69nma_070001-22-Nov-2017/
[root@hlmcent69nma hlmcent69nma_070001-22-Nov-2017]# ll
total 28
drwxr-xr-x. 2 root root 4096 Nov 22 07:00 cpu
drwxr-xr-x. 2 root root 4096 Nov 22 07:00 disks
drwxr-xr-x. 2 root root 4096 Nov 22 07:00 log
drwxr-xr-x. 2 root root 4096 Nov 22 07:00 memory
drwxr-xr-x. 2 root root 4096 Nov 22 07:00 network
drwxr-xr-x. 2 root root 4096 Nov 22 07:00 sar
-rw-r--r--. 1 root root  489 Nov 22 07:00 vminfo

  

脚本详情如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
#!/bin/bash
#
# Verify Linux Distro - this will work with RHEL, SuSE, Ubuntu
# Verify if iostat is installed if not it will install
# Collect memory info and stats
# Collect disk data
# Collect sar if available
# Collect logs
########################################
 
exec 2> /var/tmp/linux-reports.log
 
# Frequency and repetitions for iostat and vmstat
fre=1
#rep=60
 
#to allow enough time to complete
safetime=20
 
clear
 
## Start to comment out
#echo "By running this script you agree for data collection from your Virtual Machine"
#echo -n "sysstat (enabling iostat) will be installed if not already installed -  Please enter Y or N [ENTER]: "
         
#    read answer
 
#if [ "$answer" = "Y" ] || [ "$answer" = "y" ]
#  then
#    echo " "
#    echo -n "Please supply a 15 digit Microsoft case number: "
#    read sr
#    LEN=$(echo ${#sr})
 
# if [ $LEN -ne 15 ]; then
#        echo $sr "Microsoft case number should be of 15 digits"
#        exit
# fi
#else
 
#if [ "$answer" = "N" ]
#        then
#        echo "you answered  "$answer "exiting.."
#        fi
#exit
#fi
 
#export case=$sr
#direc="/var/tmp/$sr"
 
## End to comment out
 
if [ $# != 2 ]; then
        echo "Invalid option, try chpa.sh <case number> <keeps>" 
        exit
fi
 
 
export case=$1
direc=/var/tmp/$1
keeps=$2
 
####################################
#
# Clean up the logs which is expired
# Bruce Pan on 2016-03-30
####################################
 
for i in `find /var/tmp/ -type f -amin +$keeps -print `
do
rm -rf $i
done
 
 
 
####################################
#
# Change frequency to collect data
# or go with default of 300 seconds
####################################
 
## Object: the script will be used to collect linux performance data recurrently. Disabled the interaction, replaced with parameters.
## Modified by Bruce Pan ( chpa@microsoft.com ) on 2016-03-30
 
## Start comment out
#echo ""
#echo "The data collection for vmstat and iostat is set to 5 minutes (300 seconds) with 1 second intervals"
#echo " "
#echo -en "Enter a new value to change from default of [300 seconds] "
 
#       read rep
 
#if [ -z "$rep" ]
#  then
#    rep="300"
#fi
## End comment out
rep="300"
 
if [ $rep -ne 0 -o $rep -eq 0 2>/dev/null ]
then
 
# An integer is either equal to 0 or not equal to 0.
# 2>/dev/null suppresses error message.
echo " "
    echo "will run vmstat and iostat for $rep seconds"
 
else
 
echo ""
    echo "Supplied Input $rep is not an Integer..exiting"
exit
fi
 
 
echo " "
echo " "
echo "Proceeding to data collection"
 
sleepduration=`expr $rep + $safetime`
###############
 
dd=`date +%H%M%S-%d-%h-%Y`                                
mkdir -p $direc/$sr/`hostname`_$dd
datadir=$direc/$sr/`hostname`_$dd
timest=`date +%d-%h-%Y_%H:%M:%S`
 
 
#########################
#
# Verify distro
#
#########################
 
which python  > /dev/null 2>&1
python_status=`echo $?` 
#echo $python_status
 
if [ "$python_status" -eq 0 ];  then
#    echo "python is installed"
    distro=`python -c 'import platform ; print platform.dist()[0]'`
    echo " "
    echo "This is Linux distro" $distro
else
    distro=$(awk '/DISTRIB_ID=/' /etc/*-release | sed 's/DISTRIB_ID=//' | tr '[:upper:]' '[:lower:]')
echo $distro
fi
 
 
#######################################################
#                                        
#   Verify if iostat is installed if not install it.  
#
########################################################
 
which iostat > /dev/null 2>&1
iostat_status=`echo $?`
# echo $iostat_status
if [ "$iostat_status" -eq 0 ];  then
    echo ""
    echo "iostat already installed..."
 
elif
    [ $distro = Ubuntu ];  then
    echo "Installing iostat"
    sudo apt-get -y install sysstat
 
elif
        [ $distro = centos ] || [ $distro = redhat ];  then
    echo "Installing iostat"
    yum -y install sysstat
        yum update -y sysstat
 
elif
    [ $distro = SuSE ];  then   
    echo "Installing iostat"
    sudo su -c "zypper install -y sysstat"
else
    echo "Not installing iostat for distro $distro script exiting - this script runs on centos, oracle, ubuntu and sles"
exit
fi
 
    cat /etc/*release > $datadir/vminfo
 
    echo "uname -a " >> $datadir/vminfo
    echo " " >> $datadir/vminfo
     
    echo " " >> $datadir/vminfo
    echo "uptime " >> $datadir/vminfo
        
    uptime >> $datadir/vminfo
    echo " " >> $datadir/vminfo
 
    waagent --version >> $datadir/vminfo
    echo " " >> $datadir/vminfo
     
    lsmod >> $datadir/vminfo
    echo "**************************** " >> $datadir/vminfo
    echo " " >> $datadir/vminfo
     
    modinfo hv_storvsc  >> $datadir/vminfo
    echo " " >> $datadir/vminfo
    echo "**************************** " >> $datadir/vminfo
  
    modinfo hv_netvsc >> $datadir/vminfo
    echo "**************************** " >> $datadir/vminfo
    echo " " >> $datadir/vminfo
     
    modinfo hv_utils  >> $datadir/vminfo   
    echo " " >> $datadir/vminfo
    echo "**************************** " >> $datadir/vminfo
     
    modinfo hv_vmbus >> $datadir/vminfo
    echo " " >> $datadir/vminfo
    echo "**************************** " >> $datadir/vminfo
 
#################################
#
# Collect memory info. and stats
#
#################################
 
mkdir $datadir/memory
memory="$datadir/memory/memory_$timest.txt"
vmst=$datadir/memory/vmstat_$timest.txt
 
    echo "Summary for VM `hostname` " > $memory
    echo " " >> $memory
    echo " " >> $memory
    free -m >> $memory
    echo " " >> $memory
    echo "*************************************** " >> $memory
    cat /proc/meminfo >> $memory
    echo " " >> $memory
 
    echo "Top 30 processe Resident Size memory usage" >> $memory
    ps aux --sort -rss | head -30 >> $memory
    echo "*************************************** " >> $memory
    echo " " >> $memory
     
    echo "Top 30 processes memory usage" >> $memory
    ps aux --sort -pmem | head -30  >> $memory
    echo " " >> $memory
    echo "*************************************** " >> $memory
    echo " " >> $memory
 
 
#################################
#
# Collect cpu info. and stats
#
#################################
 
mkdir $datadir/cpu
cpu="$datadir/cpu/cpu_$timest.txt"
 
    echo "Summary for VM `hostname` " > $cpu
    date >> $cpu
    echo " " >> $cpu
    echo " " >> $cpu
    echo "*************************************** " >> $cpu
    cat /proc/cpuinfo >> $cpu
    echo " " >> $cpu
    echo " " >> $cpu
     
    echo "Top cpu processes usage" >> $cpu
    ps aux --sort=-pcpu | head -n 20 >> $cpu
    echo "*************************************** " >> $cpu
    echo " " >> $cpu
 
 
     
##############################
#
# Collect disk info and stats
#
##############################
 
mkdir $datadir/disks
disks=$datadir/disks/disks_$timest.txt
 
    #df -h, df -i, fdisk -l,blkid, /etc/fstab , /etc/mdadm.conf  cat /proc/mdstat   
 
    echo "disk utilisation" >> $disks
    df -h >> $disks
    echo " " >> $disks
 
    echo "disk inode utilisation" >> $disks
    df -i >> $disks
    echo " "
     
    echo "disks attached " >> $disks
    fdisk -l >> $disks
    echo " " >> $disks
     
    echo " " >> $disks
    echo "disks blkid " >> $disks
    blkid >> $disks
    echo " " >> $disks
 
    echo "swap file configuration" >> $disks
    swapon -s >> $disks
 
    cp /etc/fstab $datadir/disks/
    cp /proc/mdstat $datadir/disks/ > /dev/null 2>&1
 
 
#####################
#                   #
# Collect vmstat and#
# iostat            #
#                   #
#####################
 
echo "collecting data..."
 
# Collect vmstat and iostat data with normal load
# Check disk values as per the official values
# https://msdn.microsoft.com/en-us/library/azure/dn197896.aspx
# If required locate no. of processors per VM - grep -i processor /proc/cpuinfo
 
    echo "vmstat data" > $vmst
    date >> $vmst
    #strftime does not work with awk on ubuntu 12
    #vmstat 1 2 | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'  >> $vmst
 
    vmstat $fre $rep  >> $vmst &
 
 
iostat_detailed=$datadir/disks/iostat_detailed.txt
iostat_summ=$datadir/disks/iostat_summ.txt
 
if [ $distro = Ubuntu ]; then
        osver=`grep -i version_id /etc/os-release`
                        if [ $osver = VERSION_ID=\"12.04\" ]; then
                        #iostatcmd1="`iostat -xd $fre $rep > $datadir/disks/iostat_detailed.txt`"
                        iostatcmd1="`iostat -xd $fre $rep > $iostat_detailed`" &
                        iostatcmd2="`iostat $fre $rep > $iostat_summ`" &
 
                elif [ $osver = VERSION_ID=\"14.04\" ] || [ $osver = VERSION_ID=\"14.10\" ] ;then
 
                        iostat -yxd $fre $rep  | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_detailed &
                        iostat -y $fre $rep  | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_summ &
                        #iostatcmd1="`iostat -yxd $fre $rep  | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_detailed`" &
                        #iostatcmd2="`iostat -y $fre $rep  | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_summ`" &
                 
    fi
 
#check suse version
elif [ $distro = SuSE ]; then
 
        osver=`grep -i version /etc/SuSE-release|awk '{print $3}'`
 
                        if [ $osver = 11 ]; then
                        iostatcmd1="`iostat -xd $fre $rep > $iostat_detailed`" &
                        iostatcmd2="`iostat $fre $rep > $iostat_summ`" &
 
                elif [ $osver = 12 ] ;then
                        iostatcmd1="`iostat -yxd $fre $rep  | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_detailed`" &
                        iostatcmd2="`iostat -y $fre $rep  | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_summ`" &
 
    fi
 
# Not checkking for Centos,Oracle or Rhel version
# if required new [ $distro = centos ] || [ $distro = redhat ];  then
 
 
else
    iostat -xd $fre $rep | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_detailed &
        iostat $fre $rep | awk '{now=strftime("%Y-%m-%d %T "); print now $0}' > $iostat_summ &
 
fi
 
#
 
###############
#             #
#    network  #
#             #
###############
 
mkdir $datadir/network
network=$datadir/network/network_$timest.txt
 
echo "Network Configuration..." >> $network
ip addr show >> $network
echo " " >> $network
 
echo "(Routing table...)" >> $network
netstat -rn >> $network
echo " " >> $network
 
ip route show >> $network
echo " " >> $network
 
echo "(DNS Configuration...)" >> $network
cat /etc/resolv.conf  >> $network
echo " " >> $network
 
echo "(Hosts file...)" >> $network
cat /etc/hosts  >> $network
echo " " >> $network
 
echo "(Network Connections...)" >> $network
netstat -tulpn >> $network
echo " " >> $network
 
echo "(Network statistics...)" >> $network
netstat -s >> $network
echo " " >> $network
########################################################
#   
# Collect only last versions of log files and sar data
#
########################################################
 
mkdir $datadir/log
mkdir -p $datadir/sar
logdir=$datadir/log
sardata=$datadir/sar
 
ls -ltr /var/log > $logdir/logfiles_listing
 
if [ $distro = Ubuntu ]; then
    logs=("wtmp" "btmp" "dmesg" "boot.log" "lastlog" "dpkg.log" "kern.log" "waagent.log" "syslog" "cloud-init.log" )
    sarlogs="/var/log/sysstat"
    #cp "${logs[@]}" $logdir > /dev/null 2>&1
     cp "${logs[@]/#//var/log/}" $logdir
    cp -R $sarlogs/. $sardata > /dev/null 2>&1
 
elif
        [ $distro = centos ] || [ $distro = redhat ]; then
    logs=("wtmp" "btmp" "dmesg" "boot.log" "lastlog" "yum.log" "messages" "waagent.log" "cron" "secure" "debug")
    sarlogs="/var/log/sa"
    #cp "${logs[@]}" $logdir > /dev/null 2>&1
     cp "${logs[@]/#//var/log/}" $logdir
    cp -R $sarlogs/. $sardata > /dev/null 2>&1
 
elif
 
        [ $distro = SuSE ]; then
    logs=("wtmp" "btmp" "boot.*" "pbl.log" "lastlog" "zypper.log" "cloudregister" "waagent.log" "messages" "warn")
    sarlogs="/var/log/sa"
    #cp "${logs[@]}" $logdir > /dev/null 2>&1
     cp "${logs[@]/#//var/log/}" $logdir
    cp -R $sarlogs/. $sardata > /dev/null 2>&1
 
fi
 
    cp /etc/waagent.conf $logdir > /dev/null 2>&1
    sysctl -a > $logdir/kernel_params.txt
     
#################################
#
#Collect the data in a tar file
#
#################################
 
#allow enough time for iostat, vmstat to complete
sleep $sleepduration
 
 
        cd $direc
        tar -zcvf $case"_"`hostname`_$dd.tar . > /dev/null 2>&1
        mv $case"_"`hostname`_$dd.tar /var/tmp
    gzip /var/tmp/$case"_"`hostname`_$dd.tar
    echo "Please send the file "/var/tmp/$case"_"`hostname`_$dd".tar.gz"
    cd /var/tmp
    rm -Rf $direc/
    

  

posted on   铭的随记  阅读(454)  评论(0编辑  收藏  举报
编辑推荐:
· .NET Core GC压缩(compact_phase)底层原理浅谈
· 现代计算机视觉入门之:什么是图片特征编码
· .NET 9 new features-C#13新的锁类型和语义
· Linux系统下SQL Server数据库镜像配置全流程详解
· 现代计算机视觉入门之:什么是视频
阅读排行:
· Sdcb Chats 技术博客:数据库 ID 选型的曲折之路 - 从 Guid 到自增 ID,再到
· .NET Core GC压缩(compact_phase)底层原理浅谈
· Winform-耗时操作导致界面渲染滞后
· Phi小模型开发教程:C#使用本地模型Phi视觉模型分析图像,实现图片分类、搜索等功能
· 语音处理 开源项目 EchoSharp
< 2025年1月 >
29 30 31 1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31 1
2 3 4 5 6 7 8

点击右上角即可分享
微信分享提示