nGrinder对监控机器收集自定义数据及源码分析
转载：https://blog.csdn.net/neven7/article/details/50782451
0.背景
性能测试工具nGrinder支持在无需修改源码的情况下，对目标服务器收集自定义数据，最多支持5类；
在性能测试详细报告页，目标服务器->你的机器ip便签页下，默认只收集CPU, Memory, Received Byte/s, Sent Byte Per Secode/s等4类数据；
可能你还需要监控其它的性能统计数据，用于分析(比如load, Full Gc)；本文先介绍实现方法；再分析nGrinder源码，看它是怎么实现的。
1.实现
1-1. 安装monitor
在你的nGrinder系统下，下载监控
安装到你测试服务所在的机器，解压tar包，执行sh run_monitor_bg.sh;
其实脚本是启了个java服务，以monitor模式启动；
之前介绍过Agent有2种模式：
gent mode: 运行进程和线程，压测目标服务；
monitor mode: 监控目标系统性能(cpu/memory)。
[root@10 ngrinder-monitor]# cat run_monitor.sh
#!/bin/sh
curpath=`dirname $0`
cd ${curpath}
java -server -cp "lib/*" org.ngrinder.NGrinderAgentStarter --mode monitor --command run $@
- 1
- 2
- 3
- 4
- 5
- 6
Agent的home路径为/root/.ngrinder_agent，你在执行sh run_monitor_bg.sh默认获取的配置信息为/root/.ngrinder_agent/agent.conf； 如果加上-o，sh run_monitor_bg.sh 读取你安装monitor目录下的__agent.conf， 该配置文件定义了Agent的模式，ip, 端口。
[root@10 .ngrinder_agent]# cat agent.conf
common.start_mode=monitor
#If you want to monitor bind to the different local ip not automatically selected ip. Specify below field.
#monitor.binding_port=hostname_or_ip
monitor.binding_port=13243
- 1
- 2
- 3
- 4
- 5
- 6
自定义数据需放在/root/.ngrinder_agent/monitor/custom.data文件里，格式如下：
类型1数据,类型2数据,类型3数据,类型4数据,类型5数据- 1
最多支持5类，每类数据用“,”分隔，注意的是： 数据是实时的写文件，不是累积数据到文件中(类似shell中的>, 不是>>)，即同一时刻，只有一行数据。
1-2. 定制收集脚本
以收集load和full GC为例：
[root@10 bin]# cat updateCustomData.sh
#!/bin/sh
#@author hugang
customDataRoot=/root/.ngrinder_agent/monitor/custom.data;
# 获取load信息
load=`/bin/cat /proc/loadavg | awk '{print $1}'`;
# 获取full gc count
if [[ $1 -gt 0 ]]; then
fgc=`jstat -gcutil $1 | tail -1 | awk '{print $8}'`;
echo $load,$fgc > $customDataRoot;
else
echo $load > $customDataRoot;
fi;
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
开始性能测试时，每秒去执行该脚本，收集数据到custom.data中：
watch -n 1 sh updateCustomData.sh 5528- 1
5528为需监控java服务进程pid;
当你性能测试结束后，monitor收集的数据会放到/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{test_id}/report/monitor_system_{ip}.data文件中:
[root@10 report]# cat monitor_system_10.13.1.139.data
ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec,customValues
10.13.1.139,LINUX,20160302151441,97102768,132112072,26.895683,32954,27897,4.93,49
10.13.1.139,LINUX,20160302151443,97075896,132112072,30.513468,45702,32306,4.93,49
10.13.1.139,LINUX,20160302151445,97034772,132112072,30.411074,110306,65391,5.02,49
10.13.1.139,LINUX,20160302151447,96972504,132112072,22.073017,84813,57503,5.02,49
...
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
1-3.结果展示
2.源码分析
nGrinder使用Sigar工具（https://support.hyperic.com/display/SIGAR/Home）收集系统信息，该工具可以收集以下数据：
System memory, swap, cpu, load average, uptime, logins
Per-process memory, cpu, credential info, state, arguments, environment, open files
File system detection and metrics
Network interface detection, configuration info and metrics
TCP and UDP connection tables
Network route table
- 1
- 2
- 3
- 4
- 5
- 6
- 7
sigar工具（http://download.csdn.net/download/neven7/9450930）示例：
[root@10 testsigar]# ls
libsigar-amd64-linux.so sigar-1.6.4.jar sigar-1.6.4.jar.zip
[root@10 testsigar]#
[root@10 testsigar]# java -jar ./sigar-1.6.4.jar
sigar> free
total used free
Mem: 132112072 96855372 35256700
-/+ buffers/cache: 34855500 97256572
Swap: 8388600 264980 8123620
RAM: 129016MB
sigar> - 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
收集系统数据的java文件为：
ngrinder-core/src/main/java/org/ngrinder/monitor/collector/SystemDataCollector.java
继承和实现关系：
SystemDataCollector extends DataCollector
DataCollector implements Runnable
- 1
- 2
- 3
- 4
SystemDataCollector的线程执行体：
public void run() {
// 初始化sigar
initSigar();
SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);
// execute()通过sigar api获取系统信息
systemMonitoringData.setSystemInfo(execute());
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
execute()获取系统信息SystemInfo(System info object to save date collected by monitor)：
/**
* Execute the collector to get the system info model.
*
* @return SystemInfo in current time
*/
public synchronized SystemInfo execute() {
SystemInfo systemInfo = new SystemInfo();
systemInfo.setCollectTime(System.currentTimeMillis());
try {
BandWidth networkUsage = getNetworkUsage();
BandWidth bandWidth = networkUsage.adjust(prev.getBandWidth());
systemInfo.setBandWidth(bandWidth);
systemInfo.setCPUUsedPercentage((float) sigar.getCpuPerc().getCombined() * 100);
Cpu cpu = sigar.getCpu();
systemInfo.setTotalCpuValue(cpu.getTotal());
systemInfo.setIdleCpuValue(cpu.getIdle());
Mem mem = sigar.getMem();
systemInfo.setTotalMemory(mem.getTotal() / 1024L);
systemInfo.setFreeMemory(mem.getActualFree() / 1024L);
systemInfo.setSystem(OperatingSystem.IS_WIN32 ? SystemInfo.System.WINDOW : SystemInfo.System.LINUX);
systemInfo.setCustomValues(getCustomMonitorData());
} catch (Throwable e) {
LOGGER.error("Error while getting system perf data:{}", e.getMessage());
LOGGER.debug("Error trace is ", e);
}
prev = systemInfo;
return systemInfo;
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
其中：getCustomMonitorData()获取自定义数据，读取custom.data文件中一行数据
private String getCustomMonitorData() {
if (customDataFile != null && customDataFile.exists()) {
BufferedReader customDataFileReader = null;
try {
customDataFileReader = new BufferedReader(new FileReader(customDataFile));
return customDataFileReader.readLine(); // these data will be parsed at
// monitor client side.
} catch (IOException e) {
// Error here is very natural
LOGGER.debug("Error to read custom monitor data", e);
} finally {
IOUtils.closeQuietly(customDataFileReader);
}
}
return prev.getCustomValues();
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
综上：类SystemDataCollector作用就是作为线程执行体，线程每次执行通过sigar获取系统信息：SystemInfo，赋值给SystemMonitoringData成员变量SystemInfo。
前面介绍启动monitor时，其实是执行了org.ngrinder.NGrinderAgentStarter类，我们再分析下该文件，ngrinder-core/src/main/java/org/ngrinder/NGrinderAgentStarter.java
/**
* Agent starter.
*
* @param args arguments
*/
public static void main(String[] args) {
NGrinderAgentStarter starter = new NGrinderAgentStarter();
final NGrinderAgentStarterParam param = new NGrinderAgentStarterParam();
checkJavaVersion();
JCommander commander = new JCommander(param);
commander.setProgramName("ngrinder-agent");
commander.setAcceptUnknownOptions(true);
try {
commander.parse(args);
} catch (Exception e) {
LOG.error(e.getMessage());
return;
}
final List<String> unknownOptions = commander.getUnknownOptions();
modeParam = param.getModeParam();
modeParam.parse(unknownOptions.toArray(new String[unknownOptions.size()]));
if (modeParam.version != null) {
System.out.println("nGrinder v" + getStaticVersion());
return;
}
if (modeParam.help != null) {
modeParam.usage();
return;
}
System.getProperties().putAll(modeParam.params);
starter.init();
final String startMode = modeParam.name();
if ("stop".equalsIgnoreCase(param.command)) {
starter.stopProcess(startMode);
System.out.println("Stop the " + startMode);
return;
}
starter.checkDuplicatedRun(startMode);
if (startMode.equalsIgnoreCase("agent")) {
starter.startAgent();
} else if (startMode.equalsIgnoreCase("monitor")) {
starter.startMonitor();
} else {
staticPrintHelpAndExit("Invalid agent.conf, '--mode' must be set as 'monitor' or 'agent'.");
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
monitor模式执行该方法：starter.startMonitor()
/**
* Start the performance monitor.
*/
public void startMonitor() {
printLog("***************************************************");
printLog("* Start nGrinder Monitor... ");
printLog("***************************************************");
try {
MonitorServer.getInstance().init(agentConfig);
MonitorServer.getInstance().start();
} catch (Exception e) {
LOG.error("ERROR: {}", e.getMessage());
printHelpAndExit("Error while starting Monitor", e);
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
MonitorServer.getInstance().start():
/**
* Start monitoring.
*
* @throws IOException exception
*/
public void start() throws IOException {
if (!isRunning()) {
jmxServer.start();
DataCollectManager.getInstance().init(agentConfig);
DataCollectManager.getInstance().start();
isRunning = true;
}
}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
DataCollectManager.getInstance().start();
/**
* start a scheduler for the data collector jobs.
*/
public void start() {
int collectorCount = MXBeanStorage.getInstance().getSize();
scheduler = Executors.newScheduledThreadPool(collectorCount);
if (!isRunning()) {
Collection<MXBean> mxBeans = MXBeanStorage.getInstance().getMXBeans();
for (MXBean mxBean : mxBeans) {
DataCollector collector = mxBean.gainDataCollector(agentConfig.getHome().getDirectory());
scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);
LOG.info("{} started.", collector.getClass().getSimpleName());
}
LOG.info("Collection interval : {}s).", getInterval());
isRunning = true;
}
}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);
线程池周期地执行SystemDataCollector中run()去获取系统数据。
@Override
public void run() {
initSigar();
SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);
systemMonitoringData.setSystemInfo(execute());
}- 1
- 2
- 3
- 4
- 5
- 6
3.总结：
后台启动的monitor， 运行的是一个java服务:
java -server -cp lib/* org.ngrinder.NGrinderAgentStarter --mode monitor --command run- 1
通过线程池周期获取系统性信息(sigar工具获取)，存放在SystemInfo;
ngrinder-controller/src/main/java/org/ngrinder/perftest/service/samplinglistener/MonitorCollectorPlugin.java中startSampling():
@Override
public void startSampling(final ISingleConsole singleConsole, PerfTest perfTest,
IPerfTestService perfTestService) {
final List<String> targetHostIP = perfTest.getTargetHostIP();
final Integer samplingInterval = perfTest.getSamplingInterval();
for (final String target : targetHostIP) {
scheduledTaskService.runAsync(new Runnable() {
@Override
public void run() {
LOGGER.info("Start JVM monitoring for IP:{}", target);
MonitorClientService client = new MonitorClientService(target, MonitorCollectorPlugin.this.port);
client.init();
if (client.isConnected()) {
File testReportDir = singleConsole.getReportPath();
File dataFile = null;
try {
dataFile = new File(testReportDir, MONITOR_FILE_PREFIX + target + ".data");
FileWriter fileWriter = new FileWriter(dataFile, false);
BufferedWriter bw = new BufferedWriter(fileWriter);
// write header info
bw.write(SystemInfo.HEADER);
bw.newLine();
bw.flush();
clientMap.put(client, bw);
} catch (IOException e) {
LOGGER.error("Error to write to file:{}, Error:{}", dataFile.getPath(), e.getMessage());
}
}
}
});
}
assignScheduledTask(samplingInterval);
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
根据SystemInfo写到/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{test_id}/report/monitor_system_{ip}.data文件中；
ngrinder-controller/src/main/java/org/ngrinder/perftest/PerfTestService.java中getMonitorGraph()根据/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{test_id}/report/monitor_system_{ip}.data获取系统信息数据
/**
* Get system monitor data and wrap the data as a string value like "[22,11,12,34,....]", which can be used directly
* in JS as a vector.
*
* @param testId test id
* @param targetIP ip address of the monitor target
* @param dataInterval interval value to get data. Interval value "2" means, get one record for every "2" records.
* @return return the data in map
*/
public Map<String, String> getMonitorGraph(long testId, String targetIP, int dataInterval) {
Map<String, String> returnMap = Maps.newHashMap();
File monitorDataFile = new File(config.getHome().getPerfTestReportDirectory(String.valueOf(testId)),
MONITOR_FILE_PREFIX + targetIP + ".data");
BufferedReader br = null;
try {
StringBuilder sbUsedMem = new StringBuilder("[");
StringBuilder sbCPUUsed = new StringBuilder("[");
StringBuilder sbNetReceived = new StringBuilder("[");
StringBuilder sbNetSent = new StringBuilder("[");
StringBuilder customData1 = new StringBuilder("[");
StringBuilder customData2 = new StringBuilder("[");
StringBuilder customData3 = new StringBuilder("[");
StringBuilder customData4 = new StringBuilder("[");
StringBuilder customData5 = new StringBuilder("[");
br = new BufferedReader(new FileReader(monitorDataFile));
br.readLine(); // skip the header.
// "ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec"
String line = br.readLine();
int skipCount = dataInterval;
// to be compatible with previous version, check the length before
// adding
while (StringUtils.isNotBlank(line)) {
if (skipCount < dataInterval) {
skipCount++;
} else {
skipCount = 1;
String[] datalist = StringUtils.split(line, ",");
if ("null".equals(datalist[4]) || "undefined".equals(datalist[4])) {
sbUsedMem.append("null").append(",");
} else {
sbUsedMem.append(Long.valueOf(datalist[4]) - Long.valueOf(datalist[3])).append(",");
}
addCustomData(sbCPUUsed, 5, datalist);
addCustomData(sbNetReceived, 6, datalist);
addCustomData(sbNetSent, 7, datalist);
addCustomData(customData1, 8, datalist);
addCustomData(customData2, 9, datalist);
addCustomData(customData3, 10, datalist);
addCustomData(customData4, 11, datalist);
addCustomData(customData5, 12, datalist);
line = br.readLine();
}
}
completeCustomData(returnMap, "cpu", sbCPUUsed);
completeCustomData(returnMap, "memory", sbUsedMem);
completeCustomData(returnMap, "received", sbNetReceived);
completeCustomData(returnMap, "sent", sbNetSent);
completeCustomData(returnMap, "customData1", customData1);
completeCustomData(returnMap, "customData2", customData2);
completeCustomData(returnMap, "customData3", customData3);
completeCustomData(returnMap, "customData4", customData4);
completeCustomData(returnMap, "customData5", customData5);
} catch (IOException e) {
LOGGER.info("Error while getting monitor {} data file at {}", targetIP, monitorDataFile);
} finally {
IOUtils.closeQuietly(br);
}
return returnMap;
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
数据提供给Controller端：
ngrinder-controller/src/man/java/org/ngrinder/perftest/controller/PerfTestController.java
private Map<String, String> getMonitorGraphData(long id, String targetIP, int imgWidth) {
int interval = perfTestService.getMonitorGraphInterval(id, targetIP, imgWidth);
Map<String, String> sysMonitorMap = perfTestService.getMonitorGraph(id, targetIP, interval);
PerfTest perfTest = perfTestService.getOne(id);
sysMonitorMap.put("interval", String.valueOf(interval * (perfTest != null ? perfTest.getSamplingInterval() : 1)));
return sysMonitorMap;
}
/**
* Get the monitor data of the target having the given IP.
*
* @param id test Id
* @param targetIP targetIP
* @param imgWidth image width
* @return json message
*/
@RestAPI
@RequestMapping("/api/{id}/monitor")
public HttpEntity<String> getMonitorGraph(@PathVariable("id") long id,
@RequestParam("targetIP") String targetIP, @RequestParam int imgWidth) {
return toJsonHttpEntity(getMonitorGraphData(id, targetIP, imgWidth));
}