Processing math: 0%

一、hadoop自带的性能基准评测工具 


(一)TestDFSIO 

1、测试写性能 
(1)若有必要,先删除历史数据 
hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar TestDFSIO -clean  (2)执行测试  hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 20 
(3)查看结果:每一次测试生成一个结果,并以附加的形式添加到TestDFSIO_results.log中 
cat TestDFSIO_results.log  ----- TestDFSIO ----- : write             Date & time: Mon May 11 09:41:34 HKT 2015         Number of files:  Total MBytes processed: 100.0       Throughput mb/sec: 21.468441391155004  Average IO rate mb/sec: 25.366744995117188   IO rate std deviation: 12.744636924030177      Test exec time sec: 27.585  ----- TestDFSIO ----- : write             Date & time: Mon May 11 09:42:28 HKT 2015         Number of files: 5  Total MBytes processed: 100.0       Throughput mb/sec: 22.779043280182233  Average IO rate mb/sec: 25.440486907958984   IO rate std deviation: 9.930490103638768      Test exec time sec: 26.67  (4)结果说明  Total MBytes processed : 总共需要写入的数据量 100MB  Throughput mb/sec :总共需要写入的数据量/(每个map任务实际写入数据的执行时间之和(这个时间会远小于Test exec time sec))==》100/(map1写时间+map2写时间+...)  Average IO rate mb/sec :(每个map需要写入的数据量/每个map任务实际写入数据的执行时间)之和/任务数==》(20/map1写时间+20/map2写时间+...)/1000,所以这个值跟上面一个值总是存在差异。  IO rate std deviation :上一个值的标准差  Test exec time sec :整个job的执行时间  2、测试读性能  (1)执行测试  hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar TestDFSIO -read -nrFiles 5 -fileSize 20 
(2)查看测试结果 
cat TestDFSIO_results.log  ----- TestDFSIO ----- : read             Date & time: Mon May 11 09:53:27 HKT 2015         Number of files: 5  Total MBytes processed: 100.0       Throughput mb/sec: 534.75935828877  Average IO rate mb/sec: 540.4888916015625   IO rate std deviation: 53.93029580221512      Test exec time sec: 26.704  (3)结果说明  结果各项意思与write相同,但其读速率比写速率快很多,而总执行时间非常接近。真正测试时,应该用较大的数据量来执行,才可体现出二者的差异。  (二)排序测试  在api文档中搜索terasort,可查询相关信息。  排序测试的三个基本步骤:  生成随机数据??>排序??>验证排序结果  关于terasort更详细的原理,见http://blog.csdn.net/yuesichiu/article/details/17298563  1、生成随机数据  hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar  teragen -Dmapreduce.job.maps=5 10000000 /tmp/hadoop/terasort 
此步骤将在hdfs中的 /tmp/hadoop/terasort  中生成数据, 
  hadoop fs -ls /tmp/hadoop/terasort  Found 6 items  -rw-r-----   3 hadoop supergroup          0 2015-05-11 11:32 /tmp/hadoop/terasort/_SUCCESS  -rw-r-----   3 hadoop supergroup  200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00000  -rw-r-----   3 hadoop supergroup  200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00001  -rw-r-----   3 hadoop supergroup  200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00002  -rw-r-----   3 hadoop supergroup  200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00003  -rw-r-----   3 hadoop supergroup  200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00004  hadoop fs -du -s -h /tmp/hadoop/terasort 
953.7 M  /tmp/hadoop/terasort 
生成的5个数据竟然是每个200M,未解,为什么不是10M??? 

2、运行测试 
hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar  terasort -Dmapreduce.job.maps=5 /tmp/hadoop/terasort /tmp/hadoop/terasort_out  Spent 354ms computing base-splits.  Spent 8ms computing TeraScheduler splits.  Computing input splits took 365ms  Sampling 10 splits of 10  Making 1 from 100000 sampled records  Computing parititions took 6659ms  Spent 7034ms computing partitions.  3、验证结果    hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar teravalidate  /tmp/hadoop/terasort_out /tmp/hadoop/terasort_report 

Spent 44ms computing base-splits. 

Spent 7ms computing TeraScheduler splits. 




二、hibench 
hibench4.0测试不成功,使用3.0代替 

1、下载并解压 

wget https://codeload.github.com/intel-hadoop/HiBench/zip/HiBench-3.0.0 

unzip HiBench-3.0.0 

2、修改文件  bin/hibench-config.sh,主要是这几个 

export JAVA_HOME=/home/hadoop/jdk1.7.0_67 

export HADOOP_HOME=/home/hadoop/hadoop 

export HADOOP_EXECUTABLE=/home/hadoop/hadoop//bin/hadoop 

export HADOOP_CONF_DIR=/home/hadoop/conf 

export HADOOP_EXAMPLES_JAR=/home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar 

export MAPRED_EXECUTABLE=/home/hadoop/hadoop/bin/mapred 

#Set the varaible below only in YARN mode 

export HADOOP_JOBCLIENT_TESTS_JAR=/home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar 



3、修改conf/benchmarks.lst,哪些不想运行的将之注释掉 

4、运行 

bin/run-all.sh 

5、查看结果 

在当前目录会生成hibench.report文件,内容如下 

Type         Date       Time     Input_data_size      Duration(s)          Throughput(bytes/s)  Throughput/node 

WORDCOUNT    2015-05-12 19:32:33 251.248 

DFSIOE-READ  2015-05-12 19:54:29 54004092852          463.863              116422505            38807501 

DFSIOE-WRITE 2015-05-12 20:02:57 27320849148          498.132              54846605             18282201 

PAGERANK     2015-05-12 20:27:25 711.391 

SORT         2015-05-12 20:33:21 243.603 

TERASORT     2015-05-12 20:40:34 10000000000          266.796              37481821             12493940 

SLEEP        2015-05-12 20:40:40 0                    .177                 0                    0 
posted on   热血的青春  阅读(2823)  评论(0编辑  收藏  举报
努力加载评论中...

点击右上角即可分享
微信分享提示