一、hadoop自带的性能基准评测工具
(一)TestDFSIO
1、测试写性能
(1)若有必要,先删除历史数据
hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar TestDFSIO -clean (2)执行测试 hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 20
(3)查看结果:每一次测试生成一个结果,并以附加的形式添加到TestDFSIO_results.log中
cat TestDFSIO_results.log ----- TestDFSIO ----- : write Date & time: Mon May 11 09:41:34 HKT 2015 Number of files: Total MBytes processed: 100.0 Throughput mb/sec: 21.468441391155004 Average IO rate mb/sec: 25.366744995117188 IO rate std deviation: 12.744636924030177 Test exec time sec: 27.585 ----- TestDFSIO ----- : write Date & time: Mon May 11 09:42:28 HKT 2015 Number of files: 5 Total MBytes processed: 100.0 Throughput mb/sec: 22.779043280182233 Average IO rate mb/sec: 25.440486907958984 IO rate std deviation: 9.930490103638768 Test exec time sec: 26.67 (4)结果说明 Total MBytes processed : 总共需要写入的数据量 100MB Throughput mb/sec :总共需要写入的数据量/(每个map任务实际写入数据的执行时间之和(这个时间会远小于Test exec time sec))==》100/(map1写时间+map2写时间+...) Average IO rate mb/sec :(每个map需要写入的数据量/每个map任务实际写入数据的执行时间)之和/任务数==》(20/map1写时间+20/map2写时间+...)/1000,所以这个值跟上面一个值总是存在差异。 IO rate std deviation :上一个值的标准差 Test exec time sec :整个job的执行时间 2、测试读性能 (1)执行测试 hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar TestDFSIO -read -nrFiles 5 -fileSize 20
(2)查看测试结果
cat TestDFSIO_results.log ----- TestDFSIO ----- : read Date & time: Mon May 11 09:53:27 HKT 2015 Number of files: 5 Total MBytes processed: 100.0 Throughput mb/sec: 534.75935828877 Average IO rate mb/sec: 540.4888916015625 IO rate std deviation: 53.93029580221512 Test exec time sec: 26.704 (3)结果说明 结果各项意思与write相同,但其读速率比写速率快很多,而总执行时间非常接近。真正测试时,应该用较大的数据量来执行,才可体现出二者的差异。 (二)排序测试 在api文档中搜索terasort,可查询相关信息。 排序测试的三个基本步骤: 生成随机数据??>排序??>验证排序结果 关于terasort更详细的原理,见http://blog.csdn.net/yuesichiu/article/details/17298563 1、生成随机数据 hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar teragen -Dmapreduce.job.maps=5 10000000 /tmp/hadoop/terasort
此步骤将在hdfs中的 /tmp/hadoop/terasort 中生成数据,
hadoop fs -ls /tmp/hadoop/terasort Found 6 items -rw-r----- 3 hadoop supergroup 0 2015-05-11 11:32 /tmp/hadoop/terasort/_SUCCESS -rw-r----- 3 hadoop supergroup 200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00000 -rw-r----- 3 hadoop supergroup 200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00001 -rw-r----- 3 hadoop supergroup 200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00002 -rw-r----- 3 hadoop supergroup 200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00003 -rw-r----- 3 hadoop supergroup 200000000 2015-05-11 11:32 /tmp/hadoop/terasort/part-m-00004 hadoop fs -du -s -h /tmp/hadoop/terasort
953.7 M /tmp/hadoop/terasort
生成的5个数据竟然是每个200M,未解,为什么不是10M???
2、运行测试
hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar terasort -Dmapreduce.job.maps=5 /tmp/hadoop/terasort /tmp/hadoop/terasort_out Spent 354ms computing base-splits. Spent 8ms computing TeraScheduler splits. Computing input splits took 365ms Sampling 10 splits of 10 Making 1 from 100000 sampled records Computing parititions took 6659ms Spent 7034ms computing partitions. 3、验证结果 hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar teravalidate /tmp/hadoop/terasort_out /tmp/hadoop/terasort_report
Spent 44ms computing base-splits.
Spent 7ms computing TeraScheduler splits.
二、hibench
hibench4.0测试不成功,使用3.0代替
1、下载并解压
wget https://codeload.github.com/intel-hadoop/HiBench/zip/HiBench-3.0.0
unzip HiBench-3.0.0
2、修改文件 bin/hibench-config.sh,主要是这几个
export JAVA_HOME=/home/hadoop/jdk1.7.0_67
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_EXECUTABLE=/home/hadoop/hadoop//bin/hadoop
export HADOOP_CONF_DIR=/home/hadoop/conf
export HADOOP_EXAMPLES_JAR=/home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar
export MAPRED_EXECUTABLE=/home/hadoop/hadoop/bin/mapred
#Set the varaible below only in YARN mode
export HADOOP_JOBCLIENT_TESTS_JAR=/home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.3.0-cdh5.1.2.jar/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar
3、修改conf/benchmarks.lst,哪些不想运行的将之注释掉
4、运行
bin/run-all.sh
5、查看结果
在当前目录会生成hibench.report文件,内容如下
Type Date Time Input_data_size Duration(s) Throughput(bytes/s) Throughput/node
WORDCOUNT 2015-05-12 19:32:33 251.248
DFSIOE-READ 2015-05-12 19:54:29 54004092852 463.863 116422505 38807501
DFSIOE-WRITE 2015-05-12 20:02:57 27320849148 498.132 54846605 18282201
PAGERANK 2015-05-12 20:27:25 711.391
SORT 2015-05-12 20:33:21 243.603
TERASORT 2015-05-12 20:40:34 10000000000 266.796 37481821 12493940
SLEEP 2015-05-12 20:40:40 0 .177 0 0
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步