hibench 对CDH5.13.1进行基准测试（测试项目hadoop\spark\)HDFS作HA高可靠性

使用CDH 5.13.1部署了HADOOP集群之后，需要进行基准性能测试。

一、hibench 安装

1.安装位置要求。

因为是全量安装，其中有SPARK的测试（SPARK2.0）。

安装位置在SPARK 服务所在的节点上面。

下载hibench编译好的包与manve的包

hibench全部编译

mvn -Dspark=2.1 -Dscala=2.11 clean package

注：hibench目录中运行

编译好的包，可以在整个集群通用。直接 复制环境变量与安装目录

2.配置环境变量

export JAVA_HOME=/usr/java/jdk1.8.0_151
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export M2_HOME=/home/maven
export PATH=$PATH:$M2_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin

3.权限配置

hibench 全部为777

执行 run_all.sh时为hdfs用户

/dev/stderr为777

每台机器安装 bc

4.配置文件说明

[root@cdhtest2 conf]# cat hadoop.conf
# Hadoop home
hibench.hadoop.home /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/hadoop

# The path of hadoop executable
hibench.hadoop.executable /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/bin/hadoop

# Hadoop configraution directory
hibench.hadoop.configure.dir /etc/hadoop/conf

# The root HDFS path to store HiBench data
hibench.hdfs.master hdfs://nameservice1

# Hadoop release provider. Supported value: apache, cdh5, hdp
hibench.hadoop.release cdh5

 

spark.conf

# Spark home
hibench.spark.home /opt/cloudera/parcels/SPARK2/lib/spark2

# Spark master
# standalone mode: spark://xxx:7077
# YARN mode: yarn-client
hibench.spark.master yarn-client

# executor number and cores when running on Yarn
hibench.yarn.executor.num 1
hibench.yarn.executor.cores 2

# executor and driver memory in standalone & YARN mode
spark.executor.memory 1g
spark.driver.memory 1g

 

注：

# executor number and cores when running on Yarn
hibench.yarn.executor.num 1
hibench.yarn.executor.cores 2

# executor and driver memory in standalone & YARN mode
spark.executor.memory 1g
spark.driver.memory 1g

按实际机器配置修改。

 

hibench.conf

hibench.masters.hostnames cdhtest2
hibench.slaves.hostnames cdhtest1

RM 服务的主机名

 

bin/run_all.sh

报表查看：

hibench根目录下面hibench.report

建议使用Excel 打开，分隔符是空格。