hadoop2.7.7+habse2.0.5+zookeeper3.4.14+hive2.3.5单机安装
环境 腾讯云centos7
1、hadoop下载
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz
2、解压
tar -xvf hadoop-2.7.7.tar.gz -C /usr/java
3、修改hadoop-2.7.7/etc/hadoop/hadoop-env.sh文件
将jdk环境添加进去： # The java implementation to use. export JAVA_HOME=/usr/java/jdk1.8
4、添加hadoop环境变量
HADOOP_HOME=/usr/java/hadoop-2.7.7 MAVEN_HOME=/usr/java/maven3.6 RABBITMQ_HOME=/usr/java/rabbitmq_server TOMCAT_HOME=/usr/java/tomcat8.5 JAVA_HOME=/usr/java/jdk1.8 CLASSPATH=$JAVA_HOME/lib/ PATH=$PATH:$JAVA_HOME/bin:$TOMCAT_HOME/bin:$RABBITMQ_HOME/sbin:$MAVEN_HOME/bin:$HADOOP_HOME/bin export PATH JAVA_HOME CLASSPATH TOMCAT_HOME RABBITMQ_HOME MAVEN_HOME HADOOP_HOME
环境变量生效：source /etc/profile
5、修改hadoop-2.7.7/etc/hadoop/core-site.xml
<!-- 指定HDFS老大（namenode）的通信地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<!-- 指定hadoop运行时产生文件的存储路径 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/java/hadoop-2.7.7/tmp</value>
</property>
6、修改hadoop-2.7.7/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/usr/java/hadoop-2.7.7/hdfs/name</value>
<description>namenode上存储hdfs名字空间元数据 </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/java/hadoop-2.7.7/hdfs/data</value>
<description>datanode上数据块的物理存储位置</description>
</property>
<!-- 设置hdfs副本数量 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
7、免密登陆
ssh-keygen -t rsa cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
8、hdfs启动与停止
./bin/hdfs namenode -format #初始化，必须对namenode进行格式化 出现：19/08/13 09:46:05 INFO common.Storage: Storage directory /usr/java/hadoop-2.7.7/hdfs/name has been successfully formatted。说明格式化成功！ ./sbin/start-dfs.sh #启动hadoop (base) [root@medecineit hadoop-2.7.7]# ./sbin/start-dfs.sh Starting namenodes on [localhost] The authenticity of host 'localhost (127.0.0.1)' can't be established. ECDSA key fingerprint is SHA256:SLOXW/SMogWE3wmK/H310vL74h0dsYohaSF31oEsdBw. ECDSA key fingerprint is MD5:fe:a4:15:38:15:e7:32:c3:9f:c3:8e:43:c6:80:6b:ac. Are you sure you want to continue connecting (yes/no)? yes localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. localhost: starting namenode, logging to /usr/java/hadoop-2.7.7/logs/hadoop-root-namenode-medecineit.out localhost: starting datanode, logging to /usr/java/hadoop-2.7.7/logs/hadoop-root-datanode-medecineit.out Starting secondary namenodes [0.0.0.0] The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established. ECDSA key fingerprint is SHA256:SLOXW/SMogWE3wmK/H310vL74h0dsYohaSF31oEsdBw. ECDSA key fingerprint is MD5:fe:a4:15:38:15:e7:32:c3:9f:c3:8e:43:c6:80:6b:ac. Are you sure you want to continue connecting (yes/no)? yes 0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts. 0.0.0.0: starting secondarynamenode, logging to /usr/java/hadoop-2.7.7/logs/hadoop-root-secondarynamenode-medecineit.out ./sbin/stop-dfs.sh #停止hadoop
9、查看是否启动相应的节点
jps命令查看 (base) [root@medecineit hadoop-2.7.7]# jps 4416 NameNode 4916 Jps 4740 SecondaryNameNode 4553 DataNode 975 Bootstrap 说明NameNode，SecondaryNameNode，DataNode启动成功。
10、web界面查看
http://ip:50070
11、配置yarn -->mapred-site.xml
复制一份文件：cp mapred-site.xml.template mapred-site.xml
<!-- 通知框架MR使用YARN -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
12、配置yarn-site.xml文件
<!-- reducer取数据的方式是mapreduce_shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
13、启动/停止yarn
./sbin/start-yarn.sh #启动 (base) [root@medecineit hadoop-2.7.7]# ./sbin/start-yarn.sh starting yarn daemons starting resourcemanager, logging to /usr/java/hadoop-2.7.7/logs/yarn-root-resourcemanager-medecineit.out localhost: starting nodemanager, logging to /usr/java/hadoop-2.7.7/logs/yarn-root-nodemanager-medecineit.out (base) [root@medecineit hadoop-2.7.7]# jps 8469 ResourceManager 8585 NodeManager 8812 Jps 975 Bootstrap 然后再启动hdfs : ./sbin/start-dfs.sh (base) [root@medecineit hadoop-2.7.7]# jps 8469 ResourceManager 9208 DataNode 9401 SecondaryNameNode 9065 NameNode 8585 NodeManager 9550 Jps 975 Bootstrap ./sbin/stop-yarn.sh #停止
14、web界面查看yarn
http://ip:8088
单机hadoop和yarn的配置完毕！
########zookeeper安装###########
1、下载地址
https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
2、解压
tar -xvf zookeeper-3.4.14.tar.gz -C /usr/java/
3、修改配置文件
cp zoo_sample.cfg zoo.cfg
将数据保存到zookeeper的data目录中
dataDir=/usr/java/zookeeper-3.4.14/data
4、启动zookeeper
./bin/zkServer.sh start #启动 ./bin/zkServer.sh status #查看状态
zookeeper完毕！
#######hbase安装##########
1、下载地址
https://www.apache.org/dyn/closer.lua/hbase/2.0.5/hbase-2.0.5-bin.tar.gz
2、解压
tar -xvf hbase-2.0.5-bin.tar.gz -C /usr/java/
3、修改hbase-env.sh
export JAVA_HOME=/usr/java/jdk1.8/
4、修改hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://medecineit:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>medecineit</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hbase.master.dns.nameserver</name>
<value>medecineit</value>
<description>DNS</description>
</property>
<property>
<name>hbase.regionserver.dns.nameserver</name>
<value>medecineit</value>
<description>DNS</description>
</property>
<property>
<name>hbase.security.authentication</name>
<value>simple</value>
</property>
<property>
<name>hbase.security.authorization</name>
<value>false</value>
</property>
<property>
<name>hbase.regionserver.hostname</name>
<value>medecineit</value>
</property>
</configuration>
##注意，红色的部分一定要加，否则远程连接hbase报错！
5、修改 regionservers
改为主机名：medecineit
6、启动hbase
./bin/start-hbase.sh #启动
(base) [root@medecineit hbase-2.0.5]# jps 8469 ResourceManager 16902 Jps 16823 HRegionServer 9208 DataNode 16152 QuorumPeerMain 9401 SecondaryNameNode 9065 NameNode 16681 HMaster 8585 NodeManager 975 Bootstrap 表明已经启动了HRegionServer，HMaster。
7、web访问
http://ip:16010/master-status
8、启动hbase shell进行表的操作
./bin/hbase shell #启动hbase shell
完毕！
#####关闭顺序####
停止集群服务的顺序 停止spark集群 master>spark/sbin/stop-slaves.sh master>spark/sbin/stop-master.sh 停止hbase集群 master>stop-hbase.sh 停止yarn集群 master>stop-yarn.sh 停止hadoop集群 master>stop-dfs.sh 停止zookeeper集群 master>runRemoteCmd.sh “zkServer.sh stop” zookeeper 停止集群服务完毕！
#####hive安装######
1、下载安装包
https://www-eu.apache.org/dist/hive/hive-2.3.5/apache-hive-2.3.5-bin.tar.gz
2、解压
tar -xzvf apache-hive-2.3.5-bin.tar.gz
3、配置hive-env.sh
export HADOOP_HOME=/usr/java/hadoop-2.7.7 export HIVE_CONF_DIR=/usr/java/hive-2.3.5/conf export HIVE_AUX_JARS_PATH=/usr/java/hive-2.3.5/lib
4、配置vim hive-site.xml文件
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://medecineit:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>yang156122</value>
<description>password to use against metastore database</description>
</property>
</configuration>
5、添加配置文件
cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties
cp hive-log4j2.properties.template hive-log4j2.properties
6、启动hive
./schematool -dbType mysql -initSchema #元数据库初始化
./hive --service hiveserver2 #启动 ./beeline -u jdbc:hive2://localhost:10000 #测试 -beeline工具测试使用jdbc方式连接 http://ip:10002/ #web界面
#####spark安装######
1、下载安装包
https://archive.apache.org/dist/spark/spark-2.1.2/spark-2.1.2-bin-hadoop2.7.tgz
2、解压
tar -xf spark-2.1.2-bin-hadoop2.7.tgz
3、修改spark-env.sh文件
cp spark-env.sh.template spark-env.sh 添加： export HADOOP_CONF_DIR=/usr/local/java/hadoop-2.7.7/etc/hadoop export JAVA_HOME=/usr/local/java/jdk1.8.0_212 export SPARK_MASTER_IP=192.168.157.100 export SPARK_WORKER_MEMORY=2g export SPARK_WORKER_CORES=2 export SPARK_WORKER_INSTANCES=1
4、配置环境变量
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
5、启动master和work
sh start-master.sh sh start-slave.sh spark://192.168.157.100:7077
6、提交
# local sh spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://hadoop100:7077 \ --executor-memory 1G \ --total-executor-cores 2 \ /usr/local/java/spark-2.1.2-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.2.jar #yarn模式 spark-submit --master yarn --executor-memory 2G --executor-cores 1 --class org.apache.spark.examples.SparkPi /usr/local/java/spark-2.1.2-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.2.jar
完毕！
本文来自博客园，作者：小白啊小白，Fighting，转载请注明原文链接：https://www.cnblogs.com/ywjfx/p/11344345.html
