如需大数据开发整套视频(hadoop\hive\hbase\flume\sqoop\kafka\zookeeper\presto\spark):请联系QQ:1974983704
由于Hive需要Yarn平台的支持,所以先配置Yarn
配置Yarn
- 配置etc/hadoop/core-site.xml
1 <configuration> 2 <property> 3 <name>fs.defaultFS</name> 4 <value>hdfs://master:9000</value> 5 </property> 6 <property> 7 <name>hadoop.tmp.dir</name> 8 <value>/home/zhangc/application/hadoopdata</value> 9 </property> 10 <property> 11 <name>hadoop.proxyuser.zhangc.hosts</name> 12 <value>*</value> 13 </property> 14 <property> 15 <name>hadoop.proxyuser.zhangc.groups</name> 16 <value>*</value> 17 </property> 18 </configuration>
- 配置etc/hadoop/yarn-site.xml
1 <configuration> 2 3 <!-- Site specific YARN configuration properties --> 4 <property> 5 <name>yarn.nodemanager.aux-services</name> 6 <value>mapreduce_shuffle</value> 7 </property> 8 <property> 9 <name>yarn.resourcemanager.address</name> 10 <value>master:18040</value> 11 </property> 12 <property> 13 <name>yarn.resourcemanager.scheduler.address</name> 14 <value>master:18030</value> 15 </property> 16 <property> 17 <name>yarn.resourcemanager.resource-tracker.address</name> 18 <value>master:18025</value> 19 </property> 20 <property> 21 <name>yarn.resourcemanager.admin.address</name> 22 <value>master:18141</value> 23 </property> 24 <property> 25 <name>yarn.resourcemanager.webapp.address</name> 26 <value>master:18088</value> 27 </property> 28 <property> 29 <name>yarn.log.server.url</name> 30 <value>http://master:19888/jobhistory/logs</value> 31 </property> 32 <property> 33 <name>yarn.nodemanager.vmem-check-enabled</name> 34 <value>false</value> 35 </property> 36 <property> 37 <name>yarn.resourcemanager.scheduler.class</name> 38 <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> 39 </property> 40 <property> 41 <name>yarn.log-aggregation-enable</name> 42 <value>true</value> 43 </property> 44 <property> 45 <name>yarn.nodemanager.pmem-check-enabled</name> 46 <value>false</value> 47 </property> 48 <property> 49 <name>yarn.nodemanager.vmem-check-enabled</name> 50 <value>false</value> 51 </property> 52 53 </configuration>
- 配置etc/hadoop/mapred-site.xml
1 <configuration> 2 <property> 3 <name>mapreduce.framework.name</name> 4 <value>yarn</value> 5 </property> 6 <property> 7 <name>mapreduce.jobhistory.done-dir</name> 8 <value>/user/history/done</value> 9 </property> 10 <property> 11 <name>mapreduce.jobhistory.intermediate-done-dir</name> 12 <value>/user/history/done_intermediate</value> 13 </property> 14 <property> 15 <name>mapreduce.jobhistory.address</name> 16 <value>master:10020</value> 17 </property> 18 <property> 19 <name>mapreduce.jobhistory.webapp.address</name> 20 <value>master:19888</value> 21 </property> 22 </configuration>
- 启动Yarn,需要提前启动hdfs
1 start-yarn.sh
验证启动:http://master:18088
Hive安装
- 解压缩hive安装包
1 tar -zxvf apache-hive-2.1.1-bin.tar.gz
- 重命名
1 mv apache-hive-2.1.1 hive-2.1.1
- 安装mysql
1 sudo yum install mysql-server
- 安装 mysql connector
1 sudo yum install mysql-connector-java
该命令会在/usr/share/java/下产生mysql-connector-java.jar
- 建立链接
1 ln -s /usr/share/java/mysql-connector-java.jar /home/zhangc/application/hive-2.1.1/lib/mysql-connector-java.jar
该命令在hive安装目录的lib目录下建立软链接,指向/usr/share/java/mysql-connector-java.jar
- 启动mysql
1 sudo service mysqld start
可通过以下命令验证mysql启动成功(显示mysql进程):
1 ps aux | grep mysql
- 修改mysql密码
修改mysql密码(改为zhangc,请根据需要自行替换newpass)
1 mysql -u root 2 use mysql; 3 UPDATE user SET Password = PASSWORD('zhangc') WHERE user = 'root'; 4 FLUSH PRIVILEGES;
允许用户root通过任意机器访问mysql:
1 GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'zhangc' WITH GRANT OPTION; 2 FLUSH PRIVILEGES;
- 配置环境变量~/.bash_profile
1 export HIVE_HOME=/home/zhangc/application/hive-2.1.1 2 export PATH=$HIVE_HOME/bin:$PATH
运行source生效:
1 source ~/.bash_profile
- 配置hive
/home/zhangc/application/hive-2.1.1/conf/hive-site.xml
1 <?xml version="1.0" encoding="UTF-8"?> 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 3 <!-- 4 Licensed under the Apache License, Version 2.0 (the "License"); 5 you may not use this file except in compliance with the License. 6 You may obtain a copy of the License at 7 8 http://www.apache.org/licenses/LICENSE-2.0 9 10 Unless required by applicable law or agreed to in writing, software 11 distributed under the License is distributed on an "AS IS" BASIS, 12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 See the License for the specific language governing permissions and 14 limitations under the License. See accompanying LICENSE file. 15 --> 16 17 <!-- Put site-specific property overrides in this file. --> 18 <configuration> 19 <property> 20 <name>hive.metastore.uris</name> 21 <value>thrift://master:9083</value> 22 </property> 23 <property> 24 <name>hive.server2.thrift.port</name> 25 <value>10000</value> 26 </property> 27 <property> 28 <name>javax.jdo.option.ConnectionURL</name> 29 <value>jdbc:mysql://master/metastore?createDatabaseIfNotExist=true</value> 30 </property> 31 <property> 32 <name>javax.jdo.option.ConnectionDriverName</name> 33 <value>com.mysql.jdbc.Driver</value> 34 </property> 35 <property> 36 <name>javax.jdo.option.ConnectionUserName</name> 37 <value>root</value> 38 </property> 39 <property> 40 <name>javax.jdo.option.ConnectionPassword</name> 41 <value>zhangc</value> 42 </property> 43 <property> 44 <name>hive.metastore.schema.verification</name> 45 <value>false</value> 46 </property> 47 <property> 48 <name>hive.metastore.warehouse.dir</name> 49 <value>/warehouse</value> 50 </property> 51 <property> 52 <name>fs.defaultFS</name> 53 <value>hdfs://master:9000</value> 54 </property> 55 <property> 56 <name>datanucleus.autoCreateSchema</name> 57 <value>true</value> 58 </property> 59 <property> 60 <name>datanucleus.autoStartMechanism</name> 61 <value>SchemaTable</value> 62 </property> 63 <property> 64 <name>datanucleus.schema.autoCreateTables</name> 65 <value>true</value> 66 </property> 67 <property> 68 <name>beeline.hs2.connection.user</name> 69 <value>root</value> 70 </property> 71 <property> 72 <name>beeline.hs2.connection.password</name> 73 <value>zhangc</value> 74 </property> 75 </configuration>
注意(可跳过这部分):
1.hive.metastore.uris中的“master“含义为metastore所在的机器
2.javax.jdo.option.ConnectionURL中的“master”为mysql安装机器的hostname
3. javax.jdo.option.ConnectionUserName和javax.jdo.option.ConnectionPassword分别为mysql的访问用户和密码,可通过以下命令验证是否有效(期中root为javax.jdo.option.ConnectionURL中配置的地址,xxx为mysql用户名): mysql –h master-u root –p
4. fs.defaultFS为HDFS的namenode启动的机器地址
5. beeline.hs2.connection.user和beeline.hs2.connection.password是beeline方式访问的用户名和密码,可任意指定,但在beeline访问时要写入你指定的这个(具体参考最后一部分)
- 启动metastore
1 nohup hive --service metastore >> /home/zhangc/application/hive-2.1.1/metastore.log 2>&1 &
- 启动hive server
1 nohup hive --service hiveserver2 >> /home/zhangc/application/hive-2.1.1/hiveserver.log 2>&1 &
查看hive metastore和hiveserver2是否启动成功
1 ps aux | grep hive
能输出两个进程,分别对应metastore和hiveserver2
- 进入hive
1 hive