在apache hadoop2.6 上部署hive 并将hive数据源存储于Mysql

集成hive 的前提是apache hadoop 集群能够正常启动。

hadoop 版本 apach2.6.0  hive 版本：1.2.1

1、安装mysql 并赋予权限：

 1.1：创建hive 用户和密码：

Create user  ‘hive’ identified by ‘123456’

1.2：创建数据库：create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

1.3：授权允许远程登陆：grant all privileges on *.* to 'user'@'hostname' identified by 'passwd' with grant option;

flush privileges;

1.4 将mysql 驱动复制到 hive/lib 目录下

　　mysql-connector-java-5.1.38-bin.jar

 

2、hive 配置文件在conf 目录下：hive-env.sh

cp hive-env.sh.template hive-env.sh

最后添加如下配置：

export JAVA_HOME=/usr/java/jdk1.7.0_79
export HIVE_HOME=/home/hadoop/hive-1.2.1-bin
export HADOOP_HOME=/home/hadoop/hadoop-2.6.0

3、hive-site.xml

3.1hdfs 上的文件需要事先创建，如果出现权限问题修改对应的权限即可：hadoop dfs -chmod -R 777 hdfs://node17:9000/hive/scratchdir

3.2需要将下述jar文件复制到hive/lib目录下
home/hadoop/hive-1.2.1-bin/lib/hive-hbase-handler-1.2.1.jar
file:///home/hadoop/hive-1.2.1-bin/lib/protobuf-java-2.5.0.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/hbase-client-1.0.0.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/hbase-common-1.0.0.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/zookeeper-3.4.6.jar,
file:///home/hadoop/hive-1.2.1-bin/lib/guava-12.0.1.jar

 

<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://node17:9000/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://node17:9000/hive/scratchdir</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/hive-1.2.1-bin/logs</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://node15:3306/hive?createDatabaseIfNotExist=true</value>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>

<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>

<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>

<property>
<name>hive.aux.jars.path</name>
<value>file:///home/hadoop/hive-1.2.1-bin/lib/hive-hbase-handler-1.2.1.jar ,file:///home/hadoop/hive-1.2.1-bin/lib/protobuf-java-2.5.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/hbase-client-1.0.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/hbase-common-1.0.0.jar,file:///home/hadoop/hive-1.2.1-bin/lib/zookeeper-3.4.6.jar,file:///home/hadoop/hive-1.2.1-bin/lib/guava-12.0.1.jar</value>
</property>

<property>
<name>hive.metastore.uris</name>
<value>thrift://node17:9083</value>       
</property>

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/home/hadoop/hive-1.2.1-bin/tmp</value>
  </property>

</configuration>

出现问题：

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

 

解决办法：

启动metastore service服务即可。

./hive --service metastore

 

解决办法：

删除 yarn 下的 jline-0.9.94.jar jar 包即可。

./hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar