Hive 组件安装配置
1. 实验一：Hive 组件安装配置
1.1. 实验目的
完成本实验，您应该能够：
- 掌握Hive 组件安装配置
- 掌握Hive 组件格式化和启动
1.2. 实验要求
- 熟悉Hive 组件安装配置
- 了解Hive 组件格式化和启动
1.3. 实验过程
1.3.1. 实验任务一：下载和解压安装文件
1.3.1.1. 步骤一：基础环境和安装准备
Hive 组件需要基于Hadoop 系统进行安装。因此，在安装 Hive 组件前，需要确保 Hadoop 系统能够正常运行。本章节内容是基于之前已部署完毕的 Hadoop 全分布系统，在 master 节点上实现 Hive 组件安装。
Hive 组件的部署规划和软件包路径如下：
（1） 当前环境中已安装 Hadoop 全分布系统。
（2） 本地安装 MySQL 数据库（账号 root，密码 Password123$）， 软件包在/opt/software/mysql-5.7.18 路径下。
（3） MySQL 端口号（3306）。
（4） MySQL 的 JDBC 驱动包/opt/software/mysql-connector-java-5.1.47.jar， 在此基础上更新 Hive 元数据存储。
（5） Hive 软件包/opt/software/apache-hive-2.0.0-bin.tar.gz。
1.3.1.2. 步骤二：解压安装文件
（1）使用 root 用户，将 Hive 安装包
/opt/software/apache-hive-2.0.0-bin.tar.gz 路解压到/usr/local/src 路径下。
[root@master ~]# tar -zxvf /opt/software/apache-hive-2.0.0-bin.tar.gz -C /usr/local/src/
（2）将解压后的 apache-hive-2.0.0-bin 文件夹更名为 hive；
[root@master ~]# mv /usr/local/src/apache-hive-2.0.0-bin/ /usr/local/src/hive/
（3）修改 hive 目录归属用户和用户组为 hadoop
[root@master ~]# chown -R hadoop:hadoop /usr/local/src/hive
1.4.2. 实验任务二：设置 Hive 环境
1.4.2.1. 步骤一：卸载MariaDB 数据库
Hive 元数据存储在 MySQL 数据库中，因此在部署 Hive 组件前需要首先在 Linux 系统下安装 MySQL 数据库，并进行 MySQL 字符集、安全初始化、远程访问权限等相关配置。需要使用 root 用户登录，执行如下操作步骤：
（1）关闭 Linux 系统防火墙，并将防火墙设定为系统开机并不自动启动。
[root@master ~]# systemctl stop firewalld
[root@master ~]# systemctl disable firewalld
（2）卸载 Linux 系统自带的 MariaDB。
-
首先查看 Linux 系统中 MariaDB 的安装情况。
[root@master ~]# rpm -qa | grep mariadb
2）卸载 MariaDB 软件包。
我这里没有就不需要卸载
1.4.2.2. 步骤二：安装MySQL 数据库
（1）按如下顺序依次按照 MySQL 数据库的 mysql common、mysql libs、mysql client 软件包。
[root@master ~]# cd /opt/software/mysql-5.7.18/
[root@master mysql-5.7.18]# rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
package mysql-community-common-5.7.18-1.el7.x86_64 is already installed
[root@master mysql-5.7.18]# rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
package mysql-community-libs-5.7.18-1.el7.x86_64 is already installed
[root@master mysql-5.7.18]# rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
package mysql-community-client-5.7.18-1.el7.x86_64 is already installed
（2）安装 mysql server 软件包。
[root@master mysql-5.7.18]# rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.18-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
package mysql-community-server-5.7.18-1.el7.x86_64 is already installed
（3）修改 MySQL 数据库配置，在/etc/my.cnf 文件中添加如表 6-1 所示的 MySQL 数据库配置项。
将以下配置信息添加到/etc/my.cnf 文件 symbolic-links=0 配置信息的下方。
default-storage-engine=innodb
innodb_file_per_table
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'
character-set-server=utf8
（4）启动 MySQL 数据库。
[root@master ~]# systemctl start mysqld
（5）查询 MySQL 数据库状态。mysqld 进程状态为 active (running)，则表示 MySQL 数据库正常运行。
如果 mysqld 进程状态为 failed，则表示 MySQL 数据库启动异常。此时需要排查/etc/my.cnf 文件。
[root@master ~]# systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2022-04-10 22:54:39 CST; 1h 0min ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Main PID: 929 (mysqld)
CGroup: /system.slice/mysqld.service
└─929 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/my...
Apr 10 22:54:35 master systemd[1]: Starting MySQL Server...
Apr 10 22:54:39 master systemd[1]: Started MySQL Server.
（6）查询 MySQL 数据库默认密码。
[root@master ~]# cat /var/log/mysqld.log | grep password
2022-04-08T16:20:04.456271Z 1 [Note] A temporary password is generated for root@localhost: 0yf>>yWdMd8_
MySQL 数据库是安装后随机生成的，所以每次安装后生成的默认密码不相同。
（7）MySQL 数据库初始化。 0yf>>yWdMd8_
执行 mysql_secure_installation 命令初始化 MySQL 数据库，初始化过程中需要设定数据库 root 用户登录密码，密码需符合安全规则，包括大小写字符、数字和特殊符号， 可设定密码为 Password123$。
在进行 MySQL 数据库初始化过程中会出现以下交互确认信息：
1） Change the password for root ? ((Press y|Y for Yes, any other key for No)表示是否更改 root 用户密码，在键盘输入 y 和回车。
2） Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No)表示是否使用设定的密码继续，在键盘输入 y 和回车。
3） Remove anonymous users? (Press y|Y for Yes, any other key for No)表示是否删除匿名用户，在键盘输入 y 和回车。
4） Disallow root login remotely? (Press y|Y for Yes, any other key for No) 表示是否拒绝 root 用户远程登录，在键盘输入 n 和回车，表示允许 root 用户远程登录。
5） Remove test database and access to it? (Press y|Y for Yes, any other key for No)表示是否删除测试数据库，在键盘输入 y 和回车。
6） Reload privilege tables now? (Press y|Y for Yes, any other key for No) 表示是否重新加载授权表，在键盘输入 y 和回车。
mysql_secure_installation 命令执行过程如下：
[root@master ~]# mysql_secure_installation
Securing the MySQL server deployment.
Enter password for user root:
The 'validate_password' plugin is installed on the server.
The subsequent steps will run with the existing configuration
of the plugin.
Using existing password for root.
Estimated strength of the password: 100
Change the password for root ? ((Press y|Y for Yes, any other key for No) : y
New password:
Re-enter new password:
Estimated strength of the password: 100
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : n
... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
- Dropping test database...
Success.
- Removing privileges on test database...
Success.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.
All done!
(7) 添加 root 用户从本地和远程访问 MySQL 数据库表单的授权。
[root@master ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> grant all privileges on *.* to root@'localhost' identified by 'Password123$';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> grant all privileges on *.* to root@'%' identified by 'Password123$';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> select user,host from mysql.user where user='root';
+------+-----------+
| user | host |
+------+-----------+
| root | % |
| root | localhost |
+------+-----------+
2 rows in set (0.00 sec)
mysql> exit;
Bye
1.3.2.3. 步骤三：配置 Hive 组件
（1）设置 Hive 环境变量并使其生效。
[root@master ~]# vim /etc/profile
export HIVE_HOME=/usr/local/src/hive
export PATH=$PATH:$HIVE_HOME/bin
[root@master ~]# source /etc/profile
（2）修改 Hive 组件配置文件。
切换到 hadoop 用户执行以下对 Hive 组件的配置操作。
将/usr/local/src/hive/conf 文件夹下 hive-default.xml.template 文件，更名为hive-site.xml。
[root@master ~]# su - hadoop
Last login: Sun Apr 10 23:27:25 CS
[hadoop@master ~]$ cp /usr/local/src/hive/conf/hive-default.xml.template /usr/local/src/hive/conf/hive-site.xml
（3）通过 vi 编辑器修改 hive-site.xml 文件实现 Hive 连接 MySQL 数据库，并设定Hive 临时文件存储路径。
[hadoop@master ~]$ vi /usr/local/src/hive/conf/hive-site.xml
1）设置 MySQL 数据库连接。
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&us eSSL=false</value>
<description>JDBC connect string for a JDBC metastore</description>
2）配置 MySQL 数据库 root 的密码。
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>Password123$</value>
<description>password to use against s database</description>
</property>
3）验证元数据存储版本一致性。若默认 false，则不用修改。
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description>
Enforce metastore schema version consistency.
True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic
False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>
4）配置数据库驱动。
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
5）配置数据库用户名 javax.jdo.option.ConnectionUserName 为 root。
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>Username to use against metastore database</description>
</property>
6 ）将以下位置的 ${system:java.io.tmpdir}/${system:user.name} 替换为“/usr/local/src/hive/tmp”目录及其子目录。
需要替换以下 4 处配置内容：
<name>hive.querylog.location</name>
<value>/usr/local/src/hive/tmp</value>
<description>Location of Hive run time structured log file</description>
<name>hive.exec.local.scratchdir</name>
<value>/usr/local/src/hive/tmp</value>
<name>hive.downloaded.resources.dir</name>
<value>/usr/local/src/hive/tmp/resources</value>
<name>hive.server2.logging.operation.log.location</name>
<value>/usr/local/src/hive/tmp/operation_logs</value>
7）在Hive安装目录中创建临时文件夹 tmp。
[hadoop@master ~]$ mkdir /usr/local/src/hive/tmp
至此，Hive 组件安装和配置完成。
1.3.2.4. 步骤四：初始化 hive 元数据
1）将 MySQL 数据库驱动（/opt/software/mysql-connector-java-5.1.46.jar）拷贝到Hive 安装目录的 lib 下；
[hadoop@master ~]$ cp /opt/software/mysql-connector-java-5.1.46.jar /usr/local/src/hive/lib/
2）重新启动 hadooop 即可
[hadoop@master ~]$ stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
10.10.10.129: stopping datanode
10.10.10.130: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
10.10.10.129: stopping nodemanager
10.10.10.130: stopping nodemanager
no proxyserver to stop
[hadoop@master ~]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.out
10.10.10.129: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.out
10.10.10.130: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.out
10.10.10.130: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.out
10.10.10.129: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.out
3）初始化数据库
[hadoop@master ~]$ schematool -initSchema -dbType mysql
which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true&us eSSL=false
Metastore Connection Driver :com.mysql.jdbc.Driver
Metastore connection User: root
Mon Apr 11 00:46:32 CST 2022 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Starting metastore schema initialization to 2.0.0
Initialization script hive-schema-2.0.0.mysql.sql
Password123$
Password123$
No current connection
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
4）启动 hive
[hadoop@master ~]$ hive
which: no hbase in (/usr/local/src/jdk/bin:/usr/local/src/hadoop/bin:/usr/local/src/hadoop/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/src/hive/bin:/home/hadoop/.local/bin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
本文来自博客园，作者：Cloudservice，转载请注明原文链接：https://www.cnblogs.com/whwh/p/16126536.html，只要学不死，就往死里学！
