数仓项目04:环境搭建(MysqlHA+Hive)
1.Mysql安装
在hadoop102和103上安装mysql
安装命令
说明如果报如下错误:
安装autoconf库即可,
yum-y install autoconf,卸载后重新安装即可
1.检查本机是否已经安装了mysql的一些软件包,防止冲突 rpm -qa | grep mysql rpm -qa | grep MySQL 卸载残留的软件包: sudo rpm -e --nodeps mysql-libs-5.1.73-7.el6.x86_64 有时候,还有其他冲突,
[root@localhost soft]# sudo rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm
准备中... ################################# [100%]
正在升级/安装...
1:MySQL-client-5.6.24-1.el6 ################################# [100%]
[root@localhost soft]# sudo rpm -ivh MySQL-server-5.6.24-1.el6.x86_64.rpm
准备中... ################################# [100%]
file /usr/share/mysql/charsets/README from install of MySQL-server-5.6.24-1.el6.x86_64 conflicts with file fro
m package mariadb-libs-1:5.5.60-1.el7_5.x86_64
使用
root@localhost soft]# yum -y remove mariadb-libs-1:5.5.60-1.el7_5.x86_64
已加载插件:fastestmirror, langpacks
正在解决依赖关系
--> 正在检查事务
---> 软件包 mariadb-libs.x86_64.1.5.5.60-1.el7_5 将被 删除
2.安装5.6 sudo rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm sudo rpm -ivh MySQL-server-5.6.24-1.el6.x86_64.rpm 3.配置root用户的密码 查看生成的随机密码: sudo cat /root/.mysql_secret 使用随机密码登录修改新的密码: 启动服务: sudo service mysql start 使用随机密码登录,后修改密码: set password=password('123456'); 4.配置root用户可以再任意机器登录的帐号 ①查看本机的所有帐号 select host,user,password from mysql.user; ②删除不是locahost的root用户 delete from mysql.user where host <> 'localhost'; ③将host=localhost修改为% update mysql.user set host='%' where user='root'; ④刷新用户 flush privileges; ⑤测试root是否可以从localhost主机名登录 mysql -uroot -p123456 ⑥测试root是否可以从hadoop103(从外部地址)主机名登录 mysql -h hadoop103 -uroot -p123456 ⑦查看当前mysql服务器收到了哪些客户端连接请求 sudo mysqladmin processlist -uroot -p123456 5.mysql自定义配置文件的存放位置 /etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf
安装记录
[hadoop@hadoop103 ~]$ rpm -qa | grep mysql mysql-libs-5.1.73-8.el6_8.x86_64 [hadoop@hadoop103 ~]$ rpm -qa | grep MySQL [hadoop@hadoop103 ~]$ sudo rpm -e --nodeps mysql-libs-5.1.73-8.el6_8.x86_64 [hadoop@hadoop103 ~]$ ls /opt/soft/ apache-flume-1.7.0-bin.tar.gz hadoop-lzo-0.4.20.jar mysql-libs.zip apache-hive-1.2.1-bin.tar.gz hadoop-lzo-master.zip presto-cli-0.196-executable.jar apache-kylin-2.5.1-bin-hbase1x.tar.gz hbase-1.3.1-bin.tar.gz presto-server-0.196.tar.gz apache-tez-0.9.1-bin.tar.gz imply-2.7.10.tar.gz sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz azkaban-executor-server-2.5.0.tar.gz jdk-8u144-linux-x64.tar.gz zeppelin-0.8.0-bin-all.tgz azkaban-sql-script-2.5.0.tar.gz kafka_2.11-0.11.0.2.tgz zookeeper-3.4.10.tar.gz azkaban-web-server-2.5.0.tar.gz kafka-clients-0.11.0.2.jar hadoop-2.7.2.tar.gz kafka-manager-1.3.3.22.zip [hadoop@hadoop103 ~]$ unzip /opt/soft/mysql-libs.zip -d /opt/module/ -bash: unzip: command not found [hadoop@hadoop103 ~]$ yum install -y unzip zip Loaded plugins: fastestmirror You need to be root to perform this command. [hadoop@hadoop103 ~]$ sudo yum install -y unzip zip Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile base | 3.7 kB 00:00 epel | 4.7 kB 00:00 epel/primary_db | 6.1 MB 00:00 extras | 3.4 kB 00:00 updates | 3.4 kB 00:00 Resolving Dependencies --> Running transaction check ---> Package unzip.x86_64 0:6.0-5.el6 will be installed ---> Package zip.x86_64 0:3.0-1.el6_7.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================== Package Arch Version Repository Size ====================================================================================================================== Installing: unzip x86_64 6.0-5.el6 base 152 k zip x86_64 3.0-1.el6_7.1 base 259 k Transaction Summary ====================================================================================================================== Install 2 Package(s) Total download size: 411 k Installed size: 1.1 M Downloading Packages: (1/2): unzip-6.0-5.el6.x86_64.rpm | 152 kB 00:00 (2/2): zip-3.0-1.el6_7.1.x86_64.rpm | 259 kB 00:00 ---------------------------------------------------------------------------------------------------------------------- Total 4.3 MB/s | 411 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Warning: RPMDB altered outside of yum. ** Found 3 pre-existing rpmdb problem(s), 'yum check' output follows: 2:postfix-2.6.6-8.el6.x86_64 has missing requires of libmysqlclient.so.16()(64bit) 2:postfix-2.6.6-8.el6.x86_64 has missing requires of libmysqlclient.so.16(libmysqlclient_16)(64bit) 2:postfix-2.6.6-8.el6.x86_64 has missing requires of mysql-libs Installing : zip-3.0-1.el6_7.1.x86_64 1/2 Installing : unzip-6.0-5.el6.x86_64 2/2 Verifying : unzip-6.0-5.el6.x86_64 1/2 Verifying : zip-3.0-1.el6_7.1.x86_64 2/2 Installed: unzip.x86_64 0:6.0-5.el6 zip.x86_64 0:3.0-1.el6_7.1 Complete! [hadoop@hadoop103 ~]$ unzip /opt/soft/mysql-libs.zip -d /opt/module/ Archive: /opt/soft/mysql-libs.zip creating: /opt/module/mysql-libs/ inflating: /opt/module/mysql-libs/MySQL-client-5.6.24-1.el6.x86_64.rpm inflating: /opt/module/mysql-libs/mysql-connector-java-5.1.27.tar.gz inflating: /opt/module/mysql-libs/MySQL-server-5.6.24-1.el6.x86_64.rpm [hadoop@hadoop103 ~]$ cd /opt/module/mysql-libs/ [hadoop@hadoop103 mysql-libs]$ ll total 76048 -rw-rw-r-- 1 hadoop hadoop 18509960 Mar 26 2015 MySQL-client-5.6.24-1.el6.x86_64.rpm -rw-rw-r-- 1 hadoop hadoop 3575135 Dec 1 2013 mysql-connector-java-5.1.27.tar.gz -rw-rw-r-- 1 hadoop hadoop 55782196 Mar 26 2015 MySQL-server-5.6.24-1.el6.x86_64.rpm [hadoop@hadoop103 mysql-libs]$ sudo rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm Preparing... ########################################### [100%] 1:MySQL-client ########################################### [100%] [hadoop@hadoop103 mysql-libs]$ sudo rpm -ivh MySQL-server-5.6.24-1.el6.x86_64.rpm Preparing... ########################################### [100%] 1:MySQL-server ########################################### [100%] warning: user mysql does not exist - using root warning: group mysql does not exist - using root 2020-11-20 14:48:14 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_fo r_timestamp server option (see documentation for more details).2020-11-20 14:48:14 0 [Note] /usr/sbin/mysqld (mysqld 5.6.24) starting as process 3314 ... 2020-11-20 14:48:14 3314 [Note] InnoDB: Using atomics to ref count buffer pool pages 2020-11-20 14:48:14 3314 [Note] InnoDB: The InnoDB memory heap is disabled 2020-11-20 14:48:14 3314 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-11-20 14:48:14 3314 [Note] InnoDB: Memory barrier is not used 2020-11-20 14:48:14 3314 [Note] InnoDB: Compressed tables use zlib 1.2.3 2020-11-20 14:48:14 3314 [Note] InnoDB: Using Linux native AIO 2020-11-20 14:48:14 3314 [Note] InnoDB: Using CPU crc32 instructions 2020-11-20 14:48:14 3314 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2020-11-20 14:48:14 3314 [Note] InnoDB: Completed initialization of buffer pool 2020-11-20 14:48:14 3314 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be cr eated!2020-11-20 14:48:14 3314 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB 2020-11-20 14:48:14 3314 [Note] InnoDB: Database physically writes the file full: wait... 2020-11-20 14:48:15 3314 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB 2020-11-20 14:48:15 3314 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB 2020-11-20 14:48:15 3314 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 2020-11-20 14:48:15 3314 [Warning] InnoDB: New log files created, LSN=45781 2020-11-20 14:48:15 3314 [Note] InnoDB: Doublewrite buffer not found: creating new 2020-11-20 14:48:15 3314 [Note] InnoDB: Doublewrite buffer created 2020-11-20 14:48:15 3314 [Note] InnoDB: 128 rollback segment(s) are active. 2020-11-20 14:48:15 3314 [Warning] InnoDB: Creating foreign key constraint system tables. 2020-11-20 14:48:16 3314 [Note] InnoDB: Foreign key constraint system tables created 2020-11-20 14:48:16 3314 [Note] InnoDB: Creating tablespace and datafile system tables. 2020-11-20 14:48:16 3314 [Note] InnoDB: Tablespace and datafile system tables created. 2020-11-20 14:48:16 3314 [Note] InnoDB: Waiting for purge to start 2020-11-20 14:48:16 3314 [Note] InnoDB: 5.6.24 started; log sequence number 0 A random root password has been set. You will find it in '/root/.mysql_secret'. 2020-11-20 14:48:16 3314 [Note] Binlog end 2020-11-20 14:48:16 3314 [Note] InnoDB: FTS optimize thread exiting. 2020-11-20 14:48:16 3314 [Note] InnoDB: Starting shutdown... 2020-11-20 14:48:17 3314 [Note] InnoDB: Shutdown completed; log sequence number 1625977 2020-11-20 14:48:17 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_fo r_timestamp server option (see documentation for more details).2020-11-20 14:48:17 0 [Note] /usr/sbin/mysqld (mysqld 5.6.24) starting as process 3336 ... 2020-11-20 14:48:17 3336 [Note] InnoDB: Using atomics to ref count buffer pool pages 2020-11-20 14:48:17 3336 [Note] InnoDB: The InnoDB memory heap is disabled 2020-11-20 14:48:17 3336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-11-20 14:48:17 3336 [Note] InnoDB: Memory barrier is not used 2020-11-20 14:48:17 3336 [Note] InnoDB: Compressed tables use zlib 1.2.3 2020-11-20 14:48:17 3336 [Note] InnoDB: Using Linux native AIO 2020-11-20 14:48:17 3336 [Note] InnoDB: Using CPU crc32 instructions 2020-11-20 14:48:17 3336 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2020-11-20 14:48:17 3336 [Note] InnoDB: Completed initialization of buffer pool 2020-11-20 14:48:17 3336 [Note] InnoDB: Highest supported file format is Barracuda. 2020-11-20 14:48:17 3336 [Note] InnoDB: 128 rollback segment(s) are active. 2020-11-20 14:48:17 3336 [Note] InnoDB: Waiting for purge to start 2020-11-20 14:48:18 3336 [Note] InnoDB: 5.6.24 started; log sequence number 1625977 2020-11-20 14:48:18 3336 [Note] Binlog end 2020-11-20 14:48:18 3336 [Note] InnoDB: FTS optimize thread exiting. 2020-11-20 14:48:18 3336 [Note] InnoDB: Starting shutdown... 2020-11-20 14:48:19 3336 [Note] InnoDB: Shutdown completed; log sequence number 1625987 A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER ! You will find that password in '/root/.mysql_secret'. You must change that password on your first connect, no other statement but 'SET PASSWORD' will be accepted. See the manual for the semantics of the 'password expired' flag. Also, the account for the anonymous user has been removed. In addition, you can run: /usr/bin/mysql_secure_installation which will also give you the option of removing the test database. This is strongly recommended for production servers. See the manual for more instructions. Please report any problems at http://bugs.mysql.com/ The latest information about MySQL is available on the web at http://www.mysql.com Support MySQL by buying support/licenses at http://shop.mysql.com New default config file was created as /usr/my.cnf and will be used by default by the server when you start it. You may edit this file to change server settings [hadoop@hadoop103 mysql-libs]$ sudo cat /root/.mysql_secret # The random password set for the root user at Fri Nov 20 14:48:16 2020 (local time): TMXkwqqmdeBpiD99 [hadoop@hadoop103 mysql-libs]$ mysql -uroot -pTMXkwqqmdeBpiD99 Warning: Using a password on the command line interface can be insecure. ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) [hadoop@hadoop103 mysql-libs]$ mysql -uroot -p TMXkwqqmdeBpiD99 Enter password: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) [hadoop@hadoop103 mysql-libs]$ mysql -uroot -p^CXkwqqmdeBpiD99 [hadoop@hadoop103 mysql-libs]$ sudo service mysql status ERROR! MySQL is not running [hadoop@hadoop103 mysql-libs]$ sudo service mysql start Starting MySQL. SUCCESS! [hadoop@hadoop103 mysql-libs]$ mysql -uroot -pTMXkwqqmdeBpiD99 Warning: Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: 5.6.24 Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> set password=password('123456'); Query OK, 0 rows affected (0.00 sec) mysql> select host,user,password from mysql.user; +-----------+------+-------------------------------------------+ | host | user | password | +-----------+------+-------------------------------------------+ | localhost | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 | | hadoop103 | root | *7A5A4D8630A3EAB7B5C3E79C33DDC69E9F67B39A | | 127.0.0.1 | root | *7A5A4D8630A3EAB7B5C3E79C33DDC69E9F67B39A | | ::1 | root | *7A5A4D8630A3EAB7B5C3E79C33DDC69E9F67B39A | +-----------+------+-------------------------------------------+ 4 rows in set (0.00 sec) mysql> delete from mysql.user where host <> 'localhost'; Query OK, 3 rows affected (0.00 sec) mysql> update mysql.user set host='%' where user='root'; Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> quit Bye [hadoop@hadoop103 mysql-libs]$ mysql -uroot -p123456 Warning: Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.6.24 MySQL Community Server (GPL) Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> Ctrl-C -- exit! Aborted [hadoop@hadoop103 mysql-libs]$ mysql -h hadoop103 -uroot -p123456 Warning: Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.6.24 MySQL Community Server (GPL) Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> Ctrl-C -- exit! Aborted [hadoop@hadoop103 mysql-libs]$ sudo mysqladmin processlist -uroot -p123456 Warning: Using a password on the command line interface can be insecure. +----+------+-----------+----+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+----+---------+------+-------+------------------+ | 4 | root | localhost | | Query | 0 | init | show processlist | +----+------+-----------+----+---------+------+-------+------------------+ [hadoop@hadoop103 mysql-libs]$ mysql -h hadoop103 -uroot -p123456 Warning: Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 7 Server version: 5.6.24 MySQL Community Server (GPL) Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> Ctrl-C -- exit! Aborted [hadoop@hadoop103 mysql-libs]$ sudo mysqladmin processlist -uroot -p123456 Warning: Using a password on the command line interface can be insecure. +----+------+-----------------+----+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------------+----+---------+------+-------+------------------+ | 8 | root | hadoop103:53254 | | Sleep | 9 | | | | 9 | root | localhost | | Query | 0 | init | show processlist | +----+------+-----------------+----+---------+------+-------+------------------+ [hadoop@hadoop103 mysql-libs]$
2.配置互为主从的Mysql
1.到/usr/share/mysql下找mysql服务端配置的模版 sudo cp my-default.cnf /etc/my.cnf 2.编辑my.cnf 在[mysqld]下配置: server_id = 103 log-bin=mysql-bin binlog_format=mixed relay_log=mysql-relay 另外一台,配置也一样,只需要修改servei_id 3.重启mysql服务 sudo service mysql restart 4.在主机上使用root@localhost登录,授权从机可以使用哪个用户登录 GRANT replication slave ON *.* TO 'slave'@'%' IDENTIFIED BY '123456'; 5.查看主机binlog文件的最新位置 show master status; 6.在从机上执行以下语句 change master to master_user='slave', master_password='123456',master_host='192.168.6.103',master_log_file='mysql-bin.000001',master_log_pos=311; 7.在从机上开启同步线程 start slave 8.查看同步线程的状态 show slave status \G change master to master_user='slave', master_password='123456',master_host='192.168.6.102',master_log_file='mysql-bin.000001',master_log_pos=311;
3.HA的mysql搭建(安装keepalive软件)
阿里云服务器不能使用keepalived,普通的ECS没有HA VIP,所以我们的ha搭建失败了。。。。
高可用虚拟IP只能在专有网络上使用
https://help.aliyun.com/document_detail/110065.html
弹性公网IP EIP(Elastic IP Address)
阿里云SLB实现 MySQL HA
https://blog.csdn.net/weixin_41507897/article/details/108331285
阿里云服务器手动实现mysql双机热备的两种方式
https://www.jb51.net/article/171848.htm
普通主机参考方法如下:
1)通过yum方式安装Keepalived sudo yum install -y keepalived 2)修改Keepalived配置文件/etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id MySQL-ha } vrrp_instance VI_1 { state master #初始状态 interface eth0 #网卡 virtual_router_id 51 #虚拟路由id priority 100 #优先级 advert_int 1 #Keepalived心跳间隔 nopreempt #只在高优先级配置,原master恢复之后不重新上位 authentication { auth_type PASS #认证相关 auth_pass 1111 } virtual_ipaddress { 192.168.1.100 #虚拟ip } } #声明虚拟服务器 virtual_server 192.168.1.100 3306 { delay_loop 6 persistence_timeout 30 protocol TCP #声明真实服务器 real_server 192.168.1.103 3306 { notify_down /var/lib/mysql/killkeepalived.sh #真实服务故障后调用脚本 TCP_CHECK { connect_timeout 3 #超时时间 nb_get_retry 1 #重试次数 delay_before_retry 1 #重试时间间隔 } } } 3)编辑脚本文件/var/lib/mysql/killkeepalived.sh #! /bin/bash sudo service keepalived stop 4)加执行权限 sudo chmod +x /var/lib/mysql/killkeepalived.sh 5)启动Keepalived服务 sudo service keepalived start 6)设置Keepalived服务开机自启 sudo chkconfig keepalived on 7)确保开机时MySQL先于Keepalived启动 第一步:查看MySQL启动次序 sudo vim /etc/init.d/mysql 第二步:查看Keepalived启动次序 sudo vim /etc/init.d/keepalived 第三步:若Keepalived先于MySQL启动,则需要按照以下步骤设置二者启动顺序 1.修改/etc/init.d/mysql sudo vim /etc/init.d/mysql 2.重新设置mysql开机自启 sudo chkconfig --del mysql sudo chkconfig --add mysql sudo chkconfig mysql on 3.修改/etc/init.d/keepalived sudo vim /etc/init.d/keepalived 4.重新设置keepalived开机自启 sudo chkconfig --del keepalived sudo chkconfig --add keepalived sudo chkconfig keepalivedon
5.Hive的安装
因为hive只是一个客户端,配置在103上面
1.配置
保证环境变量中有JAVA_HOME,HADOOP_HOME,HIVE_HOME即可
2.配置hive的元数据存储在mysql中
①拷贝Mysql的驱动到 $HIVE_HOME/lib中
②编辑hive-site.xml文件,配置元数据的存储位置
[hadoop@hadoop103 hive]$ cat conf/hive-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://hadoop103:3306/metastore?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>123456</value> <description>password to use against metastore database</description> </property> <property> <name>hive.cli.print.header</name> <value>true</value> </property> <property> <name>hive.cli.print.current.db</name> <value>true</value> </property> </configuration>
③metastore的库的编码必须为latin1(手动创建)
6.Tez的安装
①解压缩,将tez的tar包上传到hdfs
②在$HIVE_HOME/conf/中,编写tez-site.xml
③编写$HIVE_HOME/conf/hive-site.xml
④编写$HIVE_HOME/conf/hive-env.sh,让hive启动时,加载tez的jar包
⑤编写yarn-site.xml,并分发,关闭虚拟内存检查
[hadoop@hadoop103 soft]$ tar -zxvf apache-tez-0.9.1-bin.tar.gz -C ../module/ [hadoop@hadoop103 module]$ mv apache-tez-0.9.1-bin/ tez-0.9.1 [hadoop@hadoop103 tez-0.9.1]$ hadoop fs -mkdir /tez [hadoop@hadoop103 tez-0.9.1]$ hadoop fs -put /opt/soft/apache-tez-0.9.1-bin.tar.gz /tez [hadoop@hadoop103 conf]$ pwd /opt/module/hive/conf [hadoop@hadoop103 conf]$ cat tez-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>tez.lib.uris</name> <value>${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz</value> </property> <property> <name>tez.use.cluster.hadoop-libs</name> <value>true</value> </property> <property> <name>tez.history.logging.service.class</name> <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLog gingService</value></property> </configuration> [hadoop@hadoop103 conf]$ mv hive-env.sh.template hive-env.sh [hadoop@hadoop103 conf]$ vi hive-env.sh #export HADOOP_HOME=/opt/module/hadoop-2.7.2 # Hive Configuration Directory can be controlled by: #export HIVE_CONF_DIR=/opt/module/hive/conf # Folder containing extra libraries required for hive compilation/execution can be controlled by: export TEZ_HOME=/opt/module/tez-0.9.1 #是你的tez的解压目录 export TEZ_JARS="" for jar in `ls $TEZ_HOME |grep jar`; do export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar done for jar in `ls $TEZ_HOME/lib`; do export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar done export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS [hadoop@hadoop103 conf]$ 在hive-site.xml文件中添加如下配置,更改hive计算引擎 <property> <name>hive.execution.engine</name> <value>tez</value> </property> 关闭元数据检查 [atguigu@hadoop102 conf]$ pwd /opt/module/hive/conf [atguigu@hadoop102 conf]$ vim hive-site.xml 增加如下配置: <property> <name>hive.metastore.schema.verification</name> <value>false</value> </property> 关掉虚拟内存检查,修改yarn-site.xml,修改后一定要分发,并重新启动hadoop集群。 <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> [hadoop@hadoop103 hadoop]$ xsync yarn-site.xml [hadoop@hadoop103 conf]$ hd stop Stopping namenodes on [hadoop102] hadoop102: stopping namenode hadoop104: stopping datanode hadoop102: stopping datanode hadoop103: stopping datanode Stopping secondary namenodes [hadoop104] hadoop104: no secondarynamenode to stop stopping yarn daemons stopping resourcemanager hadoop104: stopping nodemanager hadoop102: stopping nodemanager hadoop103: stopping nodemanager no proxyserver to stop stopping historyserver [hadoop@hadoop103 conf]$ hd start Starting namenodes on [hadoop102] hadoop102: starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hadoop-namenode-hadoop102.out hadoop104: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hadoop-datanode-hadoop104.out hadoop103: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hadoop-datanode-hadoop103.out hadoop102: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hadoop-datanode-hadoop102.out Starting secondary namenodes [hadoop104] hadoop104: starting secondarynamenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-hadoop-secondarynamenode-hadoop 104.outstarting yarn daemons starting resourcemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hadoop-resourcemanager-hadoop103.out hadoop104: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hadoop-nodemanager-hadoop104.out hadoop102: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hadoop-nodemanager-hadoop102.out hadoop103: starting nodemanager, logging to /opt/module/hadoop-2.7.2/logs/yarn-hadoop-nodemanager-hadoop103.out starting historyserver, logging to /opt/module/hadoop-2.7.2/logs/mapred-hadoop-historyserver-hadoop102.out [hadoop@hadoop103 conf]$ hive Logging initialized using configuration in jar:file:/opt/module/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j /impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/module/tez-0.9.1/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinde r.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] hive (default)> insert into student values(1,"zhangsan"); Query ID = hadoop_20201130083630_3c6049ee-0b0b-434b-b35d-29773dd87ce5 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1606696495844_0001) -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 .......... SUCCEEDED 1 1 0 0 0 0 -------------------------------------------------------------------------------- VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 11.11 s -------------------------------------------------------------------------------- Loading data to table default.student Table default.student stats: [numFiles=1, numRows=1, totalSize=11, rawDataSize=10] OK _col0 _col1 Time taken: 18.439 seconds hive (default)>
7.DBeaver测试数据库