CDH+Hive部署

CDH介绍：

CDH 全称 Cloudera’s Distribution Including Apache Hadoop，是 Cloudera 公司提供的一个 Apache Hadoop 发行版。CDH 将 Hadoop 与其他十几个关键的开源项目集成，并且加入了集群自动化安装、中心化管理、集群监控、报警等功能。

Hive介绍：

Hive是基于Hadoop的一个数据仓库工具，可以将结构化的数据文件映射为一张表，并提供类SQL查询功能。Hive处理的数据存储在HDFS，Hive分析数据底层的实现是MapReduce，执行程序运行在Yarn上。

优点

（1)操作接口采用类SQL语法，提供快速开发的能力（简单、容易上手）。
（2)避免了去写MapReduce，减少开发人员的学习成本。
（3)Hive的执行延迟比较高，因此Hive常用于数据分析，对实时性要求不高的场合。
（4)Hive优势在于处理大数据，对于处理小数据没有优势，因为Hive的执行延迟比较高
（5)Hive支持用户自定义函数，用户可以根据自己的需求来实现自己的函数。

缺点

Hive的HQL表达能力有限

（1）迭代式算法无法表达
（2）数据挖掘方面不擅长，由于MapReduce数据处理流程的限制，效率更高的算法却无法实现。

Hive的效率比较低

（1）Hive自动生成的MapReduce作业，通常情况下不够智能化
（2）Hive调优比较困难

节点规划：

Cloudera Manager Server 节点提供 WebUI 操作界面，管理 Cloudera Manager Agent 节点，在运行 Agent 的机器上部署大数据相关组件。

IP地址	主机名	角色	硬件配置	操作系统
192.168.11.160	hadoop1	Server,Agent	8C+16G+500G	CentOS7.6
192.168.11.161	hadoop2	Agent	8C+16G+500G	CentOS7.6
192.168.11.162	hadoop3	mysql+Agent	8C+16G+500G	CentOS7.6

版本信息：

JDK：1.8.0

Mysql：8.0.22

Cloudera-Manager：6.3.1

Hadoop：3.0.0

Hive：2.1.1

Hue：4.3.0

Zookeeper：3.4.5

Kafka：2.2.1

将所有所需程序包上传至服务器

环境准备：

1.设置主机名并添加到/etc/hosts文件内

hostnamectl set-hostname hadoop1
hostnamectl set-hostname hadoop2
hostnamectl set-hostname hadoop3

2.设置SSH免密登录

ssh-keygen -t rsa
然后敲（三个回车），就会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）*

将公钥拷贝到要免密登录的目标机器上

ssh-copy-id hadoop1
ssh-copy-id hadoop2
ssh-copy-id hadoop3

3.关闭防火墙

systemctl status firewalld

systemctl stop firewalld
systemctl enable firewalld

4.安装JDK(重要)

# 安装jdk
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 
# 添加环境变量
echo '# jdk' >> /etc/profile
echo 'export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera' >> /etc/profile
echo 'export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile
# 加载环境变量
source /etc/profile
java -version

5.安装MySQL

5.1查看MySQL是否安装

rpm -qa|grep mysql
# mysql-libs-5.1.73-7.el6.x86_64

5.2若有安装将其卸载

rpm -e --nodeps mysql-libs-5.1.73-7.el6.x86_64

删除原有MySql依赖(也适用安装出错干掉原来的) 
yum remove mysql-libs

5.3安装mysql初始化数据

5.3.1上传rpm包并安装（8.0或者5.7）

ls
# mysql-community-client-8.0.22-1.el7.x86_64.rpm
# mysql-community-client-plugins-8.0.22-1.el7.x86_64.rpm
# mysql-community-common-8.0.22-1.el7.x86_64.rpm
# mysql-community-libs-8.0.22-1.el7.x86_64.rpm
# mysql-community-libs-compat-8.0.22-1.el7.x86_64.rpm
# mysql-community-server-8.0.22-1.el7.x86_64.rpm

yum localinstall *.rpm -y

5.3.2修改配置文件/etc/my.cnf

[mysqld]
datadir=/var/lib/mysql/data/
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
#lower_case_table_names=1
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
#query_cache_limit = 8M
#query_cache_size = 64M
#query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/binlog/mysql-bin
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=11
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
prompt=mysql5729_db01 [\\d]>
sql_mode=STRICT_ALL_TABLES
socket=/var/lib/mysql/mysql.sock

5.3.3初始化mysql，初始化完成后注意查看密码

mysqld --defaults-file=/etc/my.cnf --initialize


# 设置忽略大小写,mysql8.0需要在初始化时设置（安装cdh不需要忽略）
mysqld --defaults-file=/etc/my.cnf --initialize --lower-case-table-names=1

5.3.4初始化后修改密码(以下方式选用一个)

set password='Root@123';
FLUSH PRIVILEGES;

# 或者
alter user 'root'@'%' identified with mysql_native_password by '密码';
UPDATE user SET Password=PASSWORD('root@123') where USER='root';

-- 8.0版本直接mysql -uroot -p连接
UPDATE user SET authentication_string=PASSWORD('root@123') where USER='root';
alter user 'boer'@'%'  IDENTIFIED BY 'Boer@123';  
alter user 'root'@'localhost' IDENTIFIED BY 'Root@123'; 
FLUSH PRIVILEGES;

更改权限(以下方式选用一个)

-- Mysql默认不允许远程登录，所以需要开启远程访问权限
update user set host = '%' where user = 'root';	
FLUSH PRIVILEGES;

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Root@123' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION; 
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION; 
FLUSH PRIVILEGES;

6.创建相关数据库

`6.1mysql5.7`

-- mysql5.7
-- scm
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Scm@147258';

-- amon
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Amon@147258';

-- rman
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'Rman@147258';

-- hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Hue@147258';

-- hive
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'Hive@147258';

-- sentry
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;   
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'Sentry@147258';

-- nav
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;      
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'Nav@147258';

-- navms
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'Navms@147258';

-- oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Oozie@147258';

-- flush
FLUSH PRIVILEGES;

6.2`mysql8.0`

-- mysql8.0
-- scm
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'scm'@'%' IDENTIFIED BY 'Scm@147258';
grant all privileges on scm.*  to 'scm'@'%' ;

-- amon
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'amon'@'%' IDENTIFIED BY 'Amon@147258';
grant all privileges on amon.*  to 'amon'@'%' ;

-- rman
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'rman'@'%' IDENTIFIED BY 'Rman@147258';
grant all privileges on rman.*  to 'rman'@'%' ;


-- hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
CREATE USER 'hue'@'%' IDENTIFIED BY 'Hue@147258';
grant all privileges on hue.*  to 'hue'@'%' ;


-- hive
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'hive'@'%' IDENTIFIED BY 'Hive@147258';
grant all privileges on metastore.*  to 'hive'@'%' ;


-- sentry
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'sentry'@'%' IDENTIFIED BY 'Sentry@147258';
grant all privileges on sentry.*  to 'sentry'@'%' ;   


-- nav
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
CREATE USER 'nav'@'%' IDENTIFIED BY 'Nav@147258';
grant all privileges on nav.*  to 'nav'@'%' ;     


-- navms
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'navms'@'%' IDENTIFIED BY 'Navms@147258';
grant all privileges on navms.*  to 'navms'@'%' ;


-- oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'Oozie@147258';
grant all privileges on oozie.*  to 'oozie'@'%' ;


-- flush
FLUSH PRIVILEGES;

7.上传连接mysql的jar包

ls
# mysql-connector-java-8.0.16.jar
修改其名称
cp mysql-connector-java-8.0.16.jar  /usr/share/java/mysql-connector-java.jar

8.安装 cloudera-manager

创建cloudera-manager目录,存放cdh安装文件。解压压缩包

mkdir /opt/cloudera-manager
tar -zxvf cm6.3.1-redhat7.tar.gz
cd cm6.3.1/RPMS/x86_64/
mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/

注：rpm安装,所有节点都安装cloudera-manager-daemons和cloudera-manager-agent

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 

# 安装agent依赖
yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt

rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

8.1修改agent配置文件

vim /etc/cloudera-scm-agent/config.ini

server_host=hadoop1

8.2主节点安装cloudera-manager-server

rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm

8.3修改 server的db.properties

vim /etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop1:3306
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=Scm@147258
com.cloudera.cmf.db.setupType=EXTERNAL

上传parcel包

ls /opt/cloudera/parcel-repo

# CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
# CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
# CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.torrent
# manifest.json

8.4初始化cm库

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h hadoop101 scm scm Scm@147258

9.启动Server与Agent

启动主节点的Server和所有节点的Agent
启动server

service cloudera-scm-server.service start

启动Agent

systemctl start cloudera-scm-agent

启动Server查看日志，等待几分钟后无问题登录

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

10.登录：

默认端口号：7180
默认用户名：admin
默认密码：admin

posted @ 2023-04-04 13:17 Jerry· 阅读(552) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· # 大数据之HUE安装及调测

· # 大数据之Hive安装配置

· 大数据之—CDH搭建

· CDH6.3.2搭建

· CDH搭建教程

阅读排行：
· 25岁的心里话
· 闲置电脑爆改个人服务器（超详细） #公网映射 #Vmware虚拟网络编辑器
· 基于 Docker 搭建 FRP 内网穿透开源项目（很简单哒）
· 零经验选手，Compose 一天开发一款小游戏！
· 一起来玩mcp_server_sqlite，让AI帮你做增删改查！！

阅读目录(Content)

此页目录为空

Jerry

念两句诗

CDH+Hive部署

CDH介绍：

Hive介绍：

节点规划：

版本信息：

环境准备：

1.设置主机名并添加到/etc/hosts文件内

2.设置SSH免密登录

3.关闭防火墙

4.安装JDK(重要)

5.安装MySQL

5.1查看MySQL是否安装

5.2若有安装将其卸载

5.3安装mysql初始化数据

6.创建相关数据库

6.1mysql5.7

6.2mysql8.0

7.上传连接mysql的jar包

8.安装 cloudera-manager

8.1修改agent配置文件

8.2主节点安装cloudera-manager-server

8.3修改 server的db.properties

上传parcel包

8.4初始化cm库

9.启动Server与Agent

启动主节点的Server和所有节点的Agent启动server

启动Agent

启动Server查看日志，等待几分钟后无问题登录

10.登录：

公告

个人信息

日历

搜索

常用链接

最新随笔

我的标签

积分与排名

随笔分类

随笔档案

阅读排行榜

评论排行榜

最新评论

`6.1mysql5.7`

6.2`mysql8.0`

启动主节点的Server和所有节点的Agent
启动server