大数据集群之基于二进制安装包方式部署Cloudera Manager(离线安装）

一、下载CDH和CM

1、CDH的选择

    下载地址：http://archive.cloudera.com/cdh5/parcels/5.15.1/ 
    el5表示RedHat 5/CentOS 5 发行版本应用软件。 　　
    el6表示RedHat 6/CentOS 6 发行版本应用软件。
　　jessie表示Debian 8的发行版本之一。
　　precise表示Ubuntu 12的发行版本。 
　　sles11表示SUSE 11的发行版本。 
　　sles12表示SUSE 12的发行版版本。 
　　trusty表示Ubantu 14的发行版本。 　
     wheezy表示Debian 7的发行版本之一。

2、CM的选择

下载地址：http://archive.cloudera.com/cm5/cm/5/

二、Cloudera Manager安装

1、将下载的Cloudera Manager分发到各个节点

[root@hexindai-c12-120 cm]# ll /export/software/cm/
total 2889644
-rw-r--r--. 1 root root 2120090032 Jul  5 09:24 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel
-rw-r--r--. 1 root root         41 Jul  5 09:24 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1
-rw-r--r--. 1 root root  838894986 Jul  4 17:46 cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz
[root@hexindai-c12-120 cm]# 
[root@hexindai-c12-120 cm]# vim /export/ssh-copy.sh 
IP="
172.20.102.121
172.20.102.122
172.20.102.123
"

for node in ${IP};do
scp -r /export/software/cm ${node}:/export/software/
echo "${node} 基础配置优化完成"
done
~ 
"/export/ssh-copy.sh" 10L, 172C written      

[root@hexindai-c12-120 cm]# sh /export/ssh-copy.sh 
cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz                                                                                                                    100%  800MB 133.3MB/s   00:06    
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel                                                                                                                             100% 2022MB 139.0MB/s   00:14    
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1                                                                                                                        100%   41   150.8KB/s   00:00    
172.20.102.121 基础配置优化完成
cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz                                                                                                                    100%  800MB 138.4MB/s   00:05    
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel                                                                                                                             100% 2022MB 148.5MB/s   00:13    
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1                                                                                                                        100%   41   114.7KB/s   00:00    
172.20.102.122 基础配置优化完成
cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz                                                                                                                    100%  800MB 124.8MB/s   00:06    
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel                                                                                                                             100% 2022MB 131.7MB/s   00:15    
CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1                                                                                                                        100%   41    75.4KB/s   00:00    
172.20.102.123 基础配置优化完成
[root@hexindai-c12-120 cm]#

2、创建解压CM默认解压的目录，如果想要使用其他的解压目录别忘了修改配置文件！

在各节点中创建默认目录：
[root@hexindai-c12-120 cm]# mkdir /opt/cloudera-manager
[root@hexindai-c12-121 software]# mkdir /opt/cloudera-manage
[root@Hexindai-C12-122 ~]# mkdir /opt/cloudera-manage
[root@Hexindai-C12-123 ~]# mkdir /opt/cloudera-manage

3、将CM解压到创建的目录中

[root@hexindai-c12-120 cm]# tar zxvf cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz -C /opt/cloudera-manager/
[root@hexindai-c12-120 cm]# ll /opt/cloudera-manager/
total 0
drwxr-xr-x. 4 1106 4001 36 Jul 31  2018 cloudera
drwxr-xr-x. 9 1106 4001 88 Jul 31  2018 cm-5.15.1
[root@hexindai-c12-120 cm]# 
注意：各节点都要操作！

[root@hexindai-c12-120 cm]# tar zxvf cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz -C /opt/cloudera-manager/ #注意：各节点都要操作！

4、在所有的节点上创建cloudera-scm用户

[root@hexindai-c12-120 cm]# useradd cloudera-scm -s /sbin/nologin 
[root@hexindai-c12-120 cm]# id cloudera-scm
uid=1001(cloudera-scm) gid=1001(cloudera-scm) groups=1001(cloudera-scm)
[root@hexindai-c12-120 cm]#

如果某个agent节点没有创建用户和组的操作，我们在安装服务时可能会出现如下提示：

5、修改Cloudera Manager Agent端的配置文件

[root@hexindai-c12-120 cm]# grep server_port /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini #server_port是server和agent的通信端口，不要随意改动！
server_port=7182
[root@hexindai-c12-120 cm]# grep server_host /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini      #CM服务器默认是本机
server_host=localhost
[root@hexindai-c12-120 cm]#

[root@hexindai-c12-121 cm]# sed -i 's/server_host=localhost/server_host=hexindai-c12-120/g' /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini
You have new mail in /var/spool/mail/root
[root@hexindai-c12-121 cm]# grep server_host /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini
server_host=hexindai-c12-120
[root@hexindai-c12-121 cm]# 
[root@Hexindai-C12-122 cm]# sed -i 's/server_host=localhost/server_host=hexindai-c12-120/g' /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini
[root@Hexindai-C12-122 cm]# grep server_host /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini
server_host=hexindai-c12-120
You have new mail in /var/spool/mail/root
[root@Hexindai-C12-122 cm]# 
[root@Hexindai-C12-123 cm]# sed -i 's/server_host=localhost/server_host=hexindai-c12-120/g' /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini
[root@Hexindai-C12-123 cm]# grep server_host /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini
server_host=hexindai-c12-120
You have new mail in /var/spool/mail/root
[root@Hexindai-C12-123 cm]#

[root@hexindai-c12-121 cm]# sed -i 's/server_host=localhost/server_host=hexindai-c12-120/g' /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini #修改各节点的server_host

6、初始化CM Server数据库（https://www.cloudera.com/documentation/enterprise/5-16-x/topics/prepare_cm_database.html#scm_prepare_syntax）

[root@hexindai-c12-120 cm]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.7.26-log MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> grant all privileges on *.* to 'cdh'@'172.20.102.12%' identified by 'wtnyihg' with grant option;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql>

[root@hexindai-c12-120 ~]# cd /opt/cloudera-manager/cm-5.15.1/share/cmf/schema/
[root@hexindai-c12-120 schema]# ls
mysql  oracle  postgresql  scm_database_functions.sh  scm_prepare_database.sh
[root@hexindai-c12-120 schema]# ./scm_prepare_database.sh mysql cdh -h 172.20.102.120 -ucdh -p --scm-host hexindai-c12-120 scm scm scm                
Enter database password: 
JAVA_HOME=/export/servers/jdk1.8.0_211
Verifying that we can write to /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-server
Sat Jul 06 13:39:36 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Creating SCM configuration file in /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-server
Executing:  /export/servers/jdk1.8.0_211/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera-manager/cm-5.15.1/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Sat Jul 06 13:39:36 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2019-07-06 13:39:37,010 [main] INFO  com.cloudera.enterprise.dbutil.DbCommandExecutor  - Successfully connected to database.
All done, your SCM database is configured correctly!
[root@hexindai-c12-120 schema]

[root@hexindai-c12-120 schema]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 29
Server version: 5.7.26-log MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cdh                |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

mysql> use cdh;
Database changed
mysql> show tables;
Empty set (0.00 sec)

mysql> 
mysql> quit
Bye
[root@hexindai-c12-120 schema]#

三、制作CDH本地源

1、server创建parcel-repo目录

[root@hexindai-c12-120 schema]# mkdir -p /opt/cloudera/parcel-repo -p #server端创建parcel-repo目录，用于存放离线CDH文件
[root@hexindai-c12-120 schema]# chown -R cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/
[root@hexindai-c12-120 schema]# ll -d /opt/cloudera/parcel-repo/
drwxr-xr-x. 2 cloudera-scm cloudera-scm 6 Jul  6 13:46 /opt/cloudera/parcel-repo/
[root@hexindai-c12-120 schema]#

2、Agent端创建parcels目录

[root@hexindai-c12-120 ~]# mkdir -p /opt/cloudera/parcels
[root@hexindai-c12-120 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
[root@hexindai-c12-121 ~]# mkdir -p /opt/cloudera/parcels
You have new mail in /var/spool/mail/root
[root@hexindai-c12-121 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
[root@Hexindai-C12-122 ~]# mkdir -p /opt/cloudera/parcels
You have new mail in /var/spool/mail/root
[root@Hexindai-C12-122 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
[root@Hexindai-C12-122 ~]# 
[root@Hexindai-C12-123 ~]# mkdir -p /opt/cloudera/parcels
[root@Hexindai-C12-123 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
[root@Hexindai-C12-123 ~]# 
#各agent节点都创建

3、制作CDH本地源

[root@hexindai-c12-120 ~]# cd /export/software/cm/
[root@hexindai-c12-120 cm]# ll
total 2889720
-rw-r--r--. 1 root root 2120090032 Jul  5 09:24 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel
-rw-r--r--. 1 root root         41 Jul  5 09:24 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1
-rw-r--r--. 1 root root  838894986 Jul  4 17:46 cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz
-rw-r--r--. 1 root root      73767 Aug 24  2018 manifest.json
[root@hexindai-c12-120 cm]# mv CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel /opt/cloudera/parcel-repo/
[root@hexindai-c12-120 cm]# mv CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha
[root@hexindai-c12-120 cm]# mv manifest.json /opt/cloudera/parcel-repo/
[root@hexindai-c12-120 cm]# ll /opt/cloudera/parcel-repo/
total 2070484
-rw-r--r--. 1 root root 2120090032 Jul  5 09:24 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel
-rw-r--r--. 1 root root         41 Jul  5 09:24 CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha
-rw-r--r--. 1 root root      73767 Aug 24  2018 manifest.json
[root@hexindai-c12-120 cm]# 
提示：
　　如果没下载到“CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha”文件，可以找到“manifest.json”文件中“parcelName”："CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel"对应的“hash"：
"deff00898e410a34cf0a1e66c5dbe87546608f0c"复制到该文件即可。这个方法也适用于其他的版本！

四、启动CM的Server,Agent端

1、启动Cloudera Manager Server端

[root@hexindai-c12-120 cm]# cd /opt/cloudera-manager/cm-5.15.1/etc/init.d/
[root@hexindai-c12-120 init.d]# ll
total 32
-rwxr-xr-x. 1 1106 4001 8871 Jul 31  2018 cloudera-scm-agent
-rwxr-xr-x. 1 1106 4001 8417 Jul 31  2018 cloudera-scm-server
-rwxr-xr-x. 1 1106 4001 4444 Jul 31  2018 cloudera-scm-server-db
[root@hexindai-c12-120 init.d]# ./cloudera-scm-server start
./cloudera-scm-server: line 109: pstree: command not found
Starting cloudera-scm-server:                              [  OK  ]
#Centos7下默认并没有安装pstree，所以会有pstree:command not found
#安装方式使用如下命令：
[root@hexindai-c12-120 init.d]# yum -y install psmisc
[root@hexindai-c12-120 init.d]# ./cloudera-scm-server status
cloudera-scm-server (pid  15861) is running...
[root@hexindai-c12-120 init.d]#

2、我们需要观察Server的日志文件（ tail -fn 50 /opt/cloudera-manager/cm-5.15.1/log/cloudera-scm-server/cloudera-scm-server.log ），若出现以下内容说明启动成功！

3、启动Cloudera Manager Agent端

[root@hexindai-c12-120 init.d]# /opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]
[root@hexindai-c12-120 init.d]# /opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-agent status
cloudera-scm-agent (pid  16614) is running...
[root@hexindai-c12-120 init.d]#
#各agent节点都需要启动

4、验证是否可以登录成功，访问WebUI界面，效果图如下：

我们安装CM的过程通过WebUI的安装向导来进行安装，推荐使用谷歌浏览器，，如果出现以下界面，恭喜你CM部署成功！

五、基于CM的WebUI部署CDH集群

    详情请参考：https://www.cnblogs.com/wtnyihg/articles/11150419.html

六、Cloudera Manager 配置HA简介

    实验环境我们部署了单机的Cloudera Manager，事实上，CM也是可以做HA的，详情请参考：https://www.cloudera.com/documentation/enterprise/latest/topics/admin_cm_ha_overview.html
    Cloudera Manager独立于CDH而存在，Cloudera Manager挂掉并不会影响CDH的集群工作，只需要及时恢复即可，所以做HA的必要性也不是很大。

七、基于Yum方式安装（离线安装）

    在某些企业的生产环境部署集群，可能不允许访问公网，这时可选一台主机来架设一个内外镜像。具体操作可参考：https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_parcel_repo.html。
    官网的这种部署，也会有文档进行记录，请参考如下连接：

https://www.cnblogs.com/yinzhengjie/articles/11019525.html

posted @ 2019-07-08 15:51 冠彭阅读(294) 评论(0) 收藏举报

刷新页面返回顶部

wtnyihg

大数据集群之基于二进制安装包方式部署Cloudera Manager(离线安装）