大数据基础之CM5（Cloudera Manager）+CDH5离线安装

CM/CDH 5.16.1

CM官方：https://www.cloudera.com/products/product-components/cloudera-manager.html

CDH官方：https://www.cloudera.com/products/open-source/apache-hadoop/key-cdh-components.html

CDH5.16.1 软件列表：https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_516.html

 

一 简介

1 CDH

CDH is Cloudera’s 100% open source platform distribution, including Apache Hadoop and built specifically to meet enterprise demands. CDH delivers everything you need for enterprise use right out of the box. By integrating Hadoop with more than a dozen other critical open source projects, Cloudera has created a functionally advanced system that helps you perform end-to-end Big Data workflows.

2 Cloudera Manager

Cloudera Manager is the industry’s trusted tool for managing Hadoop in production.

 

二 安装 

1 准备jdk

1）卸载可能的openjdk

# rpm -qa|grep openjdk
java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64
java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64
# rpm -e --nodeps java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64 java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64

2）安装oracle jdk

# wget https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
# rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
# ln -s /usr/java/jdk1.8.0_141-cloudera/bin/java /usr/bin/java

2 创建用户

# useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3 下载

# wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
# mkdir /opt/cloudera-manager
# tar xvzf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager

# mkdir -p /opt/cloudera/parcels
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

4 准备server

1）确保文件存在：/usr/share/java/mysql-connector-java.jar

# cp mysql-connector-java.jar /usr/share/java/mysql-connector-java.jar

or

# yum install mysql-connector-java

2）初始化mysql

mysql> create database scm;
mysql> grant all privileges on scm.* to scm@'%' identified by 'scm';

# /opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh -h 192.168.0.1 mysql scm scm scm

3）其他

# mkdir /var/lib/cloudera-scm-server
# chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server

4）下载cdh

# wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
# wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
# wget http://archive.cloudera.com/cdh5/parcels/latest/manifest.json
# mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha

# mkdir -p /opt/cloudera/parcel-repo
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
# mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo

5 启动server

# /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start

日志位于 /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server/

6 逐个启动agent

# vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini
server_host=$server_ip

# /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

日志位于 /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-agent/

7 开机启动

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent

修改

CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
->
CMF_DEFAULTS=${CMF_DEFAULTS:-/opt/cloudera-manager/cm-5.16.1/etc/default}

否则会报错

File not found: /usr/sbin/cmf-agent

修改后同步到所有节点，然后配置开机启动

# ln -s /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
# chkconfig --add cloudera-scm-agent
# chkconfig cloudera-scm-agent on
# chkconfig --list cloudera-scm-agent

# ln -s /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
# chkconfig --add cloudera-scm-server
# chkconfig cloudera-scm-server on
# chkconfig --list cloudera-scm-server

8 访问

http://$server_ip:7180 用户名密码为 admin/admin 进入向导页面，一步步配置完成；

 

 

当使用一些功能时需要安装新的库，具体如下：

比如Activity Monitor需要

mysql> create database amon;
mysql> grant all privileges on amon.* to amon@'%' identified by 'amon';

 

可能遇到的问题

1）如果看不到下载版本的cdh，检查CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1是否修改为CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha

2）正在安装选定 Parcel
报错：主机运行状况不良。

查看agent日志：

# tail -f /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-agent/cloudera-scm-agent.log
[15/Feb/2019 14:44:28 +0000] 7241 MainThread agent ERROR Error, CM server guid updated, expected b6f92bed-e8fc-45e0-b627-fead8cd288e6, received 1db60b99-1f0c-4226-bd2f-a9baa568cf96

解决方法

# /bin/rm /opt/cloudera-manager/cm-5.16.1/lib/cloudera-scm-agent/cm_guid
# /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent restart

安装parcel正常

3）如果安装过程中出现任何问题，将server和agent停掉，然后重复‘初始化mysql’过程，然后重启server和agent即可；

4）安装组件比如hdfs，部署客户端配置需要依赖perl

# yum install perl perl-devel

 

参考：https://www.cloudera.com/documentation/enterprise/5-16-x/topics/installation.html