Cloudera Manager安装之利用parcels方式安装3或4节点集群（包含最新稳定版本或指定版本的安装）（添加服务）（CentOS6.5）（五）

参考博客

Cloudera Manager安装之利用parcels方式安装单节点集群

Cloudera Manager安装之Cloudera Manager 5.3.X安装（三）（tar方式、rpm方式和yum方式）

　　ClouderManager官网的安装历史稳定版

https://www.cloudera.com/documentation/manager/5-1-x/Cloudera-Manager-Installation-Guide/cm5ig_install_older_cm5.html

说在前面的话（看清楚就好！！！）

　　我这篇博客，是两种方式都使用过：（在/etc/yum.repos.d下的）

[hadoop@clouderamanager01 yum.repos.d]$ sudo vi cloudera-manager.repo
[hadoop@clouderamanager02 yum.repos.d]$ sudo vi cloudera-manager.repo
[hadoop@clouderamanager03 yum.repos.d]$ sudo vi cloudera-manager.repo

即（博主我推荐用这种！！！）（但是不需）

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/
gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1

或者（但是，需要自定义存储库）

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=https://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
enabled = 1
gpgcheck = 1

　　我这里是，使用CM 5.3.x利用parcels方式集群安装CDH 5.3.x的3节点集群！（开始开工）

下载以下三个文件就好：

http://archive.cloudera.com/cdh5/parcels/5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
http://archive.cloudera.com/cdh5/parcels/5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.3.6/manifest.json

　　以下是clouderamanager01这台机器上（主节点）

[hadoop@clouderaManager01 softwares]$ pwd
/opt/softwares
[hadoop@clouderaManager01 softwares]$ ll
total 4
drwxr-xr-x 2 hadoop hadoop 4096 Apr  8 13:49 cm-5.3.6
[hadoop@clouderaManager01 softwares]$ sudo mkdir cdh-5.3.6
[hadoop@clouderaManager01 softwares]$ ll
total 8
drwxr-xr-x 2 root   root   4096 Apr  8 20:00 cdh-5.3.6
drwxr-xr-x 2 hadoop hadoop 4096 Apr  8 13:49 cm-5.3.6
[hadoop@clouderaManager01 softwares]$ cd cdh-5.3.6/
[hadoop@clouderaManager01 cdh-5.3.6]$ pwd
/opt/softwares/cdh-5.3.6
[hadoop@clouderaManager01 cdh-5.3.6]$ ll
total 0
[hadoop@clouderaManager01 cdh-5.3.6]$ sudo rz

[hadoop@clouderaManager01 cdh-5.3.6]$ sudo rz

[hadoop@clouderaManager01 cdh-5.3.6]$ sudo rz

[hadoop@clouderaManager01 cdh-5.3.6]$ ll
total 1473900
-rw-r--r-- 1 root root 1509217191 Apr  7 19:38 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html
-rw-r--r-- 1 root root         41 Apr  7 19:24 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1
-rw-r--r-- 1 root root      42661 Apr  7 19:24 manifest.json
[hadoop@clouderaManager01 cdh-5.3.6]$

　　以下是clouderamanager01这台机器上安装之后，默认产生的（在这里，你看看就好）

[hadoop@clouderaManager01 opt]$ pwd
/opt
[hadoop@clouderaManager01 opt]$ ll
total 16
drwxr-xr-x  4 cloudera-scm cloudera-scm 4096 Apr  8 18:31 cloudera
drwxr-xr-x  2 root         root         4096 Apr  8 18:13 cloudera-manager
drwxr-xr-x. 2 root         root         4096 Nov 22  2013 rh
drwxr-xr-x  4 hadoop       hadoop       4096 Apr  8 20:00 softwares
[hadoop@clouderaManager01 opt]$ cd cloudera
[hadoop@clouderaManager01 cloudera]$ ll
total 8
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jul 30  2015 csd
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jul 30  2015 parcel-repo
[hadoop@clouderaManager01 cloudera]$ cd parcel-repo/
[hadoop@clouderaManager01 parcel-repo]$ pwd
/opt/cloudera/parcel-repo
[hadoop@clouderaManager01 parcel-repo]$ ll
total 0
[hadoop@clouderaManager01 parcel-repo]$

　　或者，有时候，安装CM完成后，出现：

以下是clouderamanager01这台机器上安装之后，默认产生的（在这里，你看看就好）

[hadoop@clouderamanager01 opt]$ pwd
/opt
[hadoop@clouderamanager01 opt]$ ll
total 8
drwxr-xr-x  4 cloudera-scm cloudera-scm 4096 Apr  8 22:48 cloudera
drwxr-xr-x. 3 hadoop       hadoop       4096 Apr  8 11:20 softwares
[hadoop@clouderamanager01 opt]$ cd cloudera/
[hadoop@clouderamanager01 cloudera]$ ll
total 8
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 csd
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 parcel-repo
[hadoop@clouderamanager01 cloudera]$

　　上传我们刚之前下载好的CDH离线包

[hadoop@clouderamanager01 softwares]$ pwd
/opt/softwares
[hadoop@clouderamanager01 softwares]$ ll
total 4
drwxr-xr-x. 2 hadoop hadoop 4096 Apr  8 11:20 cm-5.3.6
[hadoop@clouderamanager01 softwares]$ sudo mkdir cdh-5.3.6
[hadoop@clouderamanager01 softwares]$ ll
total 8
drwxr-xr-x  2 root   root   4096 Apr  8 23:04 cdh-5.3.6
drwxr-xr-x. 2 hadoop hadoop 4096 Apr  8 11:20 cm-5.3.6
[hadoop@clouderamanager01 softwares]$ cd cdh-5.3.6/
[hadoop@clouderamanager01 cdh-5.3.6]$ pwd
/opt/softwares/cdh-5.3.6
[hadoop@clouderamanager01 cdh-5.3.6]$ ll
total 0
[hadoop@clouderamanager01 cdh-5.3.6]$ sudo rz

[hadoop@clouderamanager01 cdh-5.3.6]$ sudo rz

[hadoop@clouderamanager01 cdh-5.3.6]$ sudo rz

[hadoop@clouderamanager01 cdh-5.3.6]$ ll
total 1473900
-rw-r--r-- 1 root root 1509217191 Apr  8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html
-rw-r--r-- 1 root root         41 Apr  8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1
-rw-r--r-- 1 root root      42661 Apr  8 12:33 manifest.json
[hadoop@clouderamanager01 cdh-5.3.6]$

　　注意，是，CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel。而不是CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html，也不是CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel！！！（这个离线包很容易在下载的时候，自己出现后缀的变名。）（若是你下载的时候，没出现变名是更好啦！）

　　需要改CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1的名称和CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html的名称

[hadoop@clouderamanager01 cdh-5.3.6]$ pwd
/opt/softwares/cdh-5.3.6
[hadoop@clouderamanager01 cdh-5.3.6]$ ll
total 1473900
-rw-r--r-- 1 root root 1509217191 Apr  8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html
-rw-r--r-- 1 root root         41 Apr  8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1
-rw-r--r-- 1 root root      42661 Apr  8 12:33 manifest.json
[hadoop@clouderamanager01 cdh-5.3.6]$ sudo mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
[hadoop@clouderamanager01 cdh-5.3.6]$ sudo mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
[hadoop@clouderamanager01 cdh-5.3.6]$ ll
total 1473900
-rw-r--r-- 1 root root 1509217191 Apr  8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
-rw-r--r-- 1 root root         41 Apr  8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
-rw-r--r-- 1 root root      42661 Apr  8 12:33 manifest.json
[hadoop@clouderamanager01 cdh-5.3.6]$

　　一定要改成，CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel和CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha。

[hadoop@clouderamanager01 cdh-5.3.6]$ pwd
/opt/softwares/cdh-5.3.6
[hadoop@clouderamanager01 cdh-5.3.6]$ ll
total 1473900
-rw-r--r-- 1 root root 1509217191 Apr  8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
-rw-r--r-- 1 root root         41 Apr  8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
-rw-r--r-- 1 root root      42661 Apr  8 12:33 manifest.json
[hadoop@clouderamanager01 cdh-5.3.6]$ more CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
9b0c758c15792be3d7da7597b376668c09a32e2d
[hadoop@clouderamanager01 cdh-5.3.6]$

　　　　查看manifest.json

http://archive.cloudera.com/cdh5/parcels/5.3.6/manifest.json

　　在clouderamanager02和clouderamanager03机器上，进入/etc/yum.repos.d/

[hadoop@clouderamanager02 ~]$ cd /etc/yum.repos.d/
[hadoop@clouderamanager02 yum.repos.d]$ pwd
/etc/yum.repos.d
[hadoop@clouderamanager02 yum.repos.d]$ ll
total 16
-rw-r--r--. 1 root root 1926 Nov 27  2013 CentOS-Base.repo
-rw-r--r--. 1 root root  638 Nov 27  2013 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  630 Nov 27  2013 CentOS-Media.repo
-rw-r--r--. 1 root root 3664 Nov 27  2013 CentOS-Vault.repo
[hadoop@clouderamanager02 yum.repos.d]$ sudo vi cloudera-manager.repo

[cloudera-manager]
name=Cloudera Manager
baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
gpgcheck=0

或者

[cloudera-manager]
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/
gpgcheck=1

　　同理，机器clouderamanager03也要去做。

关于/opt/cloudera/parcel-repo和/opt/cloudera/parcel（必须弄懂，别怕）

　　192.168.80.148 clouderamanager01　（部署ClouderaManager-server和Mirror server）（部署Agent）　　　　（分配1G，我这里只是体验过程，当然你可以分配更多或更小)　

　　192.168.80.149 clouderamanager02 （部署Agent）　　　　　　　　　　　　　　　　　　　　　（分配1G，我这里只是体验过程，当然你可以分配更多或更小)

　　192.168.80.150 clouderamanager03 （部署Agent）

　　部署ClouderaManager-server和Mirror server的机器，默认是存储在/opt/cloudera/parcel-repo。

　　部署Agent的机器，默认是存储在/opt/cloudera/parcel。

　　即，这也是为什么，我在下面这篇博客里写到。

Cloudera Manager安装之Cloudera Manager 5.3.X安装（三）（tar方式、rpm方式和yum方式）

　　博客里，提及到的tar方式来安装。

　　进入，CM安装的机器，自动生成的默认/opt/cloudera/parcel-repo目录下（在clouderamanager01主节点）

[hadoop@clouderamanager01 opt]$ pwd
/opt
[hadoop@clouderamanager01 opt]$ ll
total 8
drwxr-xr-x  4 cloudera-scm cloudera-scm 4096 Apr  8 22:48 cloudera
drwxr-xr-x. 4 hadoop       hadoop       4096 Apr  8 23:04 softwares
[hadoop@clouderamanager01 opt]$ cd cloudera/
[hadoop@clouderamanager01 cloudera]$ pwd
/opt/cloudera
[hadoop@clouderamanager01 cloudera]$ ll
total 8
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 csd
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 parcel-repo
[hadoop@clouderamanager01 cloudera]$ cd parcel-repo/
[hadoop@clouderamanager01 parcel-repo]$ pwd
/opt/cloudera/parcel-repo
[hadoop@clouderamanager01 parcel-repo]$ ll
total 0
[hadoop@clouderamanager01 parcel-repo]$

　　将我们之前上传在/opt/softwares/cdh-5.3.6（其实这个用作为备份，万一以后丢失了再可以重新安装。免得再下载）的CDH离线安装包，

移动到这个/opt/cloudera/parcel-repo目录上去。

[hadoop@clouderamanager01 parcel-repo]$ pwd
/opt/cloudera/parcel-repo
[hadoop@clouderamanager01 parcel-repo]$ ll
total 0
[hadoop@clouderamanager01 parcel-repo]$ sudo cp /opt/softwares/cdh-5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel .
[hadoop@clouderamanager01 parcel-repo]$ sudo cp /opt/softwares/cdh-5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha .
[hadoop@clouderamanager01 parcel-repo]$ ll
total 1473856
-rw-r--r-- 1 root root 1509217191 Apr  8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
-rw-r--r-- 1 root root         41 Apr  8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
[hadoop@clouderamanager01 parcel-repo]$ sudo cp /opt/softwares/cdh-5.3.6/manifest.json .
[hadoop@clouderamanager01 parcel-repo]$ ll
total 1473900
-rw-r--r-- 1 root root 1509217191 Apr  8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
-rw-r--r-- 1 root root         41 Apr  8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
-rw-r--r-- 1 root root      42661 Apr  8 23:35 manifest.json
[hadoop@clouderamanager01 parcel-repo]$

[hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server status
cloudera-scm-server (pid  1888) is running...
[hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server restart
Stopping cloudera-scm-server:                              [  OK  ]
Starting cloudera-scm-server:                              [  OK  ]
[hadoop@clouderamanager01 parcel-repo]$

　　等上几分钟，

　　如果Clouder Manager的7180端口，没起来，怎么办？

[hadoop@clouderamanager01 parcel-repo]$  sudo netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      901/sshd            
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      985/master          
tcp        0      0 0.0.0.0:7432                0.0.0.0:*                   LISTEN      1850/postgres       
tcp        0      0 :::80                       :::*                        LISTEN      995/httpd           
tcp        0      0 :::22                       :::*                        LISTEN      901/sshd            
tcp        0      0 ::1:25                      :::*                        LISTEN      985/master          
tcp        0      0 :::7432                     :::*                        LISTEN      1850/postgres       
[hadoop@clouderamanager01 parcel-repo]$ sudo reboot

Broadcast message from hadoop@clouderamanager01
    (/dev/pts/0) at 23:45 ...

The system is going down for reboot NOW!
[hadoop@clouderamanager01 parcel-repo]$ 
Connection closed by foreign host.

Disconnected from remote host(clouderamanager01) at 15:46:15.

Type `help' to learn how to use Xshell prompt.

　　或者

　　改为本地的，进入/etc/yum.repos.d

　　即

sudo chmod 755  clouder-manager.repo

[hadoop@clouderamanager01 yum.repos.d]$ pwd
/etc/yum.repos.d
[hadoop@clouderamanager01 yum.repos.d]$ ll
total 20
-rw-r--r--. 1 root root 1926 Nov 27  2013 CentOS-Base.repo
-rw-r--r--. 1 root root  638 Nov 27  2013 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  630 Nov 27  2013 CentOS-Media.repo
-rw-r--r--. 1 root root 3664 Nov 27  2013 CentOS-Vault.repo
-rw-r--r--  1 root root  202 Apr  9 01:59 cloudera-manager.repo
[hadoop@clouderamanager01 yum.repos.d]$ sudo chmod 755 cloudera-manager.repo
[hadoop@clouderamanager01 yum.repos.d]$ ll
total 20
-rw-r--r--. 1 root root 1926 Nov 27  2013 CentOS-Base.repo
-rw-r--r--. 1 root root  638 Nov 27  2013 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  630 Nov 27  2013 CentOS-Media.repo
-rw-r--r--. 1 root root 3664 Nov 27  2013 CentOS-Vault.repo
-rwxr-xr-x  1 root root  202 Apr  9 01:59 cloudera-manager.repo
[hadoop@clouderamanager01 yum.repos.d]$

　　将

[cloudera-manager]
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/
gpgcheck = 1

　　改为

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/   
gpgcheck = 1

　　注意，此时，所有的机器（比如我这里是clouderamanager01、clouderamanager02和clouderamanager03）都要更改为

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
enabled =1 
gpgcheck = 1

　　于是，将所有机器的/etc/yum.repos.d/clouder-manager.repo改好为本地的之后，再次刷新就好

clouderamanager[01-03]

　　有时候会是如下，　　

　　这是，因为。我之前前面3台机器都没有启动服务，所以会是如下。三个自动会勾选。

 http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/

在安装过程中，出现如下问题

[hadoop@clouderaManager01 html]$ pwd
/var/www/html
[hadoop@clouderaManager01 html]$ ll
total 4
drwxr-xr-x 3 root root 4096 Apr  8 20:37 cm5
[hadoop@clouderaManager01 html]$ sudo mkdir -p cm5/redhat/6/x86_64/cm/5.3.6/repodata
[hadoop@clouderaManager01 html]$ ll
total 4
drwxr-xr-x 3 root root 4096 Apr  8 20:37 cm5
[hadoop@clouderaManager01 html]$ cd cm5/redhat/6/x86_64/cm/5.3.6/repodata/
[hadoop@clouderaManager01 repodata]$ pwd
/var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/repodata
[hadoop@clouderaManager01 repodata]$ ll
total 0
[hadoop@clouderaManager01 repodata]$

[hadoop@clouderamanager01 repodata]$ pwd
/var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/repodata
[hadoop@clouderamanager01 repodata]$ lll
-bash: lll: command not found
[hadoop@clouderamanager01 repodata]$ ll
total 0
[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ sudo rz

[hadoop@clouderamanager01 repodata]$ ll
total 124
-rw-r--r-- 1 root root 91122 Apr  7 22:36 filelists.xml.gz
-rw-r--r-- 1 root root   198 Apr  7 22:36 filelists.xml.gz.asc
-rw-r--r-- 1 root root   570 Apr  7 22:36 other.xml.gz
-rw-r--r-- 1 root root   198 Apr  7 22:36 other.xml.gz.asc
-rw-r--r-- 1 root root  4208 Apr  7 22:37 primary.xml.gz
-rw-r--r-- 1 root root   198 Apr  7 22:37 primary.xml.gz.asc
-rw-r--r-- 1 root root   951 Apr  7 22:37 repomd.xml
-rw-r--r-- 1 root root   198 Apr  7 22:37 repomd.xml.asc
[hadoop@clouderamanager01 repodata]$

　　如果还是没有得到解决的话，则

[hadoop@clouderamanager01 yum.repos.d]$ sudo vi cloudera-manager.repo

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
enabled =1 
gpgcheck = 1

　　这里。所有的节点都要一样的。如果其下有其他的，删除（比如cloudera-manager.repo.~!或其他的cloudera-manager.repo.repnew等）

　　以及重启下

[hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server status
cloudera-scm-server (pid  1888) is running...
[hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server restart
Stopping cloudera-scm-server:                              [  OK  ]
Starting cloudera-scm-server:                              [  OK  ]
[hadoop@clouderamanager01 parcel-repo]$

　　等上几分钟。

　　如果还是没有解决的话，说明那个文件cloudera-manager-agent-5.3.6-1.cm536.p0.244.el6.x86_64.rpm已经损坏了。重新下载它，再上传。

　　然后，再重启

sudo service cloudera-scm-server restart

　　就可以解决问题了。

　　如果还是没解决的话，那就换成在线的吧！（哎）

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/
gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1

　　然后，接着下面

　　重复，执行上面的过程。这里省略，.....

直至如下界面.....

　　如果你的网速慢的话，则会出现，

　　比如我还遇到过这样的一种情况，也就是说，clouderamanager01先安装成功，然后再次执行上面的重头开始，再来clouderamanager02和clouderamanager03安装成功！

安装Parcel到所有主机上（我这里是clouderaManager01、clouderaManager02和clouderaManager03这台机器上）
　　因为Parcel文件已经提前下载好了，第一个进度条会直接显示已下载，然后开始分配。这个过程实际上就是将Parcel内的软件解压缩安装到各个主机上的过程。耐心等待，时间会比较长，我安装3台主机（即在clouderaManager01、02和03上，安装3节点集群）大概是需要30分钟时间。（这里看个人的网速）

　　其实啊，这里分配就是解压。

主机检查
　　检查就可以了，建议不要跳过。静静地等一分钟多点的时间

　　检查结果如下：

http://clouderamanager01:7180/cmf/express-wizard/wizard#step=hostInspectorStep

　　因为我这里，之前是在clouderaManager01、clouderaManager02、clouderaManager03和clouderaManager04安装过。

然后，后面，再clouderamanager01、clouderamanager02和clouderamanager03。

　　所以，会已经有了记录的痕迹。（当然，若是你第一次安装的话，就不会有这个问题出现的哈！）

　 Cloudera Manager安装之Cloudera Manager安装前准备(CentOS6.5）（一）

　　永久修改，我们直接修改配置文件，我们直接修改/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=clouderaManager01

　　对应三台都要修改。我这里是自己，当时失误！主机名弄错大小写字母M了。

　　然后

sudo reboot

　　　这个问题怎么解决呢？

　　说白了swap就是交换区，当内存容不下这些数据时，就移动到磁盘那去。由于我们是大数据，不希望移动到磁盘。

[hadoop@clouderaManager01 ~]$ sudo free -m 
             total       used       free     shared    buffers     cached
Mem:           980        917         63          0          1         32
-/+ buffers/cache:        883         96
Swap:         1999        552       1447
[hadoop@clouderaManager01 ~]$ sudo cat /proc/sys/vm/swappiness
60
[hadoop@clouderaManager01 ~]$ sudo sysctl -w vm.swappiness=10
vm.swappiness = 10
[hadoop@clouderaManager01 ~]$

　　clouderaManager01和clouderaManager02和clouderaManager03都要执行。

　　这个怎么来解决呢？是我们的Linux版本的bug问题。

[hadoop@clouderaManager01 ~]$ su root
Password: 
[root@clouderaManager01 hadoop]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
[root@clouderaManager01 hadoop]# vi /etc/rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local


echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

　　clouderaManager01和clouderaManager02和clouderaManager03都要执行。

　　带你认识下安装成功之后的目录结构

[root@clouderaManager01 cloudera]# pwd
/opt/cloudera
[root@clouderaManager01 cloudera]# ll
total 16
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 csd
drwxr-xr-x 2 root         root         4096 Apr  9 04:14 parcel-cache
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Apr  8 23:52 parcel-repo
drwxr-xr-x 4 root         root         4096 Apr  9 04:33 parcels
[root@clouderaManager01 cloudera]# cd parcels/
[root@clouderaManager01 parcels]# pwd
/opt/cloudera/parcels
[root@clouderaManager01 parcels]# ll
total 4
lrwxrwxrwx  1 root root   26 Apr  9 04:33 CDH -> CDH-5.3.6-1.cdh5.3.6.p0.11
drwxr-xr-x 11 root root 4096 Jul 29  2015 CDH-5.3.6-1.cdh5.3.6.p0.11
[root@clouderaManager01 parcels]# cd CDH
[root@clouderaManager01 CDH]# pwd
/opt/cloudera/parcels/CDH
[root@clouderaManager01 CDH]# ls
bin  etc  include  jars  lib  lib64  libexec  meta  share
[root@clouderaManager01 CDH]#

[root@clouderaManager01 CDH]# pwd
/opt/cloudera/parcels/CDH
[root@clouderaManager01 CDH]# ll
total 80
drwxr-xr-x  2 root root  4096 Jul 29  2015 bin
drwxr-xr-x 26 root root  4096 Jul 29  2015 etc
drwxr-xr-x  4 root root  4096 Jul 29  2015 include
drwxr-xr-x  2 root root 49152 Jul 29  2015 jars
drwxr-xr-x 36 root root  4096 Jul 29  2015 lib
drwxr-xr-x  2 root root  4096 Jul 29  2015 lib64
drwxr-xr-x  3 root root  4096 Jul 29  2015 libexec
drwxr-xr-x  2 root root  4096 Jul 29  2015 meta
drwxr-xr-x  3 root root  4096 Jul 29  2015 share
[root@clouderaManager01 CDH]# cd bin/
[root@clouderaManager01 bin]# ls
avro-tools  hadoop                hcat          kite-dataset  oozie          solrctl         sqoop2-server            sqoop-help               sqoop-merge      zookeeper-client
beeline     hadoop-0.20           hdfs          llama         oozie-setup    spark-executor  sqoop2-tool              sqoop-import             sqoop-metastore  zookeeper-server
catalogd    hadoop-fuse-dfs       hive          llamaadmin    parquet-tools  spark-shell     sqoop-codegen            sqoop-import-all-tables  sqoop-version    zookeeper-server-cleanup
cli_mt      hadoop-fuse-dfs.orig  hiveserver2   load_gen      pig            spark-submit    sqoop-create-hive-table  sqoop-job                statestored      zookeeper-server-initialize
cli_st      hbase                 impalad       mahout        pyspark        sqoop           sqoop-eval               sqoop-list-databases     whirr
flume-ng    hbase-indexer         impala-shell  mapred        sentry         sqoop2          sqoop-export             sqoop-list-tables        yarn
[root@clouderaManager01 bin]#

安装完成之后，再安装所需的服务（其实，这一步，若你的内存和空间足够，最好选择所有服务）

　　其实啊，这步是选择集群中要安装的CDH服务，为了将来新增服务麻烦，我们这里选择所有服务。其中有暂时用不到的服务，我们可以在界面中停止服务，这样可以节省内存资源

　　这里，添加Cloudera Management Service服务

　　然后，关于安装其他的，如Zookeeper、Hive、HBase和Spakr、Hadoop等。这些。我就不赘述了。

　　因为，这是，我在自己的电脑虚拟机里搭建的。过卡和内存不够，以下不贴图和不写了。

安装完毕后点继续

　　贴图，这里不写了。

角色分配
　　分配各个节点的角色

数据库设置
　　CDH默认内嵌了一个数据库，如果你需要MySQL或者Oracle请选择使用自定义数据库，

　　我这里，先点击测试连接，即选择CDH默认的内嵌数据库成功后点继续。

审核更改
　　这里全部默认就可以了，直接继续。

安装完成，初始化
　　到这里集群已经安装部署完成了，CDH会自动对你选择的服务进行初始化设置，你也可以稍后自行开启。

初始化完成后，将进入主页面

常见错误

[hadoop@clouderaManager01 parcel-repo]$ sudo service cloudera-scm-server restart

./cloudera-scm-server: line 109: pstree: command not found
Starting cloudera-scm-server:                              [  OK  ]

　　主要因为没有安装pstree包。

解决办法：

[hadoop@clouderaManager01 parcel-repo]$ sudo yum install epel-release (安装yum 源)
[hadoop@clouderaManager01 parcel-repo]$ sudo yum provides pstree (找到提供pstree命令的包)
[hadoop@clouderaManager01 parcel-repo]$ sudo yum install psmisc (安装对应的包)


然后，
[hadoop@clouderaManager01 parcel-repo]$ sudo service cloudera-scm-server restart
Stopping cloudera-scm-server:                              [  OK  ]
Starting cloudera-scm-server:                              [  OK  ]

Detecting Cloudera Manager Server...
    Detecting Cloudera Manager Server...
    BEGIN host -t PTR 192.168.1.198
    198.1.168.192.in-addr.arpa domain name pointer localhost.
    END (0)
    using localhost as scm server hostname
    BEGIN which python
    /usr/bin/python
    END (0)
    BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' localhost 7182
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "<string>", line 1, in connect
    socket.error: [Errno 111] Connection refused
    END (1)
    could not contact scm server at localhost:7182, giving up
    waiting for rollback request

解决办法：

　　在失败的主机上，终端执行

sudo mv /usr/bin/host /usr/bin/host.bak

　　然后重试。

　　安装过程一直卡在"正在获取安装锁"状态界面

　　解决办法：
在卡住的主机上，删除/tmp文件夹中的所有scm_prepare_node.开头的文件，清空yum缓存，然后重启主机。

sudo cd /tmp 
sudo rm -rf  scm_prepare_node.*
sudo yum clean all
sudo reboot

安装完毕后NameNode格式化失败，报以下错误

IncorrectVersionException: Unexpected version of storage directory /data/dfs

　　解决办法：

　　删除NameNode节点的/data/dfs文件夹，重新添加主机。

如何更改节点ip

　　解决办法：
　　重新设置每台主机的hosts文件，完成后进入主机界面，点击重新运行升级向导，再走一遍之前的流程即可。

参考链接

http://www.mamicode.com/info-detail-1596233.html

http://www.jianshu.com/p/57179e03795f

http://www.aboutyun.com/thread-20741-1-1.html

扩展

　　参考

Cloudera Manager安装之利用parcels方式（在线或离线）安装3或4节点集群（包含最新稳定版本或指定版本的安装）（添加服务）（Ubuntu14.04）（五）

　　最好，在这之前，先将

注意，在此之前。

　　cmbigdata1机器上，则需要执行

bigdata@cmbigdata1:~$ sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm_password

[sudo] password for bigdata: 
Sorry, try again.
[sudo] password for bigdata: 
JAVA_HOME=/usr/local/jdk/jdk1.8.0_60

Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.7.0_80/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly! bigdata@ubuntucmbigdata1:~$ sudo service cloudera-scm-server start cloudera-scm-server is already running bigdata@cmbigdata1:~$

bigdata@cmbigdata1:~$ sudo service cloudera-scm-agent start
cloudera-scm-agent is already running
bigdata@cmbigdata1:~$

　　然后，在cmbigdata2、cmbigdata3、cmbigdata4上

bigdata@cmbigdata2:~$ sudo service cloudera-scm-agent start
cloudera-scm-agent is already running
bigdata@cmbigdata1:~$ 



bigdata@cmbigdata3:~$ sudo service cloudera-scm-agent start
cloudera-scm-agent is already running
bigdata@cmbigdata1:~$ 





bigdata@cmbigdata4:~$ sudo service cloudera-scm-agent start
cloudera-scm-agent is already running
bigdata@cmbigdata1:~$

　　因为，我的集群情况是如下

　　
　　这是，因为。我之前前面4台机器都没有启动服务，所以会是如下。四个自动会勾选。

 http://cmbigdata1/cm5/redhat/6/x86_64/cm/5.3.6/

http://cmbigdata1:7180/cmf/express-wizard/resume#step=installStep

　　对于，第一个黄色警告，其实就是时间同步的问题。

[bigdata@cmbigdata1 ~]$ ntpdate pool.ntp.org
[bigdata@cmbigdata2 ~]$ ntpdate pool.ntp.org
[bigdata@cmbigdata3 ~]$ ntpdate pool.ntp.org
[bigdata@cmbigdata4 ~]$ ntpdate pool.ntp.org

　　对于，第二个黄色警告的解决办法！

    Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 60。使用 sysctl 命令在运行时更改该设置并编辑 /etc/sysctl.conf 以在重启后保存该设置。您可以继续进行安装，但可能会遇到问题，Cloudera Manager 报告您的主机由于交换运行状况不佳。以下主机受到影响： 
cmbigdata[1-4]

[bigdata@cmbigdata1 ~]$ su root
Password: 
[root@cmbigdata1 bigdata]# echo 0 > /proc/sys/vm/swappiness
[root@cmbigdata1 bigdata]#

　　然后，接下来，这一步，其实跟我另外写的一篇用ambari是一样的。

Ambari安装之部署3个节点的HA分布式集群

　　最后，我的初步安装是

　　因为，在以后，自己是可以修改的。

所以，CDH默认内嵌了一个数据库，无法通过。我这里选择需要MySQL或者Oracle请选择使用自定义数据库，

　　比如Hive， oozie 元数据库配置，使用我们安装的mysql来存放元数据，其中数据库名，用户名和密码需要预先创建好的。这里。请移步

　如果你是一名有经验大数据工程师，无论是安装jdk（注意：CentOS6.5有自带的jdk），还是安装mysql。都是先查看系统是否已经安装Mysql包。

[root@master app]# rpm -qa|grep mysql 
mysql-libs-5.1.71-1.el6.x86_64
[root@master app]# rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
[root@master app]# rpm -qa|grep mysql 
[root@master app]#

[root@cmbigdata1 ~]# yum install mysql-server
Is this ok [y/N]: y

[root@cmbigdata1 ~]# service mysqld start

[root@cmbigdata1 ~]# yum install mysql-connector-java
Is this ok [y/N]: y

[root@cmbigdata ~]# mysql -u root -p
Enter password:           //默认密码为空，输入后回车即可
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.1.73 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
mysql> set password for root@localhost=password('rootroot'); 　　密码设置为rootroot
默认情况下Mysql只允许本地登录，所以只需配置root@localhost就好

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> exit;
Bye
[root@cmbigdata1 ~]#

[root@cmbigdata1 ~]# mysql -uroot -prootroot

　　我这里，是第一次安装。所以直接进入

mysql> create database hive DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql>

mysql> create database clouder_manager DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on clouder_manager.* TO 'clouder_manager'@'%' IDENTIFIED BY 'clouder_manager';
Query OK, 0 rows affected (0.00 sec)

mysql>

mysql> create database amon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
Query OK, 0 rows affected (0.00 sec)

mysql>

mysql> create database smon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';
Query OK, 0 rows affected (0.00 sec)

mysql> create database rman DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
Query OK, 0 rows affected (0.00 sec)

mysql> create database hmon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';
Query OK, 0 rows affected (0.00 sec)

mysql> create database nav DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all on nav.* TO 'hmon'@'%' IDENTIFIED BY 'nav_password';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> select user,host,password from mysql.user;
+-----------------+------------+-------------------------------------------+
| user            | host       | password                                  |
+-----------------+------------+-------------------------------------------+
| root            | localhost  | *6C362347EBEAA7DF44F6D34884615A35095E80EB |
| root            | cmbigdata1 |                                           |
| root            | 127.0.0.1  |                                           |
|                 | localhost  |                                           |
|                 | cmbigdata1 |                                           |
| hive            | %          | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC |
| clouder_manager | %t         | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 |
| amon            | %          | *4A1243A12C2625553EC0465DD4E640DB056AA07A |
| smon            | %          | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 |
| rman            | %          | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E |
| hmon            | %          | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 |
+-----------------+------------+-------------------------------------------+
11 rows in set (0.00 sec)

mysql> exit;
Bye
[root@cmbigdata1 ~]#

　　然后，执行，还是如此

　　参考

Ubuntu 14.04 MySQL 安装配置 For Cloudera Manages 5

　　解决办法

mysql> grant all on *.* TO 'hive'@'cmbigdata1' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO 'amon'@'cmbigdata1' IDENTIFIED BY 'amon_password';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO 'clouder_manager'@'%cmbigdata1' IDENTIFIED BY 'clouder_manager';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO 'smon'@'cmbigdata1' IDENTIFIED BY 'smon_password';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO 'rman'@'cmbigdata1' IDENTIFIED BY 'rman_password';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* TO 'hmon'@'cmbigdata1' IDENTIFIED BY 'hmon_password';
Query OK, 0 rows affected (0.01 sec)

mysql> grant all on *.* TO 'nav'@'cmbigdata1' IDENTIFIED BY 'nav_password';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql>

　　别忘了oozie

mysql> create database oozie DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.22 sec)

mysql> grant all on *.* TO 'oozie'@'cmbigdata1' IDENTIFIED BY 'oozie_password';
Query OK, 0 rows affected (0.18 sec)

mysql> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password';
Query OK, 0 rows affected (0.04 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.15 sec)

mysql> select user,host,password from mysql.user;
+-----------------+-------------+-------------------------------------------+
| user            | host        | password                                  |
+-----------------+-------------+-------------------------------------------+
| root            | localhost   | *6C362347EBEAA7DF44F6D34884615A35095E80EB |
| root            | cmbigdata1  |                                           |
| root            | 127.0.0.1   |                                           |
|                 | localhost   |                                           |
|                 | cmbigdata1  |                                           |
| hive            | %           | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC |
| clouder_manager | %t          | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 |
| amon            | %           | *4A1243A12C2625553EC0465DD4E640DB056AA07A |
| smon            | %           | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 |
| rman            | %           | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E |
| hmon            | %           | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 |
| hive            | cmbigdata1  | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC |
| amon            | cmbigdata1  | *4A1243A12C2625553EC0465DD4E640DB056AA07A |
| clouder_manager | %cmbigdata1 | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 |
| smon            | cmbigdata1  | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 |
| rman            | cmbigdata1  | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E |
| hmon            | cmbigdata1  | *DC736DB1383C5AC16BFE895C856F8D30214AF19A |
| nav             | cmbigdata1  | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 |
| oozie           | cmbigdata1  | *81A1BB46F79EBD0AA76E6EFAA31D62458CFCAF62 |
| oozie           | %           | *81A1BB46F79EBD0AA76E6EFAA31D62458CFCAF62 |
+-----------------+-------------+-------------------------------------------+
20 rows in set (0.15 sec)

mysql>

　　下面，这里，是可以以后再更改的。

　　说白了，这里是启动的总界面，但是呢，我们若这里没启动成功，没关系，可以在后面，单独来启动！（具体如下的详细步骤）

http://cmbigdata1:7180/cmf/clusters/2/express-add-services/index#step=commandDetailsStep

　　遇到这个问题

正在启动 Key-Value Store Indexer 服务

Failed to execute command Start on service Key-Value Store Indexer

　　启动与停止Key-Value Store Indexer

　　通过，这样，也是可以达到目的的。对于这个问题，以后再来解决！

启动与停止Oozie和相关配置

　　对于这里，安装出现这个问题，无法解决，后面继续不了。其实，不然，（可以单独解决）

　　通过，这样，也是可以达到目的的。

　　启动与停止Hue，并相关配置

　　启动与停止Cloudera Management Service，并相关配置

http://cmbigdata1:7180/cmf/home

　　关于安装kafka，见

CDH集成Kafka

参考

http://www.doc88.com/p-0042075356791.html

posted @ 2017-04-07 23:03 大数据和AI躺过的坑阅读(6830) 评论(0) 编辑收藏举报

刷新页面返回顶部

大数据和人工智能躺过的坑

Cloudera Manager安装之利用parcels方式安装3或4节点集群（包含最新稳定版本或指定版本的安装）（添加服务）（CentOS6.5）（五）

Cloudera Manager安装之利用parcels方式安装单节点集群

Cloudera Manager安装之Cloudera Manager 5.3.X安装（三）（tar方式、rpm方式和yum方式）

Cloudera Manager安装之利用parcels方式（在线或离线）安装3或4节点集群（包含最新稳定版本或指定版本的安装）（添加服务）（Ubuntu14.04）（五）

Ambari安装之部署3个节点的HA分布式集群

公告