CDH5集群部署
1、基础环境配置
1.1、系统环境
[root@master01 ~]# cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) [root@master01 ~]# uname -r 3.10.0-229.el7.x86_64
1.2、关闭selinux
setenforce 0 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
1.3、关闭防火墙
[root@master01 ~]# systemctl stop firewalld.service [root@master01 ~]# systemctl disable firewalld.service [root@master01 ~]# firewall-cmd --state not running
1.4、修改主机名
master上修改主机名，集群其他节点也需更改。
hostnamectl --static set-hostname master01
1.5、配置本地hosts或者DNS
这里使用本地hosts解析，注意CHD5集群主机名与hosts必须一致。若搭建了DNS服务器就加入到DNS服务配置中。
cat >>/etc/hosts<<EOF 192.168.159.146 master01 192.168.159.147 node01 192.168.159.148 node02 192.168.159.149 node03 EOF
1.6、ntp时间同步服务配置
node{01,02,03} 节点都需要与master01时间同步，CDH集群对时间要求很高，时区设置也需一致。
具体配置参看：ntp时间同步
1.7、安装 jdk
JDK 版本： 1.8.0_151
[root@localhost~]# rpm -ivh jdk-8u151-linux-x64.rpm
准备中... ################################# [100%]
正在升级/安装...
1:jdk1.8-2000:1.8.0_151-fcs ################################# [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
[root@localhost ~]# java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
2.根据linux版本提取yum源
wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -O /etc/yum.repos.d/cloudera-manager.repo
2.1、parcel包
网址：http://archive.cloudera.com/cdh5/parcels/latest/
文件：CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel
CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1
2.2、CM包
网址：http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.12.0/RPMS/x86_64/
文件：cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
cloudera-manager-agent-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
cloudera-manager-server-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
2.3、安装上面三个rpm包
yum localinstall --nogpgcheck *.rpm -y
注意：
1.master上三个都需安装
2.node上先装cloudera-manager-daemons再装cloudera-manager-agent
3.安装MySQL5.6
3.1、安装mysql5.6扩展源
rpm -Uvh http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm 提取成功之后会在 /etc/yum.repo.d 路径下方生成两个 yum 文件 [root@master01 ~]# /etc/yum.repos.d/|grep mysql mysql-community.repo mysql-community-source.repo
3.2、安装mysql
yum install mysql-server -y
3.3、启动mysql服务
systemctl start mysqld
3.4、设置账号密码
mysql -u root
mysql> use mysql;
mysql> UPDATE user SET Password = PASSWORD('111111') WHERE user = 'root';
mysql> FLUSH PRIVILEGES;
3.5、导入连接 jar 包
mkdir -p /usr/share/java/ cp mysql-connector-java.jar /usr/share/java/
3.6、执行 CDH 建表语句
/usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p111111 --scm-host localhost scm scm scm_password
3.7、创建数据库账号和密码
create database hive; create database hue; create database oozie; grant all on hive.* to hive@'%' identified by 'hive'; grant all on hue.* to hue@'%' identified by 'hue'; grant all on oozie.* to oozie@'%' identified by 'oozie'; flush privileges;
4.导入parcel包
设定集群所需要的parcel包：拷贝parcel包到/opt/cloudera/parcel-repo/ cd /opt/cloudera/parcel-repo/ mv CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha1 CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha
5.启动服务并加入开启自启动
systemctl start cloudera-scm-server systemctl start cloudera-scm-agent systemctl enable cloudera-scm-server systemctl enable cloudera-scm-agent
正常启动之后，进入页面 http://192.168.159.146:7180 选择要安装的主机ip
常见问题
1.关闭交换内存
sysctl vm.swappiness=10 echo 'vm.swappiness=10'>>/etc/sysctl.conf
2.关闭透明页面压缩
echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled
3.如果一直获取锁，建议删除/tmp目录中的scm_prepare_node.*的文件，以及.scm_prepare_node.lock文件
4.如果出现cm_guid和uuid不符合，则关闭scm-agent，并且删除/var/lib/cloudera-scm-agent/uuid和/var/lib/cloudera-scm-agent/cm_guid文件
systemctl stop cloudera-scm-agent rm -f /var/lib/cloudera-scm-agent/uuid rm -f /var/lib/cloudera-scm-agent/cm_guid
