cdh 5.13 centos6.9安装

1.所有节点准备工作

1）。关闭防火墙

2）。关闭selinux 并重启系统

3）。建立NTP服务器，所有数据节点每天定时同步时间。

主节点在ntp.conf中增加

restrict 192.168.0.0 mask 255.255.0.0 nomodify

节点增加自动执行

* 2 * * * ntpdate 192.168.139.130

4）。下载安装包

cloudera-manager-agent-5.13.0-1.cm5130.p0.55.el6.x86_64.rpm
cloudera-manager-daemons-5.13.0-1.cm5130.p0.55.el6.x86_64.rpm
cloudera-manager-server-5.13.0-1.cm5130.p0.55.el6.x86_64.rpm
 jdk-6u31-linux-amd64.rpm

  oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

 

vim /etc/security/limits.conf

• soft nofile 65535

• hard nofile 65535

2.主节点--CM

1）。下载

安装包至下面目录/opt/cloudera/parcel-repo

CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel
 CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel.sha
 CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel.sha1
manifest.json

注意：CDH-5.13.0-1*.sha1要修改为*.sha

2）。CM端安装

cloudera-manager-server 

cloudera-manager-daemons

oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

3.节点安装

cloudera-manager-agent-5.13.0-1.cm5130.p0.55.el6.x86_64.rpm
cloudera-manager-daemons-5.13.0-1.cm5130.p0.55.el6.x86_64.rpm
 jdk-6u31-linux-amd64.rpm
 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

这几个包安装

4.MYSQL驱动安装（每个节点）

mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.31/mysql-connector-java-5.1.31-bin.jar /usr/share/java/mysql-connector-java.jar
cp /home/mysql-connector-java-5.1.45-bin.jar /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/oozie/lib/

cp mysql-connector-java-5.1.45-bin.jar /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hive/lib/
cp mysql-connector-java-5.1.45-bin.jar /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hue/lib/

cp /home/mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysl-connector-java.jar
cp /home/mysql-connector-java-5.1.45-bin.jar /var/lib/oozie

5.数据库安装
yum install mysql-server -y
数据库参数优化

加入自动启动
chkconfig mysqld on
server mysqld start
初始化密码

cat /var/log/mysqld.log |grep "password"

修改密码
alter user 'root'@'localhost' identified by '********';
远程访问
use mysql;
 grant all privileges on *.* to 'root'@'%' identified by '********' with grant option;
flush privileges; 

mysqladmin -uroot password '*****'
授权访问

grant all privileges on *.* to 'root'@'n1' identified by 'xxxx' with grant option;

grant all privileges on *.* to 'root'@'%' identified by 'xxxx' with grant option;

flush privileges;

 

注意 n1为管理节点的主机名，
创建数据库scm\hive\hue\oozie 字符集 utf8

初始化SCM数据库

/usr/share/cmf/schema/scm_prepare_database.sh mysql scm root ********

4.客户端优化

1）。交换分区优化

需要在/etc/sysctl.conf修改，加上：

vm.swappiness=10

生效： sysctl -p

首先查看透明大页是否启用，[always] never表示已启用，always [never]表示已禁用

[root@n12 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
[always] madvise never

• 1
• 2

如果是启用状态，修改/etc/rc.local文件并添加以下两行， 
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag 
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled 

 

centos7

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

上述修改要重启才能生效，如果希望暂时生效，可以直接运行上面两个命令。

spark2.2安装

java 1.8支持

Configuring a Custom Java Home Location

Java, which Cloudera services require, may be installed at a custom location. Follow the installation instructions in Java Development Kit Installation.

If you choose to use a custom Java location, modify the host configuration to ensure the JDK can be found:

1. Open the Cloudera Manager Admin Console.
2. In the main navigation bar, click the Hosts tab and optionally click a specific host link.
3. Click the Configuration tab.
4. Select Category > Advanced.
5. Set the Java Home Directory property to the custom location.
6. Click Save Changes.
7. Restart all services.

 错误解决

ERROR 1286 (42000): Unknown table engine ‘InnoDB’

在my.cnf

中增加 

ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so
plugin_dir=/usr/lib64/mysql/plugin

删除mysql gh r ib_logfile0  ib_logfile1文件。