CDH 6.0.1 集群搭建 「Process」

这次搭建我使用的机器 os 是 Centos7.4 RH 系的下面以流的方式纪录搭建过程以及注意事项

 

Step1:

配置域名相关，因为只有三台机器组集群，所以直接使用了 hosts 的方法：

修改主机名
hostnamectl set-hostname ryze-1.bigdata.com

然后在 /etc/hosts 文件中配置相关机器的域名 和 域名简写
x.x.x.x ryze-1.bigdata.com ryze-1
x.x.x.x zed-1.bigdata.com  zed-1
x.x.x.x zed-2.bigdata.com  zed-2
并且测试其能互通

配置 /etc/sysconfig/network
HOSTNAME=foo-1.example.com

验证配置
uname -a 需要和 hostname 得到一致的域名

 

Step2:

关闭防火墙

# 防火墙关闭
iptables-save > /root/firewall.rules
sudo chkconfig iptables off
sudo service iptables stop

 

Step3:

启动 NTP 服务

yum install ntp

设置一个同步时间服务器
/etc/ntp.conf

# 笔者使用的 aliyun 这里应该是可以跳过的，我检查了安装 ntp 之后，里面配置了非常多 aliyun 的相关节点。
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

# 开启 NTP 服务
sudo systemctl start ntpd

# 配置 NTP 服务自启动
sudo systemctl enable ntpd

# 向某个服务器同步时间
ntpdate -u <ntp_server>

# 同步系统时间
hwclock --systohc

 

所有机器完成上面配置之后，我们开始进入安装的步骤。

 

Step1:

首先 Cloudera 为用户已经准备好了专用的程序仓库，我们需要将其下载下来

wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/

# Import the repository signing GPG key
sudo rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

 

Step2: 安装 为 CDH 6.0.x 安装 jdk1.8

# yum 安装
sudo yum install oracle-j2sdk1.8

# 自己下载包安装
tar xvfz /path/to/jdk-8u<update_version>-linux-x64.tar.gz -C /usr/java/
注意不能更换路径

这里如果使用 yum 安装 cloudera 的源的话会安装 1.8u141 并且会补充一个文件

• The RHEL-compatible and Ubuntu operating systems supported by Cloudera Enterprise 6 all use AES-256 encryption by default for tickets. To support AES-256 bit encryption in JDK versions lower than 1.8u161, you must install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File on all cluster and Hadoop user machines. Cloudera Manager can automatically install the policy files, or you can install them manually. For JCE Policy File installation instructions, see the README.txt file included in the jce_policy-x.zip file. JDK 1.8u161 and higher enable unlimited strength encryption by default, and do not require policy files.
• On SLES platforms, do not install or try to use the IBM Java version bundled with the SLES distribution. CDH does not run correctly with that version.

需要注意一下，Java1.8u161 以上的版本似乎不受影响。

 

Step3:

安装 cloudera manager 到目标机 和 配置 TLS

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

警告如果没有证书，就不要配这个否则后面会出现 agent 无法上报的问题。
然后配置 auto TSL
sudo JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services

That's it! When you start Cloudera Manager Server, it will have TLS enabled, and all hosts that you add to the cluster, as well as any supported services, will automatically have TLS configured and enabled.

 

Step4: 安装 CM 使用的数据库

# 安装数据库大礼包
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld

# 然后开始数据库配置
sudo systemctl stop mysqld

# 拷贝日志文件
Move old InnoDB log files /var/lib/mysql/ib_logfile0 and /var/lib/mysql/ib_logfile1 out of /var/lib/mysql/ to a backup location

# 配置数据库文件 /etc/my.conf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

# 添加到自启动项
sudo systemctl enable mysqld

# 重新将 MySQL 启动起来
sudo systemctl start mysqld

安装 JDBC 驱动

# Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html (in .tar.gz format). As of the time of writing, you can download version 5.1.46 using wget as follows:
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

# Extract the JDBC driver JAR file from the downloaded file. For example:
tar zxvf mysql-connector-java-5.1.46.tar.gz

# Copy the JDBC driver, renamed, to /usr/share/java/. If the target directory does not yet exist, create it. For example:
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

使用 utf8 编码规则创建新的数据库服务于需要使用到数据库的服务：

Databases for Cloudera Software

Service	Database	User
Cloudera Manager Server	scm	scm
Activity Monitor	amon	amon
Reports Manager	rman	rman
Hue	hue	hue
Hive Metastore Server	metastore	hive
Sentry Server	sentry	sentry
Cloudera Navigator Audit Server	nav	nav
Cloudera Navigator Metadata Server	navms	navms
Oozie	oozie	oozie

这里我们会使用的服务：

Cloudera Manager Server

Reports Manager

Hue

Hive Metastore Server

Cloudera Navigator Audit Server

Cloudera Navigator Metadata Server

但是我打算全部都建起来，后面再来看

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

 

Step5: setup database

由于我们的数据库搭建在远端 所以我们使用：

Example 2: Running the script when MySQL or MariaDB is installed on another host
This example demonstrates how to run the script on the Cloudera Manager Server host (cm01.example.com) and connect to a remote MySQL or MariaDB host (db01.example.com):

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm
Enter database password:
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

 

Step6: 安装 CDH 和 其他的组件

# start cloudera-scm-server
sudo systemctl start cloudera-scm-server

# 通过该命令观察启动流程
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
当看到
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
the Cloudera Manager Admin Console is ready

然后就可以通过 ip:8070 上去了

之后就是选择版本，选择完了之后进入到集群安装界面：

 

扫描 hosts 地址使用 FQDN 来标识，填写之后进行扫描。

Range Definition    Matching Hosts
10.1.1.[1-4]    10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4

host[1-3].example.com    host1.example.com, host2.example.com, host3.example.com
host[07-10].company.com    host07.example.com, host08.example.com, host09.example.com, host10.example.com
Note: Unqualified hostnames (short names) must be unique in a Cloudera Manager instance. For example, you cannot have both host01.example.com and host01.standby.example.com managed by the same Cloudera Manager Server.
You can specify multiple addresses and address ranges by separating them with commas, semicolons, tabs, or blank spaces, or by placing them on separate lines. Use this technique to make more specific searches instead of searching overly wide ranges. Only scans that reach hosts running SSH will be selected for inclusion in your cluster by default. You can enter an address range that spans over unused addresses and then clear the nonexistent hosts later in the procedure, but wider ranges require more time to scan.

单机 serach 即可扫描，扫描到节点之后进行下一步。

 

Select Repository 使用默认选项直接选择下一步。

 

Accept JDK License 阅读协议☑️选择下一步。

 

ssh 设置，选择合适的选项下一步。

 

然后开始安装 agents 下载速度慢可以直接用 proxychain 或者 找快的地方在 https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/ 下好之后上传。

 

然后开始安装 parcels 因为这个在国内安装下载速度也是超级慢所以推荐直接用网速快的地方去 https://archive.cloudera.com/cdh6/6.0.1/parcels/  下载对应的包。

然后上传到服务器放到 cm 机器的 /opt/cloudera/parcel-repo/ 目录下。要下包 sha1 文件和 manifest 文件。

 

按照提示解决 warning

已启用透明大页面压缩，可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和
“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置。以下主机将受到影

Starting with CDH 6, PostgreSQL-backed Hue requires the Psycopg2 version to be at least 2.5.4, see the documentation for more information. 
This warning can be ignored if hosts will not run CDH 6, or will not run Hue with PostgreSQL. The following hosts have an incompatible Psycopg2 version of '2.5.1': 

 

Step7: Set Up a Cluster Using the Wizard

选择想要安装的服务

 

进行角色配置，master 集中在一台机器的话尽量让该机器的配置比较强。

比如我们的 ryze-1 的机器就比较强，所以承担了更多的 master 节点的角色。

 

设置要安装服务使用的数据库，之前我们 setup 了数据库信息，这里直接填写之前 setup 的信息

 

审计查看配置参数有没有需要修改的

 

命令详细信息执行相关操作，启动相关服务

 

summary 提示信息 -> 服务已安装、配置并在群集中运行 XD。

 

Reference:

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/installation_reqts.html  Before You Install

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cm_cdh.html  Installing Cloudera Manager, CDH, and Managed Services

http://www.uuboku.com/444.html  mysql配置sql_mode中STRICT_TRANS_TABLES和STRICT_ALL_TABLES 区别

http://www.cnblogs.com/zhoujinyi/p/3179279.html  InnoDB O_DIRECT选项漫谈（一）【转】

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/cm_ig_installing_configuring_dbs.html#cmig_topic_5_1  Required Databases

https://www.oschina.net/question/3964891_2287725  CDH6在安装agent时，提示安装失败 无法接收 Agent 发出的检测信号

https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/  yum 包下载包地址

https://archive.cloudera.com/cdh6/6.0.1/parcels/  parcels 下载包地址