CDH平台CM方式安装
1.CDH的安装
1.1环境配置
|
软件环境 |
VMware 15 |
|
系统环境 |
Centos 6.10（最小化安装） |
|
CDH版本 |
V5.10 |
|
MySQL版本 |
V5.6 |
|
主机数量 |
hadoop001.hadoop002.hadoop03 |
|
主机内存 |
8G.4G.4G |
|
硬盘大小 |
60G |
|
CPU数量 |
2核 |
|
网络规划 |
19.1.1.11/19.1.1.12/19.1.1.13 |
|
备注：如下又相关文件及配置链接，仅供参考 yum：https://blog.csdn.net/u010738184/article/details/78912075 parcels：http://archive.cloudera.com/cdh5/parcels/5.10.2.5/ cm：http://archive.cloudera.com/cm5/repo-as-tarball/5.10.2/cm5.10.2-centos6.tar.gz mysql：自行百度 |
|
1.2主机基础配置（三台机器都要进行配置）
1.2.1关闭iptables和selinux
[root@hadoop001 ~]# chkconfig iptable off
[root@hadoop001 ~]# vim /etc/selinux/config
SELINUX=disabled
1.2.2配置hosts
[root@hadoop001 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
19.1.1.11 hadoop001
19.1.1.12 hadoop002
19.1.1.13 hadoop003
1.2.3安装必要的包
[root@hadoop001 ~]# yum -y install gcc libaio
1.2.4安装Oracle Java环境
[root@hadoop001 ~]# tar -zxvf jdk-8u201-linux-x64.tar.gz
[root@hadoop001 ~]# mv jdk1.8.0_201/ /usr/java/
[root@hadoop001 ~]# vim /etc/profile
#set java path
export JAVA_HOME=/usr/java/jdk1.8.0_201
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
[root@hadoop001 ~]# java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
1.2.5设置时区时间同步，因为我用的是虚拟机，就不用设置这个，基本就是通过Linux上的NTP服务进行同步，百度谷歌很多教程
1.3离线文件yum配置（如果个人网络可行，可不设置，我这边在hadoop003机器上设置）
[root@hadoop003 ~]# yum -y install httpd
[root@hadoop003 ~]# chkconfig httpd on
[root@hadoop003 ~]# service httpd on
[root@hadoop003 ~]# cd /var/www/html/
[root@hadoop003 html]# mkdir parcels
[root@hadoop003 html]# mkdir cm5/redhat/6/x86_64/ -p
[root@hadoop003 html]# tar -zxvf cm5.10.0-centos6.tar.gz
[root@hadoop003 html]# mv cm/ cm5/redhat/6/x86_64/
[root@hadoop003 ~]# cd /etc/yum.repo.d
[root@hadoop003 yum.repos.d]# vim cloudera-manager.repo
[cloudera-manager]
name = Cloudera Manager, Version 5.10.0
baseurl = https://19.1.1.13/cm5/redhat/6/x86_64/cm/5.10.0/
gpgcheck = 0
[root@hadoop003 yum.repos.d]# yum list | grep cloudera-manager-server
cloudera-manager-server.x86_64 5.10.0-1.cm5100.p0.85.el6 cloudera-manager
cloudera-manager-server-db-2.x86_64 5.10.0-1.cm5100.p0.85.el6 cloudera-manager
1.4MySQL安装配置（安装在hadoop001机）
[root@hadoop001 ~]# cd /usr/local/
[root@hadoop001 local]# tar xzvf mysql-5.6.23-linux-glibc2.5-x86_64.tar.gz
[root@hadoop001 local]# mv mysql-5.6.23-linux-glibc2.5-x86_64 mysql
[root@hadoop001 local]# groupadd dba
[root@hadoop001 local]# useradd -g dba -G root -d /usr/local/mysql mysqladmin
[root@hadoop001 local]# echo 'password' | passwd --stdin mysqladmin
[root@hadoop001 local]# cp /etc/skel/.* /usr/local/mysql
[root@hadoop001 local]# vim /etc/my.cnf
[client]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
[mysqld]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M
table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED
server-id = 1
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
pid-file = /usr/local/mysql/data/hostname.pid
log-warnings
sysdate-is-now
binlog_format = MIXED
log_bin_trust_function_creators=1
log-error = /usr/local/mysql/data/hostname.err
log-bin=/usr/local/mysql/arch/mysql-bin
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:500M:autoextend
innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 2
innodb_log_file_size = 200M
innodb_buffer_pool_size = 512M
innodb_additional_mem_pool_size = 50M
innodb_log_buffer_size = 16M
innodb_lock_wait_timeout = 100
innodb_flush_log_at_trx_commit = 1
innodb_locks_unsafe_for_binlog=1
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
innodb_purge_threads=1
innodb_use_native_aio=on
innodb_file_per_table = 1
lower_case_table_names=1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[mysqlhotcopy]
interactive-timeout
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
[root@hadoop001 local]# chown mysqladmin:dba /etc/my.cnf
[root@hadoop001 local]# chmod 640 /etc/my.cnf
[root@hadoop001 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@hadoop001 local]# chmod -R 755 /usr/local/mysql
[root@hadoop001 local]# su - mysqladmin
[mysqladmin@hadoop001 ~]$ mkdir arch backup
[mysqladmin@hadoop001 ~]$ ./scripts/mysql_install_db
[root@hadoop001 ~]# cd /usr/local/mysql
[root@hadoop001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql
[root@hadoop001 mysql]# chmod +x /etc/rc.d/init.d/mysql
[root@hadoop001 mysql]# chkconfig --add mysql
[root@hadoop001 mysql]# chkconfig --level 345 mysql on
[root@hadoop001 mysql]# vim /etc/rc.local
su - mysqladmin -c "/etc/init.d/mysql start --federated"
[root@hadoop001 mysql]# su - mysqladmin
[mysqladmin@hadoop001 ~]$ rm -rf my.cnf
[mysqladmin@hadoop001 ~]$ mysqld_safe &
514 1823 1 0 Apr02 ? 00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/usr/local/mysql/data --pid-file=/usr/local/mysql/data/hostname.pid --federated
514 2495 1823 0 Apr02 ? 00:01:09 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --federated --log-error=/usr/local/mysql/data/hostname.err --pid-file=/usr/local/mysql/data/hostname.pid --socket=/usr/local/mysql/data/mysql.sock --port=3306
[mysqladmin@hadoop001 ~]$ mysql
mysql> use mysql
mysql> update user set password=password('password') where user='root';
Query OK, 4 rows affected (0.00 sec)
Rows matched: 4 Changed: 4 Warnings: 0
mysql> flush privileges;
Query OK, 0 rows affected (0.17 sec)
[mysqladmin@hadoop001 ~]$ vi .bash_profile
MYSQL_BASE=/usr/local/mysql
export MYSQL_BASE
PATH=$PATH:${MYSQL_BASE}/bin:/usr/kerberos/bin:/opt/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/mysql
export PATH
unset USERNAME
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1
1.5安装server
1.5.1rpm包安装
[root@hadoop001 ~]# yum install -y cloudera-manager-daemons
[root@hadoop001 ~]# yum install -y cloudera-manager-server
1.5.2安装mysql连接驱动
[root@hadoop001 ~]# cd /usr/share/java
[root@hadoop001 ~]# unzip mysql-connector-java-5.1.37.zip
[root@hadoop001 ~]# cd mysql-connector-java-5.1.37
[root@hadoop001 ~]# cp mysql-connector-java-5.1.37-bin.jar ../mysql-connector-java.jar
1.5.3配置mysql表
[root@hadoop001 ~]# su - mysqladmin
hadoop001:mysqladmin:/usr/local/mysql:>mysql -u root -ppassword
mysql> create database cmf DEFAULT CHARACTER SET utf8;
mysql> grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'password';
mysql> flush privileges;
mysql> create database amon DEFAULT CHARACTER SET utf8;
mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'password';
mysql> flush privileges;
1.5.4配置server连接mysql信息
[root@hadoopcm-01 cloudera-scm-server]# cd /etc/cloudera-scm-server/
[root@hadoopcm-01 cloudera-scm-server]# vim db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=19.1.1.11
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=password
com.cloudera.cmf.db.setupType=EXTERNAL
1.5.5启动服务
[root@hadoop001 cloudera-scm-server]# service cloudera-scm-server start
[root@hadoop001 cloudera-scm-server]# service cloudera-scm-server status
cloudera-scm-server (pid 1746) is running...
1.6开始CM安装CDH各组件，按照要求一步步进行安装即可，我们暂时安装HDFS+YARN+Zookeeper即可
1.6.1浏览器输入：http://19.1.1.11/7180
User：admin
Password：admin
1.7重点提示
1.7.1 com.cloudera.cmf.db.setupType=EXTERNAL 必须要如此设置
1.7.2 mysql-connector-java.jar 驱动包必须使用这个名称
1.7.3 echo 10 > /proc/sys/vm/swappiness
1.7.4 echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
1.7.5 java的目录必须放置在/usr/java下，不然你需要额外修改配置，不然报错
