kudu_CM安装准备工作
Cloudera Manager简介：
hadoop: https://yq.aliyun.com/articles/60759
----------------------------------------------下载----------------------------------------------
--下载安装包：
1.Cloudera Manager仓库镜像包下载地址：
https://archive.cloudera.com/cdh5/parcels/5.12/
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha1
manifest.json
2.CDH parcel安装包地址：
http://archive.cloudera.com/cm5/repo-as-tarball/5.12.2/
cm5.12.2-centos7.tar.gz
3.kudu的parcel安装包地址：
http://archive.cloudera.com/kudu/parcels/latest/
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel.sha1
4.kudu 的csd安装包地址：
http://archive.cloudera.com/kudu/csd/
注意： 通过 Cloudera Manager 安装parcel时sha1格式的文件需要提前修改为sha。
----------------------------------------------准备工作----------------------------------------------
--配置
修改主机名：
hostnamectl set-hostname Node1
--1.修改hosts文件 所有节点
[root@node1 .ssh]# cat /etc/hosts
127.0.0.1 localhost.Node1
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.171 Node1
192.168.2.172 Node2
192.168.2.173 Node3
--2.配置公钥认证 免密登录
2.1 配置公钥认证
ssh-keygen -t rsa
一路回车，生成无密码的密钥对
2.2 将公钥添加到认证文件中：
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
2.3 scp文件到所有受管节点（ Node2 和 Node3）的~/.ssh目录：
scp ~/.ssh/authorized_keys root@Node2:~/.ssh/ scp ~/.ssh/authorized_keys root@Node3:~/.ssh/
2.4如果没有.ssh目录，手动创建并赋权限
[root@node1 ~]# mkdir .ssh [root@node1 ~]# chmod 700 .ssh/
--3.关闭防火墙
[root@node1 .ssh]# systemctl stop firewalld.service [root@node1 .ssh]# firewall-cmd --state not running
--4.关闭SELINUX
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
setenforce 0
--5.配置NTP服务 同步时间
--6.优化虚拟内存需求率(所有节点)
1)检查虚拟内存需求率
cat /proc/sys/vm/swappiness
2)临时降低虚拟内存需求率
sysctl vm.swappiness=0
永久降低虚拟内存需求率
使用命令 vi /etc/sysctl.conf 增加
vm.swappiness = 0
并运行如下命令使生效
sysctl -p
--7.解决透明大页面问题(所有节点)
1)检查透明大页面问题
执行代码：
echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled
以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置。
cat /sys/kernel/mm/transparent_hugepage/defrag
如果显示为：
[always] madvise never
2)临时关闭透明大页面问题
echo never > /sys/kernel/mm/transparent_hugepage/defrag
确认配置生效：
cat /sys/kernel/mm/transparent_hugepage/defrag
应该显示为：
always madvise [never]
3)配置开机自动生效
使用命令 vim /etc/rc.local,加入如下内容
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
--8.安装JDK
卸载自带的OpenJdk，
使用 rpm -qa | grep java 查询java相关的包，
使用 rpm -e --nodeps 包名 卸载。
--------------------------------------- -----------------安装配置MariaDB---------------------------------------
a. 确保/var/lib/mysql目录有足够大的磁盘空间
b. 卸载自带的mysql。
yum remove mysql
1.添加yum源
vi /etc/yum.repos.d/MariaDB.repo [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/10.1/centos7-amd64 gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB gpgcheck=1
2.安装MariaDB
yum install MariaDB-server MariaDB-client MariaDB-devel --enablerepo=MariaDB.repo
3.配置my.conf vim /etc/my.cnf 用以下内容替换 [mysqld] transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: # symbolic-links = 0 key_buffer = 16M key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 550 #expire_logs_days = 10 #max_binlog_size = 100M #log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system #and chown the specified folder to the mysql user. log_bin=/var/lib/mysql/mysql_binary_log binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
3.启动MariaDB
service mysql start
启动失败： 失败1： /usr/bin/mysqld_safe_helper: Can't create/write to file '/var/log/mysqld.log' (Errcode: 13 "Permission denied") 授权： chown -R mysql /var/log/ 失败2： 2018-02-05 11:22:22 139647932491520 [ERROR] mysqld: Can't create/write to file '/var/run/mysqld/mysqld.pid' (Errcode: 13 "Permission denied") 2018-02-05 11:22:22 139647932491520 [ERROR] Can't start server: can't create PID file: Permission denied 解决： [root@Node1 log]# cat /var/log/mysqld.log [root@Node1 run]# mkdir mysqld [root@Node1 log]# chown -R mysql /var/run/mysqld [root@Node1 log]# chgrp -R mysql /var/run/mysqld
4.查看MariaDB版本
mysql --version
5.设置开机启动
chkconfig mysql on
6.初始化数据库
sudo /usr/bin/mysql_secure_installation Enter current password for root (enter for none): Enter Set root password? [Y/n] y New password: 123456 Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! Remove anonymous users? [Y/n] y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n ... skipping. By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] y ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB!
7.使用mysql -uroot -p进入mysql命令行，创建数据库和用户：
[root@Node1 log]# mysql -uroot -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 12 Server version: 10.1.30-MariaDB MariaDB Server Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database hive DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> create database hue DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> create database oozie DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> flush privileges; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> exit Bye
-------------------------------------------安装mysql JDBC 驱动（管理节点）-----------------------------------------
下载mysql JDBC 驱动放到目录 /usr/share/java/ 并修改名为mysql-connector-java.jar
下载地址：https://dev.mysql.com/downloads/connector/j/
[root@Node1 mysql-connector-java-5.1.45]# cp mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar
----------------------------------------- 安装配置 Cloudera Manager（管理节点）---------------------------------------
建立Cloudera Manager安装文件自定义存储库
a.安装httpd服务器
查询一下是否已经安装了apache rpm -qa httpd 如果还没有则进行安装 yum -y install httpd 启动apache service httpd start 开机自启动 chkconfig httpd on
1.将Cloudera Manager仓库镜像包cm5.12.2-centos7.tar.gz
解压到/var/www/html目录，文件目录结构如下
[root@Node1 cm]# tar zxvf /root/softwares/kudu/cm5.12.2-centos7.tar.gz -C /var/www/html
2.通过rpm安装包本地安装 Cloudera Manager
到目录 /var/www/html/cm/5/RPMS/x86_64
--本地安装rpm包，不验证签名
yum --nogpgcheck localinstall cloudera-manager-daemons-5.12.*.rpm cloudera-manager-server-5.12.*.rpm enterprise-debuginfo-5.12.*.rpm
3.Parcel和csd格式文件上传
a. 上传下列文件到Parcel包的存放路径： /root/cloudera/parcel-repo/
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha1
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel.sha1
manifest.json
4.上传最新的kudu csd文件到csd包存放路径： /root/cloudera/csd/
KUDU-5.10.2.jar
5.配置 Cloudera Manager Server 数据库
使用命令scm_prepare_database.sh创建Cloudera Manager Server数据库配置文件
命令格式如下
/usr/share/cmf/schema/scm_prepare_database.sh database-type [options] database-name username password
如：
/usr/share/cmf/schema/scm_prepare_database.sh mysql -hNode1 -uroot -p123456 --scm-host Node1 scm scm scm 成功： [root@node1 cloudera-scm-server]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -hNode1 -uroot -p123456 --scm-host Node1 scm scm scm JAVA_HOME=/usr/java/jdk1.7.0_79 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.7.0_79/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly! [root@node1 cloudera-scm-server]#
执行完成后生成数据库配置文件/etc/cloudera-scm-server/db.properties # Auto-generated by scm_prepare_database.sh on 2018年 01月 10日 星期三 14:42:32 CST # # For information describing how to configure the Cloudera Manager Server # to connect to databases, see the "Cloudera Manager Installation Guide." # com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=localhost com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.setupType=EXTERNAL com.cloudera.cmf.db.password=scm
报错：
mysql>use mysql;
mysql>desc user;
mysql>GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY "123456"; //为root添加远程连接的能力。
mysql>update user set Password = password('123456') where User='root';
mysql>select Host,User,Password from user where User='root';
mysql>flush privileges;
mysql>exit
grant all on scm.* to 'root'@'%' identified by '123456' with grant option;
cat /etc/cloudera-scm-server/db.properties
6.启动Cloudera Manager Server
[root@node1 ~]# service cloudera-scm-server start Starting cloudera-scm-server (via systemctl): Warning: cloudera-scm-server.service changed on disk. Run 'systemctl daemon-reload' to reload units. [ 确定 ] [root@node1 ~]# systemctl daemon-reload [root@node1 ~]# service cloudera-scm-server start Starting cloudera-scm-server (via systemctl): [ 确定 ]
启动失败报错没有JDK 一个是OS的环境变量，一个是脚本运行时候的设置 [root@Node1 init.d]# pwd /etc/init.d [root@Node1 init.d]# vim cloudera-scm-server 添加： export JAVA_HOME=/root/jdk1.7.0_79 再次启动： [root@Node1 init.d]# service cloudera-scm-server start 查看日志： [root@Node1 init.d]# cat /var/log/cloudera-scm-server/cloudera-scm-server.out JAVA_HOME=/root/jdk1.7.0_79 /usr/share/cmf/bin/cmf-server:行129: /root/jdk1.7.0_79/bin/java: 权限不够 /usr/share/cmf/bin/cmf-server: 第 129 行:exec: /root/jdk1.7.0_79/bin/java: 无法执行: 权限不够 修改jdk路径 [root@Node1 ~]# cd /usr/ [root@Node1 usr]# mkdir java/ [root@Node1 init.d]# echo $JAVA_HOME /usr/java/jdk1.7.0_79 再次启动： [root@Node1 init.d]# service cloudera-scm-server start
--CM
等待大概两分钟，访问 http://192.168.2.171:7180/ 进入管理端 (登陆名:admin 密码:admin)
--kudu
http://192.168.2.171:8051/
