CDH/Hadoop 5.15 installation steps

I will talk the main steps to install CDH 5.15 on Linux(CENT OS 6.10).  The installation method is Manual installation with tarball and parcels. The newest version if CDH 6.0.1 when I write this guide but it does not support CENT OS 6.10 so I have to install 5.15.

Software components List:

• CDH, the parcel. CDH-5.15.1-1.cdh5.15.1.p0.4-el6.parcel
• CM: cloudery manager. cloudera-manager-el6-cm5.15.1_x86_64.tar.gz
• mysql 5.7.73 and jdbc driver: mysql-connector-java-8.0.11.jar
• jvm: jdk-8u181-linux-x64.rpm

Here is how to get them:

JVM:
wget http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.rpm?AuthParam=1539265359_30b3a4b9e17f3ed3b0962980168c2721

CDH: 
wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.15.1_x86_64.tar.gz
wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.15.1-1.cdh5.15.1.p0.4-el6.parcel
wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.15.1-1.cdh5.15.1.p0.4-el6.parcel.sha1
wget http://archive.cloudera.com/cdh5/parcels/5.15.1.4/manifest.json

MySQL Yum Repo:
wget https://repo.mysql.com//mysql57-community-release-el6-11.noarch.rpm

Hareware: 

• ....more powerful, more better...

Machine/OS Setup: 

• Making sure the time is sychronized across machines(Configure ntp if needed). 
• Network: change hostname you like. I have 4 servers ns01, ns02, ns03, ns04. Add them into the /etc/hosts file like below. Making sure you have high bandwidth accross the nodes. Mine is 210MB/Sec.
  

  192.168.0.79 ns04
  192.168.0.77 ns02
  192.168.0.232 ns01
  192.168.0.114 ns03

• Tune swap(each node):
  vim /etc/sysctl.conf
  vm.swappiness = 10
  
• Disable security Linux.  SELINUX=disabled in /etc/selinux/config
• Turn off firewall(each node):
  service iptables stop.
  chkconfig iptables off
• SSH without pasword:
  1. run ssh-keygen -t rsa in each node. type Enter when prompted.
  2. run cp .ssh/id_rsa.pub .ssh/pub_key.ns0$n in each node to save the public key to a seperate file. $n is the number of the machine.
  3. copy those files to one of the machine like ns01.  
     
     
  4. run cat pub_key.ns0* >> authorized_keys in ns01
  5. boardcase the file to all other machines.
     

JVM setup(each node)

......do it in each machine. Please use the rpm distribution to install(rpm -ivh jdk*.rpm) and set the JAVA_HOME to /usr/java/latest. I got an error like 'deploy client configuration for spark'  during setup using the jdk*.tar.gz distribution.

MySQL Installation/Setup(one node, ns01)

• Install and init mysql:
  https://opensourcedbms.com/dbms/installing-mysql-5-7-on-centosredhatfedora/
• Sql to create databases/users for Hadoop
  

  create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  create database activity DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
  
  GRANT ALL PRIVILEGES ON  hive.*  to 'hive'@'%' identified by 'hive' with grant option ;
  flush privileges ;
  GRANT ALL PRIVILEGES ON  oozie.*  to 'oozie'@'%' identified by 'oozie' with grant option ;
  flush privileges ;
  GRANT ALL PRIVILEGES ON  hue.*  to 'hue'@'%' identified by 'hue' with grant option ;
  flush privileges ;
  GRANT ALL PRIVILEGES ON  activity.*  to 'activity'@'%' identified by 'activity' with grant option ;
  flush privileges ;

CDH Install

• untar the cm to target folder /opt (ns01 node). 
  tar -xzf cloudera-manager-el6-cm5.15.1_x86_64.tar.gz -C /opt
• copy mysql jdbc driver to avoid the 'java.lang.ClassNotFoundException: com.mysql.jdbc.Driver' (ns01 node)
  cp mysql-connector-java-8.0.11.jar /opt/cm-5.15.1/share/cmf/lib/
  cp mysql-connector-java-8.0.11.jar /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hive/lib/   
  ........................................(cp mysql driver to the proper location for other components need mysql connection to create and init database,tables......)
• Creat Cloudera Manager DB(ns01 node)
  

  (Run in MySQL)
  GRANT ALL PRIVILEGES ON  scm.*  to 'scm'@'%' identified by 'scm' with grant option ;

  (Run in Shell)
  /opt/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -p*** --scm-host ns01 scm scm scm

• copy the hadoop parcel to the parcel-repo folder. rename (mv) the *.sha1 file to .sha . See below result of the folder.(ns01 node)
  
• update the cloudera manager host server so the agents can connect to it.(ns01 node)
  vim /opt/cm-5.15.1/etc/cloudera-scm-agent/config.ini
  
• copy cloudera manager to other machines as well. You can tar the folder /opt/cm-5.15.1 and scp to other machines. 
• Add cloudera-scm User ( each node )
  useradd --system --home /opt/cm-5.15.1/run/cloudera-scm-server --no-create-home --shell=/bin/false -comment "Cloudera SCM User" cloudera-scm
• Reboot all machines before start cm. I write a simple script rebootAll.sh to do it.
• Run the service and configuration manager(scm) on server node and agent nodes.  You can also run the agent on the server node so you will have one more node to install Hadoop.
  /opt/cm-5.15.1/etc/init.d/cloudera-scm-server start  (run  in only 1 node)
  /opt/cm-5.15.1/etc/init.d/cloudera-scm-agent start  (run in all nodes)
• launch the CM by: http://ns01:7180 from you browser
  
• Wait several seconds if needed. If you can see the login page(credential: admin/admin), there should be no big problem. I will paste some of the UI here.
  hosts list, you can see I have 4 nodes.
  
  The repository you have configured.
  
  

  Assign the roles for node.