8-hadoop的运行模式（完全分布式）（下）
集群时间同部
1、系统定时任务
1、重启crond服务：service crond restart
2、crontab 定时任务设置
基本语法：crontab [选项]
选项
* * * * * 执行的任务
第五个“*” 一周中的星期几 0-7 0/7都是星期日
特殊符号
时间同部
时间同步的方式：找一个机器，作为时间服务器
所有的机器与这台集群时间进行定时的同步，比如，每隔十分钟，同步一次时间。
1、检查ntp是否安装
rpm -qa | grep ntp
如果不现实三个列表则表示需要进行安装
2、进行修改配置文件
vim /etc/ntp.conf
（授权192.168.43.0-192.168.1.255网段上的所有机器可以从这台机器上查询和同步时间）
修改2（集群在局域网中，不使用其他互联网上的时间）
添加3（当该节点丢失网络连接，依然可以采用本地时间作为时间服务器为集群中的其他节点提供时间同步）
server 127.127.1.0
fudge 127.127.1.0 stratum 10
3、修改/etc/sysconfig/ntpd 文件
SYNC_HWCLOCK=yes
4、重启ntp服务
service ntpd status
service ntpd start
5、ntpd服务开机启动
chkconfig ntpd on
其他机器进行配置
1、在其他机器配置10分钟与时间服务器同步一次
2、修改任意机器时间
date -s "2018-9-11 11:11:11"
3、十分钟后查看机器是否与时间服务器同步
date
说明：测试的时候可以将10分钟调整为1分钟，节省时间。
Hadoop编译源码
准备工作
1. CentOS联网
配置CentOS能连接外网。Linux虚拟机ping www.baidu.com 是畅通的
注意：采用root角色编译，减少文件夹权限出现问题
2. jar包准备(hadoop源码、JDK8、maven、ant 、protobuf)
（1）hadoop-2.7.2-src.tar.gz
（2）jdk-8u144-linux-x64.tar.gz
（3）apache-ant-1.9.9-bin.tar.gz（build工具，打包用的）
（4）apache-maven-3.0.5-bin.tar.gz
（5）protobuf-2.5.0.tar.gz（序列化的框架）
安装
1、JDK解压、配置环境变量 JAVA_HOME和PATH，验证java-version(如下都需要验证是否配置成功)
2、Maven解压、配置 MAVEN_HOME和PATH
解压命令：tar -zxvf apache-maven-3.0.5-bin.tar.gz -C /opt/module/
配置环境变量
export MAVEN_HOME=/opt/module/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin
测试
[root@localhost apache-maven-3.0.5]# mvn -version Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 08:51:28-0500) Maven home: /opt/module/apache-maven-3.0.5 Java version: 1.8.0_131, vendor: Oracle Corporation Java home: /opt/jdk1.8.0_131/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-957.el7.x86_64", arch: "amd64", family: "unix"
配置配置文件（使用阿里云）
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>central</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
3、ant解压，配置ANT_HOME和PATH
[root@localhost software]# tar -zxvf apache-ant-1.9.9-bin.tar.gz -C /opt/module/
配置profile文件
export ANT_HOME=/opt/module/apache-ant-1.9.9
export PATH=$PATH:$ANT_HOME/bin
测试
[root@localhost software]# ant -version
Apache Ant(TM) version 1.9.9 compiled on February 2 2017
4、安装 glibc-headers 和 g++ 命令如下
yum install glibc-headers
yum install gcc-c++
5、安装make和cmake
[root@hadoop101 apache-ant-1.9.9]# yum install make
[root@hadoop101 apache-ant-1.9.9]# yum install cmake
6、解压protobuf ，进入到解压后protobuf主目录
/opt/module/protobuf-2.5.0，然后相继执行命令
解压：
[root@localhost software]# tar -zxvf protobuf-2.5.0.tar.gz -C /opt/module/
进入目录执行
[root@localhost protobuf-2.5.0]# ./configure
[root@localhost protobuf-2.5.0]# make
[root@localhost protobuf-2.5.0]# make check
[root@localhost protobuf-2.5.0]# make install
[root@localhost protobuf-2.5.0]# ldconfig
配置环境变量
export LD_LIBRARY_PATH=/opt/module/protobuf-2.5.0
export PATH=$PATH:$LD_LIBRARY_PATH
验证
[root@localhost protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
7、安装openssl
yum install openssl-devel
8、安装ncurses-devel
yum install ncurses-devel
编译源码：
1. 解压源码到/opt/目录
tar -zxvf hadoop-2.7.2-src.tar.gz -C /opt/
2. 进入到hadoop源码主目录
[root@localhost hadoop-2.7.2-src]# pwd
/opt/hadoop-2.7.2-src
3. 通过maven执行编译命令
mvn package -Pdist,native -DskipTests -Dtar
[root@localhost hadoop-2.7.2-src]# mvn package -Pdist,native -DskipTests -Dtar
[INFO] Scanning for projects...
Downloading: http://maven.aliyun.com/nexus/content/groups/public/org/apache/felix/maven-bundle-plugin/2.5.0/maven-bundle-plugin-2.5.0.pom
Downloaded: http://maven.aliyun.com/nexus/content/groups/public/org/apache/felix/maven-bundle-plugin/2.5.0/maven-bundle-plugin-2.5.0.pom (5 KB at 1.3 KB/sec)
Downloading: http://maven.aliyun.com/nexus/content/groups/public/org/apache/felix/felix-parent/2.1/felix-parent-2.1.pom
Downloaded: http://maven.aliyun.com/nexus/content/groups/public/org/apache/felix/felix-parent/2.1/felix-parent-2.1.pom (10 KB at 16.1 KB/sec)
Downloading: http://maven.aliyun.com/nexus/content/groups/public/org/apache/apache/9/apache-9.pom
Downloaded: http://maven.aliyun.com/nexus/content/groups/public/org/apache/apache/9/apache-9.pom (15 KB at 24.6 KB/sec)
Downloading: http://maven.aliyun.com/nexus/content/groups/public/biz/aQute/bnd/bndlib/2.3.0/bndlib-2.3.0.pom
Downloaded: http://maven.aliyun.com/nexus/content/groups/public/biz/aQute/bnd/bndlib/2.3.0/bndlib-2.3.0.pom (2 KB at 1.7 KB/sec)
Downloading: http://maven.aliyun.com/nexus/content/groups/public/biz/aQute/bnd/parent/2.3.0/parent-2.3.0.pom
Downloaded: http://maven.aliyun.com/nexus/content/groups/public/biz/aQute/bnd/parent/2.3.0/parent-2.3.0.pom (5 KB at 5.6 KB/sec)
Downloading: http://maven.aliyun.com/nexus/content/groups/public/org/osgi/org.osgi.core/4.3.1/org.osgi.core-4.3.1.pom
Downloaded: http://maven.aliyun.com/nexus/content/groups/public/org/osgi/org.osgi.core/4.3.1/org.osgi.core-4.3.1.pom (2 KB at 2.9 KB/sec)
Downloading: http://maven.aliyun.com/nexus/content/groups/public/org/apache/felix/org.apache.felix.bundlerepository/1.6.6/org.apache.felix.bundlerepository-1.6.6.pom
Downloaded: http://maven.aliyun.com/nexus/content/groups/public/org/apache/felix/org.apache.felix.bundlerepository/1.6.6/org.apache.felix.bundlerepository-1.6.6.pom (6 KB at 8.2 KB/sec)
Downloading: http://maven.aliyun.com/nexus/content/groups/public/org/osgi/org.osgi.core/4.1.0/org.osgi.core-4.1.0.pom
时间漫长.....
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /opt/hadoop-2.7.2-src/hadoop-dist/target/hadoop-dist-2.7.2-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................ SUCCESS [2:48.338s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [1:08.726s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [56.553s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.165s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [31.233s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [45.739s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [4:27.282s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [2:02.521s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [14.281s]
[INFO] Apache Hadoop Common .............................. SUCCESS [5:31.863s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [6.101s]
[INFO] Apache Hadoop KMS ................................. SUCCESS [9:40.036s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.110s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [3:09.067s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [7:23.052s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [1:00.818s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [4.777s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.042s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.043s]
[INFO] hadoop-yarn-api ................................... SUCCESS [37.515s]
[INFO] hadoop-yarn-common ................................ SUCCESS [2:13.058s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.086s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [9.417s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [19.177s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [3.238s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [6.699s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [19.363s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [4.747s]
[INFO] hadoop-yarn-client ................................ SUCCESS [5.866s]
[INFO] hadoop-yarn-server-sharedcachemanager ............. SUCCESS [3.250s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.042s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [3.084s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [1.829s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.033s]
[INFO] hadoop-yarn-registry .............................. SUCCESS [4.886s]
[INFO] hadoop-yarn-project ............................... SUCCESS [5.978s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.089s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [19.877s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [17.751s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.625s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [8.433s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [5.847s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [23.013s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [1.687s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [5.189s]
[INFO] hadoop-mapreduce .................................. SUCCESS [2.996s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [9.337s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [24.982s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [3.055s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [5.380s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [4.690s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [2.553s]
[INFO] Apache Hadoop Ant Tasks ........................... SUCCESS [2.073s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [2.967s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [10.646s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [4.477s]
[INFO] Apache Hadoop Amazon Web Services support ......... SUCCESS [1:30.319s]
[INFO] Apache Hadoop Azure support ....................... SUCCESS [16.141s]
[INFO] Apache Hadoop Client .............................. SUCCESS [9.456s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.240s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [4.777s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [12.871s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.035s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [58.014s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------
[INFO] Total time: 51:49.207s [INFO] Finished at: Mon Jul 22 13:05:49 EDT 2019
[INFO] Final Memory: 118M/336M
[INFO] ------------------------------------------------------------------------
[root@localhost hadoop-2.7.2-src]#
出现则表示编译成功！！！
4. 成功的64位hadoop包在/opt/hadoop-2.7.2-src/hadoop-dist/target下
[root@hadoop101 target]# pwd
/opt/hadoop-2.7.2-src/hadoop-dist/target
