CentOS7.4 + Hadoop2.7.5安装配置管理（伪分布式）
1. 规划
1.1. 机器列表
|
NameNode |
SecondaryNameNode |
DataNodes |
|
192.168.1.80 |
192.168.1.80 |
192.168.1.80 |
|
|
|
|
1.2. 机器列表
|
机器IP |
主机名 |
用户组/用户 |
|
192.168.1.80 |
centoshadoop.smartmap.com |
hadoop/hadoop |
|
|
|
|
2. 添加用户
[root@centoshadoop ~]# useradd hadoop
[root@centoshadoop ~]# passwd hadoop
Changing password for user hadoop.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
[root@centoshadoop ~]#
3. 设置IP与服务器名映射
[root@centoshadoop ~]# vi /etc/hosts
192.168.1.80 centoshadoop centoshadoop.smartmap.com
并修改Windows中的hosts文件
C:\Windows\System32\drivers\etc
4. 免密码登录
4.1. 免密码登录范围
要求能通过免登录包括使用IP和主机名都能免密码登录：
1) NameNode能免密码登录所有的DataNode
2) SecondaryNameNode能免密码登录所有的DataNode
3) NameNode能免密码登录自己
4) SecondaryNameNode能免密码登录自己
5) NameNode能免密码登录SecondaryNameNode
6) SecondaryNameNode能免密码登录NameNode
7) DataNode能免密码登录自己
8) DataNode不需要配置免密码登录NameNode、SecondaryNameNode和其它DataNode。
4.2. SSH无密码配置
4.2.1. 所有节点准备密钥
4.2.1.1. 创建密钥对
[hadoop@centoshadoop ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:4hSyoWOuj5MS+ME0ukspvAbfowEbv8iL4ZhfFrETtns hadoop@centoshadoop.smartmap.com
The key's randomart image is:
+---[RSA 2048]----+
| |
| |
| * . |
| oo O . |
|+++.* o S |
|*B+. * . |
|=X+o+ E |
|%B+=o. |
|@X*. . |
+----[SHA256]-----+
[hadoop@centoshadoop ~]$ ls -la .ssh/
total 8
drwx------ 2 hadoop hadoop 38 Apr 16 21:00 .
drwx------ 3 hadoop hadoop 74 Apr 16 21:00 ..
-rw------- 1 hadoop hadoop 1675 Apr 16 21:00 id_rsa
-rw-r--r-- 1 hadoop hadoop 414 Apr 16 21:00 id_rsa.pub
[hadoop@centoshadoop ~]$
4.2.1.2. 创建存放公钥的文件
[hadoop@centoshadoop ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@centoshadoop ~]$ chmod 600 ~/.ssh/authorized_keys
[hadoop@centoshadoop ~]$ ls -la ~/.ssh/
total 12
drwx------ 2 hadoop hadoop 61 Apr 16 21:02 .
drwx------ 3 hadoop hadoop 74 Apr 16 21:00 ..
-rw------- 1 hadoop hadoop 414 Apr 16 21:02 authorized_keys
-rw------- 1 hadoop hadoop 1675 Apr 16 21:00 id_rsa
-rw-r--r-- 1 hadoop hadoop 414 Apr 16 21:00 id_rsa.pub
[hadoop@centoshadoop ~]$
4.2.2. 验证
[hadoop@centoshadoop ~]$ ssh hadoop@192.168.1.80
The authenticity of host '192.168.1.80 (192.168.1.80)' can't be established.
ECDSA key fingerprint is SHA256:lgN0eOtdLR2eqHh+fabe54DGpV08ZiWo9oWVS60aGzw.
ECDSA key fingerprint is MD5:28:c0:cf:21:35:29:3d:23:d3:62:ca:0e:82:7a:4b:af.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.80' (ECDSA) to the list of known hosts.
Last login: Mon Apr 16 21:00:08 2018 from 192.168.1.100
[hadoop@centoshadoop ~]$ exit
logout
Connection to 192.168.1.80 closed.
[hadoop@centoshadoop ~]$ ssh hadoop@192.168.1.80
Last login: Mon Apr 16 21:04:40 2018 from 192.168.1.80
[hadoop@centoshadoop ~]$ exit
logout
Connection to 192.168.1.80 closed.
[hadoop@centoshadoop ~]$
5. 上传软件
将下载好的JDK与hadoop包上传到服务器上
以root用户上传目录为 /opt/
6. 安装JDK
6.1. 解压JDK包
[hadoop@centoshadoop ~]$ su - root
Password:
Last login: Mon Apr 16 20:41:23 CST 2018 from 192.168.1.100 on pts/0
[root@centoshadoop ~]# cd /opt/
[root@centoshadoop opt]# ll
total 346708
-rw-r--r-- 1 root root 216929574 Apr 16 20:17 hadoop-2.7.5.tar.gz
-rw-r--r-- 1 root root 138094686 Apr 14 20:04 jdk-7u45-linux-x64.tar.gz
[root@centoshadoop opt]# mkdir java
[root@centoshadoop opt]# tar -zxvf jdk-7u45-linux-x64.tar.gz
jdk1.7.0_45/
jdk1.7.0_45/db/
jdk1.7.0_45/db/lib/
……
jdk1.7.0_45/man/man1/javadoc.1
jdk1.7.0_45/THIRDPARTYLICENSEREADME.txt
jdk1.7.0_45/COPYRIGHT
[root@centoshadoop opt]# ls -la
total 346708
drwxr-xr-x. 4 root root 97 Apr 16 21:14 .
dr-xr-xr-x. 17 root root 244 Apr 16 20:35 ..
-rw-r--r-- 1 root root 216929574 Apr 16 20:17 hadoop-2.7.5.tar.gz
drwxr-xr-x 2 root root 6 Apr 16 21:13 java
drwxr-xr-x 8 10 143 233 Oct 8 2013 jdk1.7.0_45
-rw-r--r-- 1 root root 138094686 Apr 14 20:04 jdk-7u45-linux-x64.tar.gz
[root@centoshadoop opt]# mv jdk1.7.0_45/ java/
[root@centoshadoop opt]# ls -la
total 346708
drwxr-xr-x. 3 root root 78 Apr 16 21:17 .
dr-xr-xr-x. 17 root root 244 Apr 16 20:35 ..
-rw-r--r-- 1 root root 216929574 Apr 16 20:17 hadoop-2.7.5.tar.gz
drwxr-xr-x 3 root root 25 Apr 16 21:17 java
-rw-r--r-- 1 root root 138094686 Apr 14 20:04 jdk-7u45-linux-x64.tar.gz
[root@centoshadoop opt]# cd java/
[root@centoshadoop java]# ls -la
total 0
drwxr-xr-x 3 root root 25 Apr 16 21:17 .
drwxr-xr-x. 3 root root 78 Apr 16 21:17 ..
drwxr-xr-x 8 10 143 233 Oct 8 2013 jdk1.7.0_45
[root@centoshadoop java]# cd ..
[root@centoshadoop opt]#
6.2. 配置环境变量
[root@centoshadoop opt]# vi /etc/profile
在其结尾加入如下的内容
export JAVA_HOME=/opt/java/jdk1.7.0_45
export JRE_HOME=/opt/java/jdk1.7.0_45/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
[root@centoshadoop opt]# source /etc/profile
[root@centoshadoop opt]# java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
[root@centoshadoop opt]#
[root@centoshadoop opt]# rm -rf jdk-7u45-linux-x64.tar.gz
[root@centoshadoop opt]#
7. Hadoop安装及其环境配置
7.1. 解压Hadoop包
[root@centoshadoop opt]# mkdir hadoop
[root@centoshadoop opt]# ls -la
total 211848
drwxr-xr-x. 4 root root 59 Apr 16 21:28 .
dr-xr-xr-x. 17 root root 244 Apr 16 20:35 ..
drwxr-xr-x 2 root root 6 Apr 16 21:28 hadoop
-rw-r--r-- 1 root root 216929574 Apr 16 20:17 hadoop-2.7.5.tar.gz
drwxr-xr-x 3 root root 25 Apr 16 21:17 java
[root@centoshadoop opt]# tar -zxvf hadoop-2.7.5.tar.gz
……
hadoop-2.7.5/share/doc/hadoop-project/hadoop-datajoin/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-ant/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-extras/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-pipes/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-openstack/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-aws/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-azure/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-minicluster/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-sls/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-tools/
hadoop-2.7.5/share/doc/hadoop-project/hadoop-dist/
hadoop-2.7.5/share/doc/hadoop-project/
[root@centoshadoop opt]# ls -la
total 211848
drwxr-xr-x. 5 root root 79 Apr 16 21:31 .
dr-xr-xr-x. 17 root root 244 Apr 16 20:35 ..
drwxr-xr-x 2 root root 6 Apr 16 21:28 hadoop
drwxr-xr-x 9 20415 101 149 Dec 16 09:12 hadoop-2.7.5
-rw-r--r-- 1 root root 216929574 Apr 16 20:17 hadoop-2.7.5.tar.gz
drwxr-xr-x 3 root root 25 Apr 16 21:17 java
[root@centoshadoop opt]# mv hadoop-2.7.5/ hadoop/
[root@centoshadoop opt]# ls -la
total 211848
drwxr-xr-x. 4 root root 59 Apr 16 21:32 .
dr-xr-xr-x. 17 root root 244 Apr 16 20:35 ..
drwxr-xr-x 3 root root 26 Apr 16 21:32 hadoop
-rw-r--r-- 1 root root 216929574 Apr 16 20:17 hadoop-2.7.5.tar.gz
drwxr-xr-x 3 root root 25 Apr 16 21:17 java
[root@centoshadoop opt]# rm -rf hadoop-2.7.5.tar.gz
7.2. 修改Hadoop包的用户
[root@centoshadoop opt]# chown hadoop:hadoop -R hadoop/
[root@centoshadoop opt]#
7.3. Hadoop环境配置
7.3.1. 把Hadoop的安装路径添加到”/etc/profile”中
[root@centoshadoop opt]# vi /etc/profile
# export JAVA_HOME=/opt/java/jdk1.7.0_45
# export JRE_HOME=/opt/java/jdk1.7.0_45/jre
# export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
# export PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME=/opt/java/jdk1.7.0_45
export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH
export HADOOP_HOME=/opt/hadoop/hadoop-2.7.5
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:/usr/local/lib:$HADOOP_HOME/lib/native
export JAVA_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH
[root@centoshadoop opt]# source /etc/profile
[root@centoshadoop opt]#
7.3.2. 创建目录
[root@centoshadoop opt]# exit
logout
[hadoop@centoshadoop ~]$ cd /opt/
[hadoop@centoshadoop opt]$
[hadoop@centoshadoop opt]# mkdir /opt/hadoop/tmp
[hadoop@centoshadoop opt]# mkdir -p /opt/hadoop/data/hdfs/name
[hadoop@centoshadoop opt]# mkdir -p /opt/hadoop/data/hdfs/data
[hadoop@centoshadoop opt]# mkdir -p /opt/hadoop/data/hdfs/checkpoint
[hadoop@centoshadoop opt]# mkdir -p /opt/hadoop/data/yarn/local
[hadoop@centoshadoop opt]# mkdir -p /opt/hadoop/pid
[hadoop@centoshadoop opt]# mkdir -p /opt/hadoop/log
[hadoop@centoshadoop opt]#
7.3.3. 配置hadoop-env.sh
[hadoop@centoshadoop opt]$ vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/hadoop-env.sh
[hadoop@centoshadoop opt]$
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/opt/java/jdk1.7.0_45
export HADOOP_PID_DIR=/home/hadoop/pid
export HADOOP_LOG_DIR=/home/hadoop/log
[hadoop@centoshadoop opt]$ source /etc/profile
[hadoop@centoshadoop opt]$ source /opt/hadoop/hadoop-2.7.5/etc/hadoop/hadoop-env.sh
[hadoop@centoshadoop opt]$ hadoop version
Hadoop 2.7.5
Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r 18065c2b6806ed4aa6a3187d77cbe21bb3dba075
Compiled by kshvachk on 2017-12-16T01:06Z
Compiled with protoc 2.5.0
From source with checksum 9f118f95f47043332d51891e37f736e9
This command was run using /opt/hadoop/hadoop-2.7.5/share/hadoop/common/hadoop-common-2.7.5.jar
[hadoop@centoshadoop opt]$
7.3.4. 配置yarn-env.sh
[hadoop@centoshadoop opt]$ vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/yarn-env.sh
[hadoop@centoshadoop opt]$
# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
export JAVA_HOME=/opt/java/jdk1.7.0_45
7.3.5. 配置etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop/tmp</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.1.80:9000</value>
</property>
</configuration>
7.3.6. 配置etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>192.168.1.80:50070</value>
</property>
<property>
<name>dfs.namenode.http-bind-host</name>
<value>192.168.1.80</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>192.168.1.80:50090</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop/data/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/hadoop/data/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/opt/hadoop/data/hdfs/checkpoint</value>
</property>
</configuration>
7.3.7. 配置etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
7.3.8. 配置etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>192.168.1.80</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/opt/hadoop/data/yarn/local</value>
</property>
</configuration>
7.3.9. 配置etc/hadoop/slaves
[hadoop@centoshadoop hadoop]$ vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/slaves
[hadoop@centoshadoop hadoop]$
192.168.1.80
8. 启动
8.1. 启动HDFS
8.1.1. 格式化NameNode
[hadoop@centoshadoop hadoop]$ hdfs namenode -format
18/04/16 22:24:38 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = centoshadoop.smartmap.com/192.168.1.80
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.5
8.1.2. 启动与停止HDFS
[hadoop@centoshadoop hadoop]$ . /opt/hadoop/hadoop-2.7.5/sbin/start-dfs.sh
/usr/bin/which: invalid option -- 'b'
/usr/bin/which: invalid option -- 's'
/usr/bin/which: invalid option -- 'h'
Usage: /usr/bin/which [options] [--] COMMAND [...]
Write the full path of COMMAND(s) to standard output.
Report bugs to< which-bugs@gnu.org>.
Starting namenodes on [centoshadoop]
The authenticity of host 'centoshadoop (192.168.1.80)' can't be established.
ECDSA key fingerprint is SHA256:lgN0eOtdLR2eqHh+fabe54DGpV08ZiWo9oWVS60aGzw.
ECDSA key fingerprint is MD5:28:c0:cf:21:35:29:3d:23:d3:62:ca:0e:82:7a:4b:af.
Are you sure you want to continue connecting (yes/no)? yes
centoshadoop: Warning: Permanently added 'centoshadoop' (ECDSA) to the list of known hosts.
centoshadoop: starting namenode, logging to /home/hadoop/log/hadoop-hadoop-namenode-centoshadoop.smartmap.com.out
192.168.1.80: starting datanode, logging to /home/hadoop/log/hadoop-hadoop-datanode-centoshadoop.smartmap.com.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:lgN0eOtdLR2eqHh+fabe54DGpV08ZiWo9oWVS60aGzw.
ECDSA key fingerprint is MD5:28:c0:cf:21:35:29:3d:23:d3:62:ca:0e:82:7a:4b:af.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/log/hadoop-hadoop-secondarynamenode-centoshadoop.smartmap.com.out
[hadoop@centoshadoop hadoop]$ jps
8210 NameNode
8334 DataNode
8611 Jps
8495 SecondaryNameNode
[hadoop@centoshadoop hadoop]$
8.2. 启动YARN
[hadoop@centoshadoop opt]$ cd /opt/hadoop/hadoop-2.7.5
[hadoop@centoshadoop hadoop-2.7.5]$
[hadoop@centoshadoop hadoop-2.7.5]$ sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/hadoop-2.7.5/logs/yarn-hadoop-resourcemanager-centoshadoop.smartmap.com.out
192.168.1.80: starting nodemanager, logging to /opt/hadoop/hadoop-2.7.5/logs/yarn-hadoop-nodemanager-centoshadoop.smartmap.com.out
[hadoop@centoshadoop hadoop-2.7.5]$
[hadoop@centoshadoop hadoop-2.7.5]$ jps
11478 NodeManager
11373 ResourceManager
10580 SecondaryNameNode
10413 DataNode
10283 NameNode
11792 Jps
[hadoop@centoshadoop hadoop-2.7.5]$
设置logger级别，看下具体原因
export HADOOP_ROOT_LOGGER=DEBUG,console
8.3. 查看节点情况
[hadoop@centoshadoop hadoop-2.7.5]$ hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
18/04/16 23:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Configured Capacity: 38635831296 (35.98 GB)
Present Capacity: 37050920960 (34.51 GB)
DFS Remaining: 37050908672 (34.51 GB)
DFS Used: 12288 (12 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (1):
Name: 192.168.1.80:50010 (centoshadoop)
Hostname: centoshadoop
Decommission Status : Normal
Configured Capacity: 38635831296 (35.98 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 1584910336 (1.48 GB)
DFS Remaining: 37050908672 (34.51 GB)
DFS Used%: 0.00%
DFS Remaining%: 95.90%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Apr 16 23:17:13 CST 2018
[hadoop@centoshadoop hadoop-2.7.5]$
8.4. 通过网页查看集群
http://192.168.1.80:50070/dfshealth.html#tab-overview
http://192.168.1.80:8088/cluster
9. 如何运行自带wordcount
9.1. 找到该程序所在的位置
[hadoop@namenode sbin]$ cd /opt/hadoop/hadoop-2.7.5/share/hadoop/mapreduce/
[hadoop@namenode mapreduce]$ ll
total 5196
-rw-r--r-- 1 hadoop hadoop 571621 Nov 14 07:28 hadoop-mapreduce-client-app-2.9.0.jar
-rw-r--r-- 1 hadoop hadoop 787871 Nov 14 07:28 hadoop-mapreduce-client-common-2.9.0.jar
-rw-r--r-- 1 hadoop hadoop 1611701 Nov 14 07:28 hadoop-mapreduce-client-core-2.9.0.jar
-rw-r--r-- 1 hadoop hadoop 200628 Nov 14 07:28 hadoop-mapreduce-client-hs-2.9.0.jar
-rw-r--r-- 1 hadoop hadoop 32802 Nov 14 07:28 hadoop-mapreduce-client-hs-plugins-2.9.0.jar
-rw-r--r-- 1 hadoop hadoop 71360 Nov 14 07:28 hadoop-mapreduce-client-jobclient-2.9.0.jar
-rw-r--r-- 1 hadoop hadoop 1623508 Nov 14 07:28 hadoop-mapreduce-client-jobclient-2.9.0-tests.jar
-rw-r--r-- 1 hadoop hadoop 85175 Nov 14 07:28 hadoop-mapreduce-client-shuffle-2.9.0.jar
-rw-r--r-- 1 hadoop hadoop 303317 Nov 14 07:28 hadoop-mapreduce-examples-2.9.0.jar
drwxr-xr-x 2 hadoop hadoop 4096 Nov 14 07:28 jdiff
drwxr-xr-x 2 hadoop hadoop 4096 Nov 14 07:28 lib
drwxr-xr-x 2 hadoop hadoop 30 Nov 14 07:28 lib-examples
drwxr-xr-x 2 hadoop hadoop 4096 Nov 14 07:28 sources
[hadoop@namenode mapreduce]$
9.2. 准备数据
9.2.1. 先在HDFS创建几个数据目录
[hadoop@namenode mapreduce]$ hadoop fs -mkdir -p /data/wordcount
[hadoop@namenode mapreduce]$ hadoop fs -mkdir -p /output/
[hadoop@namenode mapreduce]$
目录/data/wordcount用来存放Hadoop自带的WordCount例子的数据文件，运行这个MapReduce任务的结果输出到/output/wordcount目录中
9.2.2. 创建测试数据
[hadoop@namenode mapreduce]$ vi /opt/hadoop/inputWord
9.2.3. 上传数据
[hadoop@namenode mapreduce]$ hadoop fs -put /opt/hadoop/inputWord /data/wordcount/
9.2.4. 显示文件
[hadoop@namenode mapreduce]$ hadoop fs -ls /data/wordcount/
Found 1 items
-rw-r--r-- 2 hadoop supergroup 241 2018-04-15 22:01 /data/wordcount/inputWord
9.2.5. 查看文件内容
[hadoop@namenode mapreduce]$ hadoop fs -text /data/wordcount/inputWord
hello smartmap
hello gispathfinder
language c
language c++
language c#
language java
language python
language r
web javascript
web html
web css
gis arcgis
gis supermap
gis mapinfo
gis qgis
gis grass
gis sage
webgis leaflet
webgis openlayers
[hadoop@namenode mapreduce]$
9.3. 运行WordCount例子
[hadoop@namenode mapreduce]$ hadoop jar /opt/hadoop/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jar wordcount /data/wordcount /output/wordcount
18/04/15 22:10:08 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.121:8032
18/04/15 22:10:09 INFO input.FileInputFormat: Total input files to process : 1
18/04/15 22:10:09 INFO mapreduce.JobSubmitter: number of splits:1
18/04/15 22:10:09 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
18/04/15 22:10:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523794216379_0001
18/04/15 22:10:10 INFO impl.YarnClientImpl: Submitted application application_1523794216379_0001
18/04/15 22:10:10 INFO mapreduce.Job: The url to track the job: http://nameNode:8088/proxy/application_1523794216379_0001/
18/04/15 22:10:10 INFO mapreduce.Job: Running job: job_1523794216379_0001
18/04/15 22:10:20 INFO mapreduce.Job: Job job_1523794216379_0001 running in uber mode : false
18/04/15 22:10:20 INFO mapreduce.Job: map 0% reduce 0%
18/04/15 22:10:30 INFO mapreduce.Job: map 100% reduce 0%
18/04/15 22:10:40 INFO mapreduce.Job: map 100% reduce 100%
18/04/15 22:10:40 INFO mapreduce.Job: Job job_1523794216379_0001 completed successfully
18/04/15 22:10:40 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=305
FILE: Number of bytes written=404363
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=356
HDFS: Number of bytes written=203
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=7775
Total time spent by all reduces in occupied slots (ms)=6275
Total time spent by all map tasks (ms)=7775
Total time spent by all reduce tasks (ms)=6275
Total vcore-milliseconds taken by all map tasks=7775
Total vcore-milliseconds taken by all reduce tasks=6275
Total megabyte-milliseconds taken by all map tasks=7961600
Total megabyte-milliseconds taken by all reduce tasks=6425600
Map-Reduce Framework
Map input records=19
Map output records=38
Map output bytes=393
Map output materialized bytes=305
Input split bytes=115
Combine input records=38
Combine output records=24
Reduce input groups=24
Reduce shuffle bytes=305
Reduce input records=24
Reduce output records=24
Spilled Records=48
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=1685
CPU time spent (ms)=2810
Physical memory (bytes) snapshot=476082176
Virtual memory (bytes) snapshot=1810911232
Total committed heap usage (bytes)=317718528
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=241
File Output Format Counters
Bytes Written=203
[hadoop@namenode mapreduce]$
9.4. 查看结果
9.4.1. 通过服务器上查看
[hadoop@namenode mapreduce]$ hadoop fs -ls /output/wordcount
Found 2 items
-rw-r--r-- 2 hadoop supergroup 0 2018-04-15 22:10 /output/wordcount/_SUCCESS
-rw-r--r-- 2 hadoop supergroup 203 2018-04-15 22:10 /output/wordcount/part-r-00000
[hadoop@namenode mapreduce]$ hadoop fs -text /output/wordcount/part-r-00000
arcgis 1
c 1
c# 1
c++ 1
css 1
gis 6
gispathfinder 1
grass 1
hello 2
html 1
java 1
javascript 1
language 6
leaflet 1
mapinfo 1
openlayers 1
python 1
qgis 1
r 1
sage 1
smartmap 1
supermap 1
web 3
webgis 2
[hadoop@namenode mapreduce]$
9.4.2. 通过Web UI查看
http://192.168.1.121:50070/explorer.html#/output/wordcount
