Linux巩固记录（4） 运行hadoop 2.7.4自带demo程序验证环境

本节主要使用hadoop自带的程序运行demo来确认环境是否正常

1.首先创建一个input.txt文件，里面任意输入些单词，有部分重复单词

2.将input文件拷贝到hdfs

3.执行hadoop程序

4.查看结果

 

完整执行命令及返回结果看下面的执行拷贝

[root@master ~]# 
[root@master ~]# ll /home/input.txt 
-rw-r--r--. 1 root root 76 Sep  2 00:55 /home/input.txt
[root@master ~]# cat /home/input.txt 
this is a test
hello hadoop

hadoop is a xxxxx

from changw.xiao@qq.com[root@master ~]# 
[root@master ~]# 
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop fs -ls /
[root@master ~]# 
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop fs -copyFromLocal /home/input.txt /hdfs-input.txt
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop fs -ls /
Found 1 items
-rw-r--r--   2 root supergroup         76 2017-09-02 00:57 /hdfs-input.txt
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop fs -cat /hdfs-input.txt
this is a test
hello hadoop

hadoop is a xxxxx

from changw.xiao@qq.com[root@master ~]# 
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop jar /home/hadoop-2.7.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar wordcount /hdfs-input.txt /wordcount-result
17/09/02 00:59:28 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.80:8032
17/09/02 00:59:29 INFO input.FileInputFormat: Total input paths to process : 1
17/09/02 00:59:29 INFO mapreduce.JobSubmitter: number of splits:1
17/09/02 00:59:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504320356950_0001
17/09/02 00:59:31 INFO impl.YarnClientImpl: Submitted application application_1504320356950_0001
17/09/02 00:59:31 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504320356950_0001/
17/09/02 00:59:31 INFO mapreduce.Job: Running job: job_1504320356950_0001
17/09/02 00:59:44 INFO mapreduce.Job: Job job_1504320356950_0001 running in uber mode : false
17/09/02 00:59:44 INFO mapreduce.Job:  map 0% reduce 0%
17/09/02 00:59:53 INFO mapreduce.Job:  map 100% reduce 0%
17/09/02 01:00:00 INFO mapreduce.Job:  map 100% reduce 100%
17/09/02 01:00:01 INFO mapreduce.Job: Job job_1504320356950_0001 completed successfully
17/09/02 01:00:01 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=118
        FILE: Number of bytes written=241861
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=174
        HDFS: Number of bytes written=76
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=6234
        Total time spent by all reduces in occupied slots (ms)=4978
        Total time spent by all map tasks (ms)=6234
        Total time spent by all reduce tasks (ms)=4978
        Total vcore-milliseconds taken by all map tasks=6234
        Total vcore-milliseconds taken by all reduce tasks=4978
        Total megabyte-milliseconds taken by all map tasks=6383616
        Total megabyte-milliseconds taken by all reduce tasks=5097472
    Map-Reduce Framework
        Map input records=6
        Map output records=12
        Map output bytes=118
        Map output materialized bytes=118
        Input split bytes=98
        Combine input records=12
        Combine output records=9
        Reduce input groups=9
        Reduce shuffle bytes=118
        Reduce input records=9
        Reduce output records=9
        Spilled Records=18
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=173
        CPU time spent (ms)=1380
        Physical memory (bytes) snapshot=298201088
        Virtual memory (bytes) snapshot=4159512576
        Total committed heap usage (bytes)=139833344
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=76
    File Output Format Counters 
        Bytes Written=76
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop fs -ls /
Found 3 items
-rw-r--r--   2 root supergroup         76 2017-09-02 00:57 /hdfs-input.txt
drwx------   - root supergroup          0 2017-09-02 00:59 /tmp
drwxr-xr-x   - root supergroup          0 2017-09-02 00:59 /wordcount-result
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop fs -ls /wordcount-result
Found 2 items
-rw-r--r--   2 root supergroup          0 2017-09-02 00:59 /wordcount-result/_SUCCESS
-rw-r--r--   2 root supergroup         76 2017-09-02 00:59 /wordcount-result/part-r-00000
[root@master ~]# /home/hadoop-2.7.4/bin/hadoop fs -cat /wordcount-result/part-r-00000
a    2
changw.xiao@qq.com    1
from    1
hadoop    2
hello    1
is    2
test    1
this    1
xxxxx    1
[root@master ~]# 
[root@master ~]# 

 

/home/hadoop-2.7.4/bin/hadoop fs -copyFromLocal /home/input.txt /hdfs-input.txt   也可以用 -put