Hadoop集群基准测试

hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar TestDFSIO -write -nrFiles 10 -size 100MB

14/07/21 21:29:42 INFO mapreduce.Job: Counters: 45
        File System Counters
                FILE: Number of bytes read=856
                FILE: Number of bytes written=872121
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=2320
                HDFS: Number of bytes written=1048576079
                HDFS: Number of read operations=43
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=12
        Job Counters 
                Killed map tasks=3
                Launched map tasks=13
                Launched reduce tasks=1
                Data-local map tasks=6
                Rack-local map tasks=7
                Total time spent by all maps in occupied slots (ms)=1109317
                Total time spent by all reduces in occupied slots (ms)=91582
        Map-Reduce Framework
                Map input records=10
                Map output records=50
                Map output bytes=750
                Map output materialized bytes=910
                Input split bytes=1200
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=910
                Reduce input records=50
                Reduce output records=5
                Spilled Records=100
                Shuffled Maps =10
                Failed Shuffles=0
                Merged Map outputs=10
                GC time elapsed (ms)=7691
                CPU time spent (ms)=30730
                Physical memory (bytes) snapshot=2555801600
                Virtual memory (bytes) snapshot=11073003520
                Total committed heap usage (bytes)=1874329600
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=1120
        File Output Format Counters 
                Bytes Written=79
14/07/21 21:29:42 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
14/07/21 21:29:42 INFO fs.TestDFSIO:            Date & time: Mon Jul 21 21:29:42 CST 2014
14/07/21 21:29:42 INFO fs.TestDFSIO:        Number of files: 10
14/07/21 21:29:42 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
14/07/21 21:29:42 INFO fs.TestDFSIO:      Throughput mb/sec: 3.403479036270876
14/07/21 21:29:42 INFO fs.TestDFSIO: Average IO rate mb/sec: 4.484561443328857
14/07/21 21:29:42 INFO fs.TestDFSIO:  IO rate std deviation: 1.7674698292439088
14/07/21 21:29:42 INFO fs.TestDFSIO:     Test exec time sec: 190.944

 

hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar TestDFSIO -write -nrFiles 10 -size 1GB

14/07/21 22:09:57 INFO mapreduce.Job: Counters: 47
        File System Counters
                FILE: Number of bytes read=875
                FILE: Number of bytes written=872159
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=2320
                HDFS: Number of bytes written=10737418320
                HDFS: Number of read operations=43
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=12
        Job Counters 
                Failed map tasks=7
                Killed map tasks=1
                Launched map tasks=18
                Launched reduce tasks=1
                Other local map tasks=8
                Data-local map tasks=6
                Rack-local map tasks=4
                Total time spent by all maps in occupied slots (ms)=4322906
                Total time spent by all reduces in occupied slots (ms)=155182
        Map-Reduce Framework
                Map input records=10
                Map output records=50
                Map output bytes=769
                Map output materialized bytes=929
                Input split bytes=1200
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=929
                Reduce input records=50
                Reduce output records=5
                Spilled Records=100
                Shuffled Maps =10
                Failed Shuffles=0
                Merged Map outputs=10
                GC time elapsed (ms)=26386
                CPU time spent (ms)=119850
                Physical memory (bytes) snapshot=2830766080
                Virtual memory (bytes) snapshot=11052699648
                Total committed heap usage (bytes)=2014642176
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=1120
        File Output Format Counters 
                Bytes Written=80
14/07/21 22:09:59 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
14/07/21 22:10:00 INFO fs.TestDFSIO:            Date & time: Mon Jul 21 22:09:59 CST 2014
14/07/21 22:10:00 INFO fs.TestDFSIO:        Number of files: 10
14/07/21 22:10:00 INFO fs.TestDFSIO: Total MBytes processed: 10240.0
14/07/21 22:10:00 INFO fs.TestDFSIO:      Throughput mb/sec: 6.388915232072354
14/07/21 22:10:00 INFO fs.TestDFSIO: Average IO rate mb/sec: 11.30741024017334
14/07/21 22:10:00 INFO fs.TestDFSIO:  IO rate std deviation: 7.81592713288423
14/07/21 22:10:00 INFO fs.TestDFSIO:     Test exec time sec: 556.575

　　

环境信息：

x220, i5-2430M, 4G*2, 320G

Windows 7 Ultimate, Vmware 8

Virtual Workstation: 2 cores, 2G, 40G

Hadoop 2.2.0