hadoop 遇到java.net.ConnectException: to 0.0.0.0:10020 failed on connection
这个问题一般是在hadoop2.x版本里会出现，Hadoop的datanode需要访问namenode的jobhistory server,如果没有修改，则默认为0.0.0.0:10020，则可以修改mapred-site.xml文件：
<property>
<name>mapreduce.jobhistory.address</name>
<!-- 配置实际的Master主机名和端口-->
<value>0.0.0.0:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<!-- 配置实际的Master主机名和端口-->
<value>0.0.0.0:19888</value>
</property>
最后不要忘记了启动jobhistory
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
********************************************************************
a:C:\words.txt 文件内容:
hello alamp s
hello qq
hello xx
hello aa
hello swk
hello zbj
hello blm
blm xixi
zbj hehe
swk haha
b. java
@Test
public void testUpload() throws IllegalArgumentException, IOException {
FSDataOutputStream out = fs.create(new Path(
"hdfs://itcast01:9000/upload2"));
FileInputStream in = new FileInputStream(new File("c:/words.txt"));
IOUtils.copyBytes(in, out, 2048, true);
}
package cn.itcast.hadoop.mr;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.setInt("mapreduce.client.submit.file.replication", 20);
Job job = Job.getInstance(conf);
//notice
job.setJarByClass(WordCount.class);
//set mapper`s property
job.setMapperClass(WCMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
FileInputFormat.setInputPaths(job, new Path("/upload2/"));
//set reducer`s property
job.setReducerClass(WCReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
FileOutputFormat.setOutputPath(job, new Path("/usr/mapperReduce"));
//submit
job.waitForCompletion(true);
}
}
c.打jar包
[root@itcast01 usr]# hadoop jar mr.jar
17/05/20 16:44:09 INFO client.RMProxy: Connecting to ResourceManager at itcast01/192.168.233.128:8032
17/05/20 16:44:10 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/05/20 16:44:11 INFO input.FileInputFormat: Total input paths to process : 1
17/05/20 16:44:11 INFO mapreduce.JobSubmitter: number of splits:1
17/05/20 16:44:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1495288787912_0006
17/05/20 16:44:12 INFO impl.YarnClientImpl: Submitted application application_1495288787912_0006
17/05/20 16:44:12 INFO mapreduce.Job: The url to track the job: http://itcast01:8088/proxy/application_1495288787912_0006/
17/05/20 16:44:12 INFO mapreduce.Job: Running job: job_1495288787912_0006
17/05/20 16:44:55 INFO mapreduce.Job: Job job_1495288787912_0006 running in uber mode : false
17/05/20 16:44:55 INFO mapreduce.Job: map 0% reduce 0%
17/05/20 16:46:24 INFO mapreduce.Job: map 100% reduce 0%
17/05/20 16:47:02 INFO mapreduce.Job: map 100% reduce 100%
17/05/20 16:47:03 INFO mapreduce.Job: Job job_1495288787912_0006 completed successfully
17/05/20 16:47:06 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=435
FILE: Number of bytes written=186471
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=210
HDFS: Number of bytes written=78
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=65092
Total time spent by all reduces in occupied slots (ms)=32649
Total time spent by all map tasks (ms)=65092
Total time spent by all reduce tasks (ms)=32649
Total vcore-seconds taken by all map tasks=65092
Total vcore-seconds taken by all reduce tasks=32649
Total megabyte-seconds taken by all map tasks=66654208
Total megabyte-seconds taken by all reduce tasks=33432576
Map-Reduce Framework
Map input records=17
Map output records=32
Map output bytes=365
Map output materialized bytes=435
Input split bytes=93
Combine input records=0
Combine output records=0
Reduce input groups=13
Reduce shuffle bytes=435
Reduce input records=32
Reduce output records=13
Spilled Records=64
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=290
CPU time spent (ms)=5530
Physical memory (bytes) snapshot=284258304
Virtual memory (bytes) snapshot=1685770240
Total committed heap usage (bytes)=136515584
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=117
File Output Format Counters
Bytes Written=78
[root@itcast01 usr]# hadoop fs -ls /
Found 9 items
-rw-r--r-- 1 root supergroup 153512879 2017-05-17 05:34 /jdk
-rw-r--r-- 1 root supergroup 62 2017-05-20 13:54 /score_in
drwx------ - root supergroup 0 2017-05-17 06:15 /tmp
-rw-r--r-- 3 root supergroup 32 2017-05-17 14:47 /upload
-rw-r--r-- 3 root supergroup 117 2017-05-20 16:35 /upload2
drwxr-xr-x - root supergroup 0 2017-05-20 16:44 /usr
drwxr-xr-x - root supergroup 0 2017-05-17 06:18 /wcout
-rw-r--r-- 1 root supergroup 70 2017-05-17 06:12 /words
drwxr-xr-x - root supergroup 0 2017-05-20 15:23 /xx
[root@itcast01 usr]# hadoop fs -cat /usr/upload2/part-r-00000
cat: `/usr/upload2/part-r-00000': No such file or directory
[root@itcast01 usr]# hadoop fs -ls /usr/
Found 4 items
drwxr-xr-x - root supergroup 0 2017-05-17 14:54 /usr/local
drwxr-xr-x - root supergroup 0 2017-05-20 16:47 /usr/mapperReduce
drwxr-xr-x - root supergroup 0 2017-05-20 16:08 /usr/swk
drwxr-xr-x - root supergroup 0 2017-05-17 11:25 /usr/test
[root@itcast01 usr]# hadoop fs -ls /usr/mapperReduce
Found 2 items
-rw-r--r-- 1 root supergroup 0 2017-05-20 16:47 /usr/mapperReduce/_SUCCESS
-rw-r--r-- 1 root supergroup 78 2017-05-20 16:46 /usr/mapperReduce/part-r-00000
[root@itcast01 usr]# hadoop fs -cat /usr/mapperReduce/part-r-00000
11
aa 1
alamp 1
blm 2
haha 1
hehe 1
hello 7
qq 1
s 1
swk 2
xixi 1
xx 1
zbj 2
[root@itcast01 usr]#
