![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105015648-651358795.png)
- 我们使用之前搭建好的Hadoop环境,可参见:
《【Hadoop环境搭建】Centos6.8搭建hadoop伪分布模式》http://www.cnblogs.com/ssslinppp/p/5923793.html
- 示例程序为《Hadoop权威指南3》中的获取最高温度的示例程序;
数据准备
输入数据为:sample.txt
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105015961-1129264146.png)
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
将samle.txt上传至HDFS
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105016898-1885763489.png)
hadoop fs -put /home/hadoop/ncdcData/sample.txt input
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105017602-868549862.png)
项目结构
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105017961-780555563.png)
MaxTemperatureMapper类
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105018992-1239023919.png)
package com.ll.maxTemperature;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class MaxTemperatureMapper extends
Mapper<LongWritable, Text, Text, IntWritable> {
private static final int MISSING = 9999;
@Override
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String year = line.substring(15, 19);
int airTemperature;
if (line.charAt(87) == '+') { // parseInt doesn't like leading plus
// signs
airTemperature = Integer.parseInt(line.substring(88, 92));
} else {
airTemperature = Integer.parseInt(line.substring(87, 92));
}
String quality = line.substring(92, 93);
if (airTemperature != MISSING && quality.matches("[01459]")) {
context.write(new Text(year), new IntWritable(airTemperature));
}
}
}
// ^^ MaxTemperatureMapper
MaxTemperatureReducer类
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105021274-722548677.png)
package com.ll.maxTemperature;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
public class MaxTemperatureReducer extends
Reducer<Text, IntWritable, Text, IntWritable> {
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int maxValue = Integer.MIN_VALUE;
for (IntWritable value : values) {
maxValue = Math.max(maxValue, value.get());
}
context.write(key, new IntWritable(maxValue));
}
}
// ^^ MaxTemperatureReducer
MaxTemperature类(主函数)
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105024149-1214107497.png)
package com.ll.maxTemperature;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class MaxTemperature {
public static void main(String[] args) throws Exception {
if (args.length != 2) {
args = new String[] {
"hdfs://localhost:9000/user/hadoop/input/sample.txt",
"hdfs://localhost:9000/user/hadoop/out2" };
}
Job job = new Job(); // 指定作业执行规范
job.setJarByClass(MaxTemperature.class);
job.setJobName("Max temperature");
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1])); // Reduce函数输出文件的写入路径
job.setMapperClass(MaxTemperatureMapper.class);
job.setCombinerClass(MaxTemperatureReducer.class);
job.setReducerClass(MaxTemperatureReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
// ^^ MaxTemperature
解释说明:
输入路径为:hdfs://localhost:9000/user/hadoop/input/sample.txt
这部分由两部分组成:
- hdfs://localhost:9000/;
- /user/hadoop/input/sample.txt
其中hdfs://localhost:9000/由文件core-size.xml进行设置:
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105027665-532082178.png)
其中/user/hadoop/input/sample.txt就是上面准备数据时sample.txt存放的路径:
输出路径为:hdfs://localhost:9000/user/hadoop/out2
需要注意的是,在执行MapReduce时,这个输出路径一定不要存在,否则会出错。
pom.xml
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105028884-1663196301.png)
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.ll</groupId>
<artifactId>MapReduceTest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>MapReduceTest</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<hadoopVersion>1.2.1</hadoopVersion>
<junit.version>3.8.1</junit.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<!-- Hadoop -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${hadoopVersion}</version>
<!-- Hadoop -->
</dependency>
</dependencies>
</project>
程序测试
Hadoop环境准备
我们使用之前搭建好的Hadoop环境,可参见:
《【Hadoop环境搭建】Centos6.8搭建hadoop伪分布模式》http://www.cnblogs.com/ssslinppp/p/5923793.html
生成jar包
下面是生成jar包过程
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105031103-1196376461.png)
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105031509-1814748862.png)
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105032431-168444936.png)
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105034197-2835706.png)
上传服务器并运行测试
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105034618-547288166.png)
使用默认的输入输出路径:
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105034931-259248170.png)
hadoop jar mc.jar
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105035728-1847141383.png)
指定输入输出路径:
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105036306-1437862416.png)
hadoop jar /home/hadoop/jars/mc.jar hdfs://localhost:9000/user/hadoop/input/sample.txt hdfs://localhost:9000/user/hadoop/out5
![](https://images2015.cnblogs.com/blog/731047/201610/731047-20161009105038212-75974913.png)
转载请标明出处:http://www.cnblogs.com/ssslinppp/