基于HBase Hadoop 分布式集群环境下的MapReduce程序开发
HBase分布式集群环境搭建成功后，连续4、5天实验客户端Map/Reduce程序开发，这方面的代码网上多得是，写个测试代码非常容易，可是真正运行起来可说是历经挫折。下面就是我最终调通并让程序在集群上运行起来的一些经验教训。
一、首先说一下我的环境：
1，集群的环境配置请见这篇博文。
2，开发客户机环境：操作系统是CentOS6.5，JDK版本是1.7.0-60，开发工具是Eclipse（原始安装是从google的ADT网站下载的ADT专用开发环境，后来加装了Java企业开发的工具，启动Flash还是ADT的：））。
3，etc/hosts文件添加了集群的所有主机。
二、必要准备工作：
1，java类库准备：从集群中scp复制了Hadoop/share下所有文件，hbase/lib目录下的所有文件到开发机的/home/username/lib目录下备用(其中username是开发用户名称)，打开eclipse，选择菜单window->Preferences,
在左侧树形菜单中导航至Java--Build Path--User Libraries，我添加了3个User Library，“hadoop-common”，“hadoop-yarn”，“hbase”
然后分别添加如下（这是User Libraries的导出文件，可以在上面的界面中导入）的jar包。
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<eclipse-userlibraries version="2">
<library name="hadoop-common" systemlibrary="false">
<archive path="/home/gzg/javadev/lib/hadoop/mapreduce/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/hadoop-auth-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/hadoop-annotations-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/hadoop-common-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-logging-1.1.3.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-compress-1.4.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/guava-11.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-collections-3.2.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/slf4j-api-1.7.5.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/slf4j-log4j12-1.7.5.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/zookeeper-3.4.5-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-configuration-1.6.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-math3-3.1.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/activation-1.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/asm-3.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/avro-1.7.5-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-1.7.0.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-core-1.8.0.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-cli-1.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-codec-1.4.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-digester-1.8.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-el-1.0.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-httpclient-3.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-io-2.4.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-lang-2.6.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-net-3.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/httpclient-4.2.5.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/httpcore-4.2.5.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-core-asl-1.8.8.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-jaxrs-1.8.8.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-xc-1.8.8.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jasper-compiler-5.5.23.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jasper-runtime-5.5.23.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/java-xmlbuilder-0.4.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jaxb-api-2.2.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jaxb-impl-2.2.3-1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-core-1.9.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-json-1.9.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-server-1.9.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jets3t-0.9.0.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jettison-1.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jetty-6.1.26.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jetty-util-6.1.26.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsch-0.1.42.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsp-api-2.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsr305-1.3.9.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/junit-4.8.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/log4j-1.2.17.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/mockito-all-1.8.5.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/paranamer-2.3.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/protobuf-java-2.5.0.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/servlet-api-2.5.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/snappy-java-1.0.4.1.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/stax-api-1.0-2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/xmlenc-0.52.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/common/lib/xz-1.0.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/mapreduce/lib/netty-3.6.2.Final.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/hdfs/hadoop-hdfs-2.3.0-cdh5.0.2.jar"/>
</library>
<library name="hadoop-yarn" systemlibrary="false">
<archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-common-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-shuffle-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-api-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-client-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-common-2.3.0-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.0.2.jar"/>
</library>
<library name="hbase" systemlibrary="false">
<archive path="/home/gzg/javadev/lib/hbase/hbase-client-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-common-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-hadoop-compat-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-it-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-protocol-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-testing-util-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-thrift-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/protobuf-java-2.5.0.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-server-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/hbase-shell-0.96.1.1-cdh5.0.2.jar"/>
<archive path="/home/gzg/javadev/lib/hbase/htrace-core-2.01.jar"/>
</library>
</eclipse-userlibraries>
注：这个很有可能并不是运行Hbase Map/Reduce任务需要的最精简的类库，恕我没有逐一测试。
2，新建一个Java/Application，我的名字就叫hbase1，然后在项目project视图中点击右键，选择“Build Path”--“Add Libraries...”--选择“User Library”，然后选中上面添加的三个User Library。
3，复制集群的三个配置文件到项目的src目录下，这三个配置文件是：
core-site.xml hdfs-site.xml yarn-site.xml
三、编写代码，这里基本按照我所遇挫折过程逐步完善代码（缺少类库之类的就略掉了）：
1，遭遇空指针错误--reverseDNS
主文件内容：
/**
* @author gzg
*
*/
public class Test1 {
/**
* @param args
* @throws IOException
* @throws ZooKeeperConnectionException
* @throws MasterNotRunningException
*/
Configuration conf=null;
public static void main(String[] args) {
try {
new TestHBaseMapred().run(false);
} catch(yh.utils.system.CommandError cep){
cep.printStackTrace();
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
TestHBaseMapRed.java内容：
public class TestHBaseMapred {
public void run(boolean toFile) throws ClassNotFoundException, IOException, InterruptedException{
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
//设置zookeeper的地址s
conf.set("hbase.zookeeper.quorum", "zk1,zk2,zk3");
//此设置好像没用
conf.set("hadoop.job.user", "hadoop");
org.apache.hadoop.mapreduce.Job job=null;
String tmpTable = "_mrtest1";
yh.utils.ElapsedTimeMeasure etm=yh.utils.ElapsedTimeMeasure.createAndStart();
try {
job = Job.getInstance(conf);
job.setJobName("count meter");
if(toFile==false){
HBaseUtil.dropTable(conf, tmpTable);
HBaseUtil.createTable(conf, tmpTable, "c");
job.setOutputFormatClass(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.class);
job.getConfiguration().set(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.OUTPUT_TABLE, tmpTable);
//
job.setReducerClass(TblReduce.class);
}else{
job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);
Path op = new Path("result");
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.setOutputPath(job, op);
//
job.setReducerClass(TxtReduce.class);
}
Scan scan = new Scan();
scan.addFamily(Bytes.toBytes("c"));
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob("dzdl_t1",scan,TblMap.class ,Text.class,LongWritable.class,job);
job.waitForCompletion(true);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("time elapsed :"+etm.stop().toString());
}
public static class TblMap extends org.apache.hadoop.hbase.mapreduce.TableMapper<Text, LongWritable>{
@Override
protected void map(ImmutableBytesWritable key,Result result,org.apache.hadoop.mapreduce.Mapper.Context ctx) throws IOException, InterruptedException{
Cell c=result.getColumnLatestCell(Bytes.toBytes("c"), Bytes.toBytes("col3"));
String col= Bytes.toString(CellUtil.cloneQualifier(c));
Text kv = new Text();
kv.set("count_result");
LongWritable lv = new LongWritable();
lv.set(1);
ctx.write(kv, lv);
//org.apache.log4j.Logger.getRootLogger().info(kv+" : 1");
}
}
public static class TblReduce extends org.apache.hadoop.hbase.mapreduce.TableReducer<Text,LongWritable,ImmutableBytesWritable>{
@Override
protected void reduce(Text key,Iterable<LongWritable> values,Context ctx) throws IOException, InterruptedException{
long t =0;
for(LongWritable v:values){
t+=v.get();
}
byte[] ok=Bytes.toBytes("count_result");
Put p=new Put(ok);
p.add(Bytes.toBytes("c"),key.copyBytes(),Bytes.toBytes(t));
p.add(Bytes.toBytes("c"),Bytes.toBytes("14112896"),Bytes.toBytes(14112896));
ctx.write(new ImmutableBytesWritable(ok),p);
}
}
public static class TxtReduce extends org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer<Text>{
@Override
public void reduce(Text key,Iterable<LongWritable> values,Context ctx) throws IOException, InterruptedException{
long t =0;
for(LongWritable v:values){
t+=v.get();
}
//Put p=new Put(key.copyBytes());
//p.add(Bytes.toBytes("c"),key.copyBytes(),Bytes.toBytes(t));
//ctx.write(new ImmutableBytesWritable(key.copyBytes()),p);
LongWritable l = new LongWritable();
l.set(t);
ctx.write(key,l);
}
}
}
此时运行，会报如下错误：
Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.net.DNS.reverseDns(DNS.java:92) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:218) at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:184) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313) at hbase1.TestHBaseMapred.run(TestHBaseMapred.java:82) at hbase1.Test1.main(Test1.java:69)
看错误，不知道为什么需要调用DNS服务，hosts文件好像没有作用， 搜索很久，终于在一个老外的问题里找到答案，conf添加如下设置解决。
conf.set("hbase.nameserver.address","nn1,nn2");
添加后，运行仍会报下面的错无，但是好像并不影响程序继续运行：
2014-08-21 14:05:04,158 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2: Name or service not known]; remaining name '111.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,170 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,171 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,172 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,173 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,173 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '111.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,174 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,174 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,175 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,175 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,199 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,200 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,201 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,201 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,202 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa' 2014-08-21 14:05:04,539 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:15
2，本地运行的问题。解决了上面的问题，程序终于可以运行了，可是我在rm的web界面中却看不到我提交的任务，再次求助bing，终于找到症结所在，需要添加设置如下：
//此项必须设置，否则程序只是在本地客户端上运行，通过rm的浏览器管理界面看不到！
conf.set("mapreduce.framework.name", "yarn");
3，添加了上面语句后，新的问题又来了，运行程序出现新的错误：
2014-08-21 14:15:08,887 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://rm2:23188/proxy/application_1408378794442_0014/ 2014-08-21 14:15:08,888 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_1408378794442_0014 2014-08-21 14:15:21,466 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0014 running in uber mode : false 2014-08-21 14:15:21,469 INFO [org.apache.hadoop.mapreduce.Job] - map 0% reduce 0% 2014-08-21 14:15:34,988 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_0, Status : FAILED 2014-08-21 14:15:35,023 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000011_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:39,485 INFO [org.apache.hadoop.mapreduce.Job] - map 7% reduce 0% 2014-08-21 14:15:39,490 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:39,497 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:39,505 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000001_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:40,663 INFO [org.apache.hadoop.mapreduce.Job] - map 0% reduce 0% 2014-08-21 14:15:40,736 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:42,783 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:44,921 INFO [org.apache.hadoop.mapreduce.Job] - map 7% reduce 0% 2014-08-21 14:15:44,925 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000005_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2014-08-21 14:15:46,019 INFO [org.apache.hadoop.mapreduce.Job] - map 0% reduce 0% 2014-08-21 14:15:46,066 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:46,070 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000002_0, Status : FAILED Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:46,073 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000009_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:46,076 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000014_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:47,086 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000007_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:15:59,146 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:00,237 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:01,312 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:01,329 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:03,396 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000008_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:03,401 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:04,488 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000009_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:04,493 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000001_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:05,530 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000010_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:06,546 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000012_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:09,672 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:10,935 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000014_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:19,239 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000012_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:19,243 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:19,246 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:20,307 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:21,326 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000011_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895) at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893) ... 8 more 2014-08-21 14:16:23,352 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2014-08-21 14:16:24,378 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0014 failed with state FAILED due to: Task failed task_1408378794442_0014_m_000013 Job failed as tasks failed. failedMaps:1 failedReduces:0 2014-08-21 14:16:24,584 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 13 Job Counters Failed map tasks=31 Killed map tasks=13 Launched map tasks=44 Other local map tasks=29 Data-local map tasks=15 Total time spent by all maps in occupied slots (ms)=1665094 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=832547 Total vcore-seconds taken by all map tasks=832547 Total megabyte-seconds taken by all map tasks=1278792192 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 *** measure elapsed time,stoped at 2014-08-21 14:16:24:588 time elapsed :0days 0hours 1minutes 29seconds 879milliseconds[89.879seconds]
试着设置 job.setJarByClass(Test1.class); 无效。此问题困扰了我两天，难道只能上传到集群集群上运行吗？这样也太不方便了，N久后，终于我有幸找到了小鱼儿的博客 的这篇文章MapReduce提交作业常见问题，终于是找到了解决方法。我采用的是文中所提第三种方法，在程序开始运行前自己首先打包自己，就是调用的是写好的shell文件，这里也贴出来，以下两个文件在和src同级的dist目录中
[gzg@centos dist]$ cat manifest.mf Manifest-Version: 1.0 Class-Path: . $HADOOP_HOME/lib hbase1_lib/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.ja r hbase1_lib/hadoop-auth-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-annotat ions-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-common-2.3.0-cdh5.0.2.jar h base1_lib/commons-logging-1.1.3.jar hbase1_lib/commons-compress-1.4.1 .jar hbase1_lib/guava-11.0.2.jar hbase1_lib/commons-collections-3.2.1 .jar hbase1_lib/slf4j-api-1.7.5.jar hbase1_lib/slf4j-log4j12-1.7.5.ja r hbase1_lib/zookeeper-3.4.5-cdh5.0.2.jar hbase1_lib/commons-configur ation-1.6.jar hbase1_lib/commons-math3-3.1.1.jar hbase1_lib/activatio n-1.1.jar hbase1_lib/asm-3.2.jar hbase1_lib/avro-1.7.5-cdh5.0.2.jar h base1_lib/commons-beanutils-1.7.0.jar hbase1_lib/commons-beanutils-co re-1.8.0.jar hbase1_lib/commons-cli-1.2.jar hbase1_lib/commons-codec- 1.4.jar hbase1_lib/commons-digester-1.8.jar hbase1_lib/commons-el-1.0 .jar hbase1_lib/commons-httpclient-3.1.jar hbase1_lib/commons-io-2.4. jar hbase1_lib/commons-lang-2.6.jar hbase1_lib/commons-net-3.1.jar hb ase1_lib/httpclient-4.2.5.jar hbase1_lib/httpcore-4.2.5.jar hbase1_li b/jackson-core-asl-1.8.8.jar hbase1_lib/jackson-jaxrs-1.8.8.jar hbase 1_lib/jackson-mapper-asl-1.8.8.jar hbase1_lib/jackson-xc-1.8.8.jar hb ase1_lib/jasper-compiler-5.5.23.jar hbase1_lib/jasper-runtime-5.5.23. jar hbase1_lib/java-xmlbuilder-0.4.jar hbase1_lib/jaxb-api-2.2.2.jar hbase1_lib/jaxb-impl-2.2.3-1.jar hbase1_lib/jersey-core-1.9.jar hbase 1_lib/jersey-json-1.9.jar hbase1_lib/jersey-server-1.9.jar hbase1_lib /jets3t-0.9.0.jar hbase1_lib/jettison-1.1.jar hbase1_lib/jetty-6.1.26 .jar hbase1_lib/jetty-util-6.1.26.jar hbase1_lib/jsch-0.1.42.jar hbas e1_lib/jsp-api-2.1.jar hbase1_lib/jsr305-1.3.9.jar hbase1_lib/junit-4 .8.2.jar hbase1_lib/log4j-1.2.17.jar hbase1_lib/mockito-all-1.8.5.jar hbase1_lib/paranamer-2.3.jar hbase1_lib/protobuf-java-2.5.0.jar hbas e1_lib/servlet-api-2.5.jar hbase1_lib/snappy-java-1.0.4.1.jar hbase1_ lib/stax-api-1.0-2.jar hbase1_lib/xmlenc-0.52.jar hbase1_lib/xz-1.0.j ar hbase1_lib/netty-3.6.2.Final.jar hbase1_lib/hadoop-hdfs-2.3.0-cdh5 .0.2.jar hbase1_lib/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2_2.jar hbase1_lib/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar hbase1_lib /hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar hbase1_lib/hado op-mapreduce-client-common-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-mapre duce-client-shuffle-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-api-2.3 .0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-client-2.3.0-cdh5.0.2.jar hbas e1_lib/hadoop-yarn-common-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-a pplications-distributedshell-2.3.0-cdh5.0.2.jar hbase1_lib/hbase-clie nt-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-common-0.96.1.1-cdh5.0.2.ja r hbase1_lib/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar hbase1_lib/hb ase-hadoop-compat-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-it-0.96.1.1- cdh5.0.2.jar hbase1_lib/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar hbase 1_lib/hbase-protocol-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-shell-0.9 6.1.1-cdh5.0.2.jar hbase1_lib/hbase-testing-util-0.96.1.1-cdh5.0.2.ja r hbase1_lib/hbase-thrift-0.96.1.1-cdh5.0.2.jar hbase1_lib/protobuf-j ava-2.5.0_2.jar hbase1_lib/htrace-core-2.01.jar hbase1_lib/hbase-serv er-0.96.1.1-cdh5.0.2.jar Main-Class: hbase1.Test1
[gzg@centos dist]$ cat makejar cd "$(dirname $0)" rm -f hbase1.jar jar cvfm hbase1.jar manifest.mf -C "../bin/" .
同时，修改了TestHBaseMapred.run方法签名，传递jar包路径，新的run方法如下：
public void run(String appJarFile,boolean toFile) throws ClassNotFoundException, IOException, InterruptedException{
//org.apache.hadoop.conf.Configuration conf = yh.utils.hbase.HBaseUtil.getConfiguration("zk1,zk2,zk3");
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.set("hbase.zookeeper.quorum", "zk1,zk2,zk3");
conf.set("hadoop.job.user", "hadoop");
conf.set("mapreduce.framework.name", "yarn");
conf.set("hbase.nameserver.address","nn1,nn2");
org.apache.hadoop.mapreduce.Job job=null;
String tmpTable = "_mrtest1";
yh.utils.ElapsedTimeMeasure etm=yh.utils.ElapsedTimeMeasure.createAndStart();
try {
job = Job.getInstance(conf);
job.setJobName("count meter");
//job.setJarByClass(Test1.class);
job.setJar(appJarFile);
if(toFile==false){
HBaseUtil.dropTable(conf, tmpTable);
HBaseUtil.createTable(conf, tmpTable, "c");
job.setOutputFormatClass(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.class);
job.getConfiguration().set(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.OUTPUT_TABLE, tmpTable);
//
job.setReducerClass(TblReduce.class);
}else{
job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);
Path op = new Path("result");
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.setOutputPath(job, op);
//
job.setReducerClass(TxtReduce.class);
}
Scan scan = new Scan();
scan.addFamily(Bytes.toBytes("c"));
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob("dzdl_t1",scan,TblMap.class ,Text.class,LongWritable.class,job);
job.waitForCompletion(true);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("time elapsed :"+etm.stop().toString());
}
4，在选择写入文件的Reduce方法是，同样遇到权限问题，我采用的解决方法是给所有用户授予了读写权限，这里注意/tmp也要授权，运行时用到
[hadoop@zk1 hbase]$ hdfs dfs -chmod 777 /user [hadoop@zk1 hbase]$ hdfs dfs -chmod 777 /tmp
另外，最后还有一点注意，设置log4j.properties文件中的级别为INFO，不要设置为WARN，否则看不到很多信息，不利于发现错误原因。
# Configure logging for testing: optionally with log file #log4j.rootLogger=WARN, stdout log4j.rootLogger=INFO, stdout # log4j.rootLogger=WARN, stdout, logfile log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n log4j.appender.logfile=org.apache.log4j.FileAppender log4j.appender.logfile.File=target/spring.log log4j.appender.logfile.layout=org.apache.log4j.PatternLayout log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n
四、At last，终于大功告成，看看map/reduce的运行过程
*** measure elapsed time,started at 2014-08-21 14:48:35:880 2014-08-21 14:48:36,601 WARN [org.apache.hadoop.util.NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-08-21 14:48:36,682 INFO [org.apache.hadoop.conf.Configuration.deprecation] - hadoop.native.lib is deprecated. Instead, use io.native.lib.available 2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.5-cdh5.0.2--1, built on 06/09/2014 16:09 GMT 2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=211.137.170.246 2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_60 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=/usr/java/jdk1.7.0_60/jre 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=/home/gzg/javadev/lib/hadoop/mapreduce/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/hadoop-auth-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/hadoop-annotations-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/hadoop-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-logging-1.1.3.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-compress-1.4.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/guava-11.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-collections-3.2.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/zookeeper-3.4.5-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-configuration-1.6.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-math3-3.1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/activation-1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/asm-3.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/avro-1.7.5-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-cli-1.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-codec-1.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-digester-1.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-el-1.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-httpclient-3.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-io-2.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-lang-2.6.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-net-3.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/httpclient-4.2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/httpcore-4.2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/gzg/javadev/lib/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/gzg/javadev/lib/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-core-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-json-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-server-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jets3t-0.9.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/jettison-1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jetty-6.1.26.jar:/home/gzg/javadev/lib/hadoop/common/lib/jetty-util-6.1.26.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsch-0.1.42.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsp-api-2.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsr305-1.3.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/junit-4.8.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/log4j-1.2.17.jar:/home/gzg/javadev/lib/hadoop/common/lib/mockito-all-1.8.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/paranamer-2.3.jar:/home/gzg/javadev/lib/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/servlet-api-2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/stax-api-1.0-2.jar:/home/gzg/javadev/lib/hadoop/common/lib/xmlenc-0.52.jar:/home/gzg/javadev/lib/hadoop/common/lib/xz-1.0.jar:/home/gzg/javadev/lib/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/gzg/javadev/lib/hadoop/hdfs/hadoop-hdfs-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-shuffle-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-api-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-client-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-client-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-common-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-hadoop-compat-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-it-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-protocol-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-testing-util-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-thrift-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/protobuf-java-2.5.0.jar:/home/gzg/javadev/lib/hbase/hbase-server-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-shell-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/htrace-core-2.01.jar:/home/gzg/javadev/hbase1/bin:/usr/java/jdk1.7.0_60/jre/lib:/usr/lib64 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=/tmp 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Linux 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=2.6.32-431.el6.x86_64 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=gzg 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=/home/gzg 2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=/home/gzg/javadev/hbase1 2014-08-21 14:48:42,230 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:42,254 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:42,261 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:42,276 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session 2014-08-21 14:48:42,335 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009d, negotiated timeout = 40000 2014-08-21 14:48:43,407 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:43,408 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:43,409 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk1/192.168.5.121:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:43,412 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk1/192.168.5.121:2181, initiating session 2014-08-21 14:48:43,421 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk1/192.168.5.121:2181, sessionid = 0x15478ef8bf2f016f, negotiated timeout = 40000 2014-08-21 14:48:53,961 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15478ef8bf2f016f closed 2014-08-21 14:48:53,961 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:53,985 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Started disable of _mrtest1 2014-08-21 14:48:54,051 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:54,053 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:54,057 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:54,067 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session 2014-08-21 14:48:54,072 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d0, negotiated timeout = 40000 2014-08-21 14:48:54,119 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d0 closed 2014-08-21 14:48:54,119 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:54,235 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:54,236 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:54,237 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:54,243 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session 2014-08-21 14:48:54,281 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009e, negotiated timeout = 40000 2014-08-21 14:48:54,325 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009e closed 2014-08-21 14:48:54,326 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:54,558 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:54,560 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:54,561 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:54,562 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session 2014-08-21 14:48:54,568 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d1, negotiated timeout = 40000 2014-08-21 14:48:54,584 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d1 closed 2014-08-21 14:48:54,584 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:54,894 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:54,899 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:54,900 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:54,901 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session 2014-08-21 14:48:54,906 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d2, negotiated timeout = 40000 2014-08-21 14:48:54,922 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d2 closed 2014-08-21 14:48:54,922 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:55,430 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:55,431 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:55,432 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk1/192.168.5.121:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:55,433 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk1/192.168.5.121:2181, initiating session 2014-08-21 14:48:55,439 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk1/192.168.5.121:2181, sessionid = 0x15478ef8bf2f0170, negotiated timeout = 40000 2014-08-21 14:48:55,456 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15478ef8bf2f0170 closed 2014-08-21 14:48:55,457 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:55,459 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Disabled _mrtest1 2014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Deleted _mrtest1 2014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing master protocol: MasterService 2014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x1647ed72def6009d 2014-08-21 14:48:55,658 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009d closed 2014-08-21 14:48:55,659 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:55,824 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=hconnection-0x30dd1e89, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:55,826 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x30dd1e89 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:55,826 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:55,836 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session 2014-08-21 14:48:55,855 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009f, negotiated timeout = 40000 2014-08-21 14:48:56,473 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x30dd1e89, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:56,474 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x30dd1e89 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:56,475 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:56,476 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session 2014-08-21 14:48:56,482 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def600a0, negotiated timeout = 40000 2014-08-21 14:48:56,500 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def600a0 closed 2014-08-21 14:48:56,501 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:56,502 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing master protocol: MasterService 2014-08-21 14:48:56,502 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x1647ed72def6009f 2014-08-21 14:48:56,506 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009f closed 2014-08-21 14:48:56,507 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 2014-08-21 14:48:56,508 INFO [root] - Table _mrtest1 created! 2014-08-21 14:48:56,909 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=90000 watcher=hconnection-0x6a86ad83, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase 2014-08-21 14:48:56,911 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error) 2014-08-21 14:48:56,912 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x6a86ad83 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181 2014-08-21 14:48:56,915 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session 2014-08-21 14:48:56,922 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d3, negotiated timeout = 40000 2014-08-21 14:48:56,965 INFO [org.apache.hadoop.hbase.mapreduce.TableOutputFormat] - Created table instance for _mrtest1 2014-08-21 14:49:07,111 INFO [org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider] - Failing over to rm2 2014-08-21 14:49:12,459 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2014-08-21 14:49:25,381 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2: Name or service not known]; remaining name '111.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,385 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,386 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,387 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,387 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,388 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '111.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,389 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,389 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,390 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,390 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,413 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,414 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,415 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,415 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,416 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa' 2014-08-21 14:49:25,595 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:15 2014-08-21 14:49:25,925 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_1408378794442_0015 2014-08-21 14:49:26,170 INFO [org.apache.hadoop.yarn.client.api.impl.YarnClientImpl] - Submitted application application_1408378794442_0015 2014-08-21 14:49:26,219 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://rm2:23188/proxy/application_1408378794442_0015/ 2014-08-21 14:49:26,220 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_1408378794442_0015 2014-08-21 14:49:38,592 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0015 running in uber mode : false 2014-08-21 14:49:38,594 INFO [org.apache.hadoop.mapreduce.Job] - map 0% reduce 0% 2014-08-21 14:50:40,813 INFO [org.apache.hadoop.mapreduce.Job] - map 7% reduce 0% 2014-08-21 14:50:47,911 INFO [org.apache.hadoop.mapreduce.Job] - map 11% reduce 0% 2014-08-21 14:50:49,945 INFO [org.apache.hadoop.mapreduce.Job] - map 13% reduce 0% 2014-08-21 14:51:03,145 INFO [org.apache.hadoop.mapreduce.Job] - map 20% reduce 0% 2014-08-21 14:51:04,159 INFO [org.apache.hadoop.mapreduce.Job] - map 20% reduce 4% 2014-08-21 14:51:10,265 INFO [org.apache.hadoop.mapreduce.Job] - map 20% reduce 7% 2014-08-21 14:51:28,566 INFO [org.apache.hadoop.mapreduce.Job] - map 24% reduce 7% 2014-08-21 14:51:30,598 INFO [org.apache.hadoop.mapreduce.Job] - map 33% reduce 7% 2014-08-21 14:51:31,613 INFO [org.apache.hadoop.mapreduce.Job] - map 33% reduce 9% 2014-08-21 14:51:34,656 INFO [org.apache.hadoop.mapreduce.Job] - map 33% reduce 11% 2014-08-21 14:51:42,840 INFO [org.apache.hadoop.mapreduce.Job] - map 40% reduce 11% 2014-08-21 14:51:46,912 INFO [org.apache.hadoop.mapreduce.Job] - map 40% reduce 13% 2014-08-21 14:51:55,050 INFO [org.apache.hadoop.mapreduce.Job] - map 47% reduce 13% 2014-08-21 14:51:56,074 INFO [org.apache.hadoop.mapreduce.Job] - map 47% reduce 16% 2014-08-21 14:52:03,177 INFO [org.apache.hadoop.mapreduce.Job] - map 53% reduce 16% 2014-08-21 14:52:08,241 INFO [org.apache.hadoop.mapreduce.Job] - map 53% reduce 18% 2014-08-21 14:52:12,303 INFO [org.apache.hadoop.mapreduce.Job] - map 60% reduce 18% 2014-08-21 14:52:14,343 INFO [org.apache.hadoop.mapreduce.Job] - map 60% reduce 20% 2014-08-21 14:52:26,528 INFO [org.apache.hadoop.mapreduce.Job] - map 64% reduce 20% 2014-08-21 14:52:27,551 INFO [org.apache.hadoop.mapreduce.Job] - map 73% reduce 20% 2014-08-21 14:52:29,581 INFO [org.apache.hadoop.mapreduce.Job] - map 73% reduce 22% 2014-08-21 14:52:32,621 INFO [org.apache.hadoop.mapreduce.Job] - map 73% reduce 24% 2014-08-21 14:52:44,811 INFO [org.apache.hadoop.mapreduce.Job] - map 80% reduce 24% 2014-08-21 14:52:47,848 INFO [org.apache.hadoop.mapreduce.Job] - map 80% reduce 27% 2014-08-21 14:52:56,950 INFO [org.apache.hadoop.mapreduce.Job] - map 84% reduce 27% 2014-08-21 14:52:57,966 INFO [org.apache.hadoop.mapreduce.Job] - map 87% reduce 27% 2014-08-21 14:52:59,997 INFO [org.apache.hadoop.mapreduce.Job] - map 87% reduce 29% 2014-08-21 14:53:42,487 INFO [org.apache.hadoop.mapreduce.Job] - map 91% reduce 29% 2014-08-21 14:53:43,506 INFO [org.apache.hadoop.mapreduce.Job] - map 93% reduce 29% 2014-08-21 14:53:45,540 INFO [org.apache.hadoop.mapreduce.Job] - map 93% reduce 31% 2014-08-21 14:53:47,575 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 31% 2014-08-21 14:53:48,584 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 33% 2014-08-21 14:53:51,611 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 44% 2014-08-21 14:53:54,642 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 55% 2014-08-21 14:53:57,679 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 66% 2014-08-21 14:54:00,710 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 71% 2014-08-21 14:54:03,749 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 79% 2014-08-21 14:54:06,791 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 88% 2014-08-21 14:54:09,829 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 96% 2014-08-21 14:54:11,854 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2014-08-21 14:54:11,873 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0015 completed successfully 2014-08-21 14:54:12,082 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 61 File System Counters FILE: Number of bytes read=324596614 FILE: Number of bytes written=651096206 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1105 HDFS: Number of bytes written=0 HDFS: Number of read operations=15 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job Counters Killed map tasks=6 Launched map tasks=21 Launched reduce tasks=1 Data-local map tasks=15 Rack-local map tasks=6 Total time spent by all maps in occupied slots (ms)=5299964 Total time spent by all reduces in occupied slots (ms)=413798 Total time spent by all map tasks (ms)=2649982 Total time spent by all reduce tasks (ms)=206899 Total vcore-seconds taken by all map tasks=2649982 Total vcore-seconds taken by all reduce tasks=206899 Total megabyte-seconds taken by all map tasks=4070372352 Total megabyte-seconds taken by all reduce tasks=423729152 Map-Reduce Framework Map input records=14112896 Map output records=14112896 Map output bytes=296370816 Map output materialized bytes=324596698 Input split bytes=1105 Combine input records=0 Combine output records=0 Reduce input groups=1 Reduce shuffle bytes=324596698 Reduce input records=14112896 Reduce output records=1 Spilled Records=28225792 Shuffled Maps =15 Failed Shuffles=0 Merged Map outputs=15 GC time elapsed (ms)=13168 CPU time spent (ms)=403830 Physical memory (bytes) snapshot=6978674688 Virtual memory (bytes) snapshot=29039456256 Total committed heap usage (bytes)=4911529984 HBase Counters BYTES_IN_REMOTE_RESULTS=194636060 BYTES_IN_RESULTS=5490524772 MILLIS_BETWEEN_NEXTS=1834495 NOT_SERVING_REGION_EXCEPTION=0 NUM_SCANNER_RESTARTS=0 REGIONS_SCANNED=15 REMOTE_RPC_CALLS=5050 REMOTE_RPC_RETRIES=0 RPC_CALLS=141167 RPC_RETRIES=0 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0 *** measure elapsed time,stoped at 2014-08-21 14:54:12:089 time elapsed :0days 0hours 5minutes 36seconds 209milliseconds[336.209seconds]
