hadoop Counter
1.job.getCounters().findCounter(MyEnum.selefDefineCounter).getValue();
1.1 Job类API解释
public class Job
extends
org.apache.hadoop.mapreduce.task.JobContextImpl
implements JobContext
Job类允许用户区配置任务、提交、控制执行和查询状态。直到改job提交后，它的方法才会生效，否则会抛出非法状态的异常。
通常情况下我创建一个应用，会通过Job类去描述一个任务的各个方面，然后提交和监视进程。
Here is an example on how to submit a job:
// Create a new Job
Job job = Job.getInstance();
job.setJarByClass(MyJob.class);
// Specify various job-specific
parameters
job.setJobName("myjob");
job.setInputPath(new
Path("in"));
job.setOutputPath(new
Path("out"));
job.setMapperClass(MyJob.MyMapper.class);
job.setReducerClass(MyJob.MyReducer.class);
// Submit the job, then poll for
progress until the job is complete
job.waitForCompletion(true);
来自 <http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/Job.html>
facet 侧面，方面
1.2 getCounters
Counters
Counters 是多个由Map/Reduce框架或者应用程序定义的全局计数器。 每一个Counter可以是任何一种 Enum类型。同一特定Enum类型的Counter可以汇集到一个组，其类型为Counters.Group。
应用程序可以定义任意(Enum类型)的Counters并且可以通过 map 或者 reduce方法中的 Reporter.incrCounter(Enum, long)或者 Reporter.incrCounter(String, String, long) 更新。之后框架会汇总这些全局counters。
来自 <http://hadoop.apache.org/docs/r1.0.4/cn/mapred_tutorial.html#Counters>
|
public Counters getCounters() Gets the counters for this job. May return null if the job has been retired and the job is no longer in the completed job store. Returns: the counters for this job. Throws: retired：退休，退役；
|
|
public class Counters Counters holds per job/task counters, defined either by the Map-Reduce framework or applications. Each Counter can be of any Enum type. Counters are bunched into CounterGroups, each comprising of counters from a particular Enum class. comprising ：组成，构成 |
控制台运行打印的计数器
|
2017-01-08 22:00:05,812 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Counters: 39 File System Counters FILE: Number of bytes read=42392 FILE: Number of bytes written=13490644 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=5564 HDFS: Number of bytes written=5618 HDFS: Number of read operations=855 HDFS: Number of large read operations=0 HDFS: Number of write operations=398 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=252 Map output materialized bytes=280 Input split bytes=107 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=280 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=2 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1452277760 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=100 File Output Format Counters Bytes Written=102 pagerank.JobPageRank$Mycount my=2 success ---- 29avg5.0E-4 |
