在Ubuntu18.04下配置hadoop集群
服务器准备
启动hadoop最小集群的典型配置是3台服务器, 一台作为Master, NameNode, 两台作为Slave, DataNode. 操作系统使用的Ubuntu18.04 Server, 安装过程就省略了, 使用的是LVM文件系统, XFS文件格式, 为了避免浪费空间, 除了划分1G给/boot以外, 其他都划为/
服务器规划
192.168.1.148 vm148 -- 作为master, NameNode, ResourceManager
192.168.1.149 vm149 -- 作为slave, DataNode, NodeManager
192.168.1.150 vm150 -- 作为slave, DataNode, NodeManager
注意: 这里是第一个坑, 主机名里面不能带下划线 _ , 会导致DataServer创建socket失败无法启动.
安装后的升级
sudo apt update sudo apt upgrade
添加普通用户
用于运行hadoop的受限用户, 我习惯用tomcat作为用户名, 这里使用adduser而不是useradd, 因为后者不带参数时, 有时候不会创建home目录
sudo adduser tomcat # 按提示输入
.如果是虚机, 这时候就可以以当前状态创建模板了.
设置hostname和hosts
# view current hostname sudo hostnamectl status # set sudo hostnamectl set-hostname vm148 # add entries to hosts sudo vi /etc/hosts # add following lines 192.168.1.148 vm148 192.168.1.149 vm149 192.168.1.150 vm150
.依次将服务器设置为vm148, vm149, vm150. 重启后登入检查是否生效, 互相ping看看是否生效
对tomcat用户互相添加免密登录
# 生成id_rsa和id_rsa.pub ssh-keygen cd .ssh/ # 创建 authorized_keys mv id_rsa.pub authorized_keys # 注意权限必须是600 chmod 600 authorized_keys # 将本服务器的私钥改名为id_rsa_mine mv id_rsa id_rsa_mine # 修改config vi config # 添加如下内容 Host vm149 IdentityFile ~/.ssh/id_rsa_mine User tomcat Host vm150 IdentityFile ~/.ssh/id_rsa_mine User tomcat Host vm148 IdentityFile ~/.ssh/id_rsa_mine User tomcat # 如果是master机器, 还需要添加如下, 用于启动Secondary name server Host 0.0.0.0 IdentityFile ~/.ssh/id_rsa_mine User tomcat
将各个服务器的authorized_keys的内容互相合并, 最后各服务器的authorized_keys文件都是一样的.
在以上工作完成后, 从各个机器尝试ssh tomcat@[主机名], 确保登录没问题, 也避免在启动服务时提示发现新key是否接受
防火墙ufw
如果是初次设置, 建议关闭, 确保不会因为防火墙而导致服务启动失败, 可以等服务配置完成后, 再根据实际的端口, 打开并配置ufw
sudo ufw disable
配置JDK
将jdk解压缩至/opt/jdk, 并创建latest软链, 完成后结构如下
$ ll /opt/jdk/ total 0 drwxr-xr-x 7 root root 245 Oct 6 13:58 jdk1.8.0_192/ lrwxrwxrwx 1 root root 12 Jan 18 05:49 latest -> jdk1.8.0_192/
需要将jps软链到/usr/bin
cd /usr/bin sudo ln -s /opt/jdk/latest/bin/jps jps
配置Hadoop
将hadoop解压缩至 /opt/hadoop, 并创建latest 软链, 完成后目录结构如下
$ ll /opt/hadoop/ total 0 drwxr-xr-x 9 root root 149 Nov 13 15:15 hadoop-2.9.2/ lrwxrwxrwx 1 root root 12 Jan 18 10:26 latest -> hadoop-2.9.2/
修改配置文件 etc/hadoop/hadoop-env.sh
需要修改的变量有两处
# The java implementation to use. export JAVA_HOME=/opt/jdk/latest # Where log files are stored. $HADOOP_HOME/logs by default. export HADOOP_LOG_DIR=/home/tomcat/run/hadoop/logs
修改配置文件 etc/hadoop/yarn-env.sh
需要修改的变量有两处
# some Java parameters export JAVA_HOME=/opt/jdk/latest # default log directory & file export YARN_LOG_DIR=/home/tomcat/run/yarn/logs
修改配置文件/etc/hadoop/slaves
将内容修改为两个slave的主机名
vm149 vm150
修改配置文件/etc/hadoop/core-site.xml
添加以下内容. 配置明细需要参考 share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/tomcat/run/hadoop</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://vm148:9000</value>
</property>
</configuration>
修改配置文件/etc/hadoop/hdfs-site.xml
添加以下内容
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
修改配置文件/etc/hadoop/mapred-site.xml
添加以下内容
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
修改配置文件/etc/hadoop/yarn-site.xml
添加以下内容. 配置明细需要参考 share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
<configuration>
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>vm148</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
将配置好的hadoop, 按当前的目录结构, 复制到另外两个服务器中
启动Hadoop
第一次启动前, 需要format nameserver, 在master服务器上执行
/opt/hadoop/latest/bin/hdfs namenode -format
然后启动hdfs服务
/opt/hadoop/latest/sbin/start-dfs.sh
然后启动yarn服务
/opt/hadoop/latest/sbin/start-yarn.sh
每一步, 都需要用jps命令查看服务是否正常启动, 对于master服务器, 正常启动后应该显示如下进程
tomcat@vm148:/opt$ jps 3173 SecondaryNameNode 3495 ResourceManager 4583 Jps 2906 NameNode
slave服务器
tomcat@vm149:~/run$ jps 3074 NodeManager 2691 DataNode 3591 Jps
.
WEB访问
服务启动后, 可以通过 http://vm148:50070/ 访问web界面
服务端口
master端
21, FTP for ? 8030, YARN resourcemanager scheduler 8031, YARN resourcemanager tracker 8032, YARN resourcemanager 8033, YARN resourcemanager admin 8088, YARN resourcemanager webapp 8090, YARN resourcemanager webapp https 9000, HDFS 50070, WEB UI 50090,
slave, data node端
50075
运行WordCount Example
首先编译java, 生成class, 生成jar. 因为JAVA_HOME已经在hadoop里配置过, 而PATH在此环境不需要, 只需要配置一个tools.jar的classpath就可以了
export HADOOP_CLASSPATH=/opt/jdk/latest/lib/tools.jar /opt/hadoop/latest/bin/hadoop com.sun.tools.javac.Main WordCount.java /opt/jdk/latest/bin/jar cf wc.jar WordCount*.class
然后将两个输入文件上载到hdfs.
/opt/hadoop/latest/bin/hadoop fs -put file01 /workspace/input/ /opt/hadoop/latest/bin/hadoop fs -ls /workspace/input /opt/hadoop/latest/bin/hadoop fs -put file02 /workspace/input/ /opt/hadoop/latest/bin/hadoop fs -cat /workspace/input/file01 /opt/hadoop/latest/bin/hadoop fs -cat /workspace/input/file02
一开始我在这里遇到了个坑: 我把文件放到/tmp/下面去了, 把/tmp作为输入目录, 结果在运行中yarn会把staging信息存在 /tmp/hadoop-yarn/staging 文件中, 然后任务就抛异常了. 教训就是: 任务文件不要放到/tmp下
执行任务
/opt/hadoop/latest/bin/hadoop jar wc.jar WordCount /workspace/input /workspace/output
这里最后一个路径是输出路径, 这个路径在运行任务前不能存在, 否则也会报错
最后的执行结果
tomcat@vm148:~$ /opt/hadoop/latest/bin/hadoop jar wc.jar WordCount /workspace/input /workspace/output 19/01/30 08:24:55 INFO client.RMProxy: Connecting to ResourceManager at vm148/192.168.1.148:8032 19/01/30 08:24:55 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 19/01/30 08:24:56 INFO input.FileInputFormat: Total input files to process : 2 19/01/30 08:24:56 INFO mapreduce.JobSubmitter: number of splits:2 19/01/30 08:24:56 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 19/01/30 08:24:56 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1547812325179_0004 19/01/30 08:24:56 INFO impl.YarnClientImpl: Submitted application application_1547812325179_0004 19/01/30 08:24:56 INFO mapreduce.Job: The url to track the job: http://vm148:8088/proxy/application_1547812325179_0004/ 19/01/30 08:24:56 INFO mapreduce.Job: Running job: job_1547812325179_0004 19/01/30 08:25:03 INFO mapreduce.Job: Job job_1547812325179_0004 running in uber mode : false 19/01/30 08:25:03 INFO mapreduce.Job: map 0% reduce 0% 19/01/30 08:25:10 INFO mapreduce.Job: map 100% reduce 0% 19/01/30 08:25:18 INFO mapreduce.Job: map 100% reduce 100% 19/01/30 08:25:18 INFO mapreduce.Job: Job job_1547812325179_0004 completed successfully 19/01/30 08:25:18 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=97 FILE: Number of bytes written=594622 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=266 HDFS: Number of bytes written=38 HDFS: Number of read operations=9 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=10309 Total time spent by all reduces in occupied slots (ms)=3850 Total time spent by all map tasks (ms)=10309 Total time spent by all reduce tasks (ms)=3850 Total vcore-milliseconds taken by all map tasks=10309 Total vcore-milliseconds taken by all reduce tasks=3850 Total megabyte-milliseconds taken by all map tasks=10556416 Total megabyte-milliseconds taken by all reduce tasks=3942400 Map-Reduce Framework Map input records=2 Map output records=10 Map output bytes=96 Map output materialized bytes=103 Input split bytes=210 Combine input records=10 Combine output records=8 Reduce input groups=5 Reduce shuffle bytes=103 Reduce input records=8 Reduce output records=5 Spilled Records=16 Shuffled Maps =2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms)=379 CPU time spent (ms)=2090 Physical memory (bytes) snapshot=778280960 Virtual memory (bytes) snapshot=5914849280 Total committed heap usage (bytes)=507510784 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=56 File Output Format Counters Bytes Written=38 tomcat@vm148:~$ /opt/hadoop/latest/bin/hadoop fs -ls /workspace/output Found 2 items -rw-r--r-- 2 tomcat supergroup 0 2019-01-30 08:25 /workspace/output/_SUCCESS -rw-r--r-- 2 tomcat supergroup 38 2019-01-30 08:25 /workspace/output/part-r-00000 tomcat@vm148:~$ /opt/hadoop/latest/bin/hadoop fs -cat /workspace/output/part-r-00000 Day 2 Good 2 Hadoop 2 Hello 2 World 2
一个简单的Map Reduce 例子
输入的内容格式是这样的, 每一行是一个日志记录, 记录了用户, IP和时间戳, 需要统计每个 (用户+IP) 出现的次数
1571 76 738 legnd 166.111.8.133 870876781 1572 121 697 kuoc 202.116.65.16 870909489 1573 121 697 kuoc 202.116.65.16 870910644 1574 121 739 maerick 870926284
代码 pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.rockbb</groupId>
<artifactId>hdtask</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>HD Task</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.4.1</version>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<configuration>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
代码 DataBean.java
package com.rockbb.hdtask;
import org.apache.hadoop.io.Writable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
public class DataBean implements Writable {
private String nameIp;
private long count;
public DataBean() {
}
public DataBean(String nameIp, long count) {
this.nameIp = nameIp;
this.count = count;
}
public String getNameIp() {
return nameIp;
}
public void setNameIp(String nameIp) {
this.nameIp = nameIp;
}
public long getCount() {
return count;
}
public void setCount(long count) {
this.count = count;
}
/**
* Important: this will be use for the final output.
*/
@Override
public String toString() {
return this.nameIp + "\t" + this.count;
}
@Override
public void write(DataOutput dataOutput) throws IOException {
dataOutput.writeUTF(nameIp);
dataOutput.writeLong(count);
}
@Override
public void readFields(DataInput dataInput) throws IOException {
this.nameIp = dataInput.readUTF();
this.count = dataInput.readLong();
}
}
代码 IpCount.java
package com.rockbb.hdtask;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
public class IpCount {
public static class IpMapper extends Mapper<LongWritable, Text, Text, DataBean> {
@Override
public void map(LongWritable keyIn, Text valueIn, Context context) throws IOException, InterruptedException {
String line = valueIn.toString();
String[] fields = line.split("\t");
String keyOut = fields[3] + '-' + fields[4];
long valueOut = 1;
DataBean bean = new DataBean(keyOut, valueOut);
context.write(new Text(keyOut), bean);
}
}
public static class IpReducer extends Reducer<Text, DataBean, Text, DataBean> {
@Override
public void reduce(Text keyIn, Iterable<DataBean> valuesIn, Context context) throws IOException, InterruptedException {
long total = 0;
for (DataBean bean : valuesIn) {
total += bean.getCount();
}
DataBean bean = new DataBean("", total);
context.write(keyIn, bean);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf);
job.setJarByClass(IpCount.class);
job.setMapperClass(IpMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(DataBean.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
job.setReducerClass(IpReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(DataBean.class);
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}
运行命令
/opt/hadoop/latest/bin/hadoop jar hdtask.jar com.rockbb.hdtask.IpCount /workspace/input/ /workspace/output3
.数据文件有2.3GB, 因为默认的block大小为128MB, 所以提交后产生了19个Map任务和一个Reduce任务, 任务的命令行输出
19/01/31 10:08:01 INFO client.RMProxy: Connecting to ResourceManager at vm148/192.168.31.148:8032 19/01/31 10:08:02 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 19/01/31 10:08:02 INFO input.FileInputFormat: Total input files to process : 1 19/01/31 10:08:02 INFO mapreduce.JobSubmitter: number of splits:19 19/01/31 10:08:02 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 19/01/31 10:08:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1547812325179_0008 19/01/31 10:08:03 INFO impl.YarnClientImpl: Submitted application application_1547812325179_0008 19/01/31 10:08:03 INFO mapreduce.Job: The url to track the job: http://vm148:8088/proxy/application_1547812325179_0008/ 19/01/31 10:08:03 INFO mapreduce.Job: Running job: job_1547812325179_0008 19/01/31 10:08:13 INFO mapreduce.Job: Job job_1547812325179_0008 running in uber mode : false 19/01/31 10:08:13 INFO mapreduce.Job: map 0% reduce 0% 19/01/31 10:08:41 INFO mapreduce.Job: map 11% reduce 0% 19/01/31 10:08:45 INFO mapreduce.Job: map 21% reduce 0% 19/01/31 10:08:47 INFO mapreduce.Job: map 23% reduce 0% 19/01/31 10:08:51 INFO mapreduce.Job: map 28% reduce 0% 19/01/31 10:08:53 INFO mapreduce.Job: map 30% reduce 0% 19/01/31 10:08:57 INFO mapreduce.Job: map 31% reduce 0% 19/01/31 10:08:59 INFO mapreduce.Job: map 38% reduce 0% 19/01/31 10:09:09 INFO mapreduce.Job: map 39% reduce 0% 19/01/31 10:09:10 INFO mapreduce.Job: map 40% reduce 0% 19/01/31 10:09:11 INFO mapreduce.Job: map 44% reduce 0% 19/01/31 10:09:14 INFO mapreduce.Job: map 46% reduce 0% 19/01/31 10:09:16 INFO mapreduce.Job: map 48% reduce 0% 19/01/31 10:09:17 INFO mapreduce.Job: map 49% reduce 0% 19/01/31 10:09:22 INFO mapreduce.Job: map 55% reduce 0% 19/01/31 10:09:24 INFO mapreduce.Job: map 56% reduce 0% 19/01/31 10:09:28 INFO mapreduce.Job: map 61% reduce 0% 19/01/31 10:09:40 INFO mapreduce.Job: map 64% reduce 0% 19/01/31 10:09:42 INFO mapreduce.Job: map 64% reduce 7% 19/01/31 10:09:46 INFO mapreduce.Job: map 66% reduce 7% 19/01/31 10:09:48 INFO mapreduce.Job: map 68% reduce 9% 19/01/31 10:09:52 INFO mapreduce.Job: map 71% reduce 9% 19/01/31 10:09:54 INFO mapreduce.Job: map 71% reduce 12% 19/01/31 10:09:58 INFO mapreduce.Job: map 73% reduce 12% 19/01/31 10:09:59 INFO mapreduce.Job: map 74% reduce 12% 19/01/31 10:10:01 INFO mapreduce.Job: map 75% reduce 12% 19/01/31 10:10:04 INFO mapreduce.Job: map 80% reduce 12% 19/01/31 10:10:06 INFO mapreduce.Job: map 81% reduce 12% 19/01/31 10:10:10 INFO mapreduce.Job: map 85% reduce 12% 19/01/31 10:10:12 INFO mapreduce.Job: map 86% reduce 12% 19/01/31 10:10:13 INFO mapreduce.Job: map 87% reduce 12% 19/01/31 10:10:15 INFO mapreduce.Job: map 88% reduce 12% 19/01/31 10:10:18 INFO mapreduce.Job: map 88% reduce 16% 19/01/31 10:10:22 INFO mapreduce.Job: map 90% reduce 16% 19/01/31 10:10:23 INFO mapreduce.Job: map 91% reduce 16% 19/01/31 10:10:24 INFO mapreduce.Job: map 91% reduce 18% 19/01/31 10:10:25 INFO mapreduce.Job: map 92% reduce 18% 19/01/31 10:10:29 INFO mapreduce.Job: map 93% reduce 18% 19/01/31 10:10:31 INFO mapreduce.Job: map 93% reduce 21% 19/01/31 10:10:32 INFO mapreduce.Job: map 94% reduce 21% 19/01/31 10:10:34 INFO mapreduce.Job: map 96% reduce 21% 19/01/31 10:10:35 INFO mapreduce.Job: map 97% reduce 21% 19/01/31 10:10:37 INFO mapreduce.Job: map 98% reduce 23% 19/01/31 10:10:38 INFO mapreduce.Job: map 99% reduce 23% 19/01/31 10:10:41 INFO mapreduce.Job: map 100% reduce 23% 19/01/31 10:10:43 INFO mapreduce.Job: map 100% reduce 30% 19/01/31 10:10:49 INFO mapreduce.Job: map 100% reduce 33% 19/01/31 10:11:25 INFO mapreduce.Job: map 100% reduce 67% 19/01/31 10:11:31 INFO mapreduce.Job: map 100% reduce 70% 19/01/31 10:11:37 INFO mapreduce.Job: map 100% reduce 74% 19/01/31 10:11:43 INFO mapreduce.Job: map 100% reduce 78% 19/01/31 10:11:49 INFO mapreduce.Job: map 100% reduce 83% 19/01/31 10:11:55 INFO mapreduce.Job: map 100% reduce 86% 19/01/31 10:12:01 INFO mapreduce.Job: map 100% reduce 89% 19/01/31 10:12:07 INFO mapreduce.Job: map 100% reduce 93% 19/01/31 10:12:13 INFO mapreduce.Job: map 100% reduce 97% 19/01/31 10:12:18 INFO mapreduce.Job: map 100% reduce 100% 19/01/31 10:12:19 INFO mapreduce.Job: Job job_1547812325179_0008 completed successfully 19/01/31 10:12:19 INFO mapreduce.Job: Counters: 50 File System Counters FILE: Number of bytes read=6635434217 FILE: Number of bytes written=9269615741 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=2551940756 HDFS: Number of bytes written=134288980 HDFS: Number of read operations=60 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Killed map tasks=3 Launched map tasks=22 Launched reduce tasks=1 Data-local map tasks=22 Total time spent by all maps in occupied slots (ms)=1737403 Total time spent by all reduces in occupied slots (ms)=178563 Total time spent by all map tasks (ms)=1737403 Total time spent by all reduce tasks (ms)=178563 Total vcore-milliseconds taken by all map tasks=1737403 Total vcore-milliseconds taken by all reduce tasks=178563 Total megabyte-milliseconds taken by all map tasks=1779100672 Total megabyte-milliseconds taken by all reduce tasks=182848512 Map-Reduce Framework Map input records=49458230 Map output records=49458230 Map output bytes=2531297616 Map output materialized bytes=2630214190 Input split bytes=2052 Combine input records=0 Combine output records=0 Reduce input groups=5453085 Reduce shuffle bytes=2630214190 Reduce input records=49458230 Reduce output records=5453085 Spilled Records=174185483 Shuffled Maps =19 Failed Shuffles=0 Merged Map outputs=19 GC time elapsed (ms)=9585 CPU time spent (ms)=389790 Physical memory (bytes) snapshot=5763260416 Virtual memory (bytes) snapshot=39333715968 Total committed heap usage (bytes)=4077912064 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=2551938704 File Output Format Counters Bytes Written=134288980
