[Hadoop] - Win7下提交job到集群上去
一般我们采用win开发+linux hadoop集群的方式进行开发,使用插件:hadoop-***-eclipse-plugin。
运行程序的时候,我们一般采用run as application或者选择run as hadoop。按照这个字面理解,我们可以认为第一种是运行在本地,第二种是运行在hadoop集群上。但是实际情况是一般如果不进行配置的话,全部是在本地进行运行的。如果需要将job提交到集群上,那么需要进行必要的设置和添加部分代码。
1、copy mapred-site.xml && yarn-site.xml文件,并修改必要的信息,将yarn指向集群。
2、给mapred-site.xml文件中添加参数mapreduce.app-submission.cross-platform,参数值为true。
3、打包本地代码提交到集群上,如果不进行该操作,会出现ClassNotFoundException。打包代码如下:
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1 import java.io.File; 2 import java.io.FileInputStream; 3 import java.io.FileOutputStream; 4 import java.io.IOException; 5 import java.util.jar.JarEntry; 6 import java.util.jar.JarOutputStream; 7 8 public class EJob { 9 10 public static File createTempJar(String root) throws IOException { 11 if (!new File(root).exists()) { 12 return null; 13 } 14 15 final File jarFile = File.createTempFile("EJob-", ".jar", new File(System 16 .getProperty("java.io.tmpdir"))); 17 18 Runtime.getRuntime().addShutdownHook(new Thread() { 19 @Override 20 public void run() { 21 jarFile.delete(); 22 } 23 }); 24 25 JarOutputStream out = new JarOutputStream(new FileOutputStream(jarFile)); 26 createTempJarInner(out, new File(root), ""); 27 out.flush(); 28 out.close(); 29 return jarFile; 30 } 31 32 private static void createTempJarInner(JarOutputStream out, File f, 33 String base) throws IOException { 34 if (f.isDirectory()) { 35 File[] fl = f.listFiles(); 36 if (base.length() > 0) { 37 base = base + "/"; 38 } 39 for (int i = 0; i < fl.length; i++) { 40 createTempJarInner(out, fl[i], base + fl[i].getName()); 41 } 42 } else { 43 out.putNextEntry(new JarEntry(base)); 44 FileInputStream in = new FileInputStream(f); 45 byte[] buffer = new byte[1024]; 46 int n = in.read(buffer); 47 while (n != -1) { 48 out.write(buffer, 0, n); 49 n = in.read(buffer); 50 } 51 in.close(); 52 } 53 } 54 }
File jarFile = EJob.createTempJar("target/classes"); ((JobConf) job.getConfiguration()).setJar(jarFile.toString()); // 其他创建job的代码不进行任何的修改
至此,就可以将job提交到集群上去了。
对应任何在非hadoop集群中提交的mr任务来讲,均需要注意一下几点:
1. 参数mapreduce.app-submission.cross-platform必须设置为true,表示是跨集群提交job
2. 如果参数mapreduce.framework.name值为yarn,那么必须将类YarnClientProtocolProvider引入到项目的classpath路径中,maven依赖如下:
// 其他正常的hadoop-mapreduce-client依赖还是需要的, 只是这个在跨平台提交的过程中是一定需要的 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-mapreduce-client-jobclient</artifactId> <version>${hadoop.version}</version> </dependency>
3. 如果集群是HA设置,那么必须给定HA配置或者采用明确指定active节点的方式。必须给定的参数有yarn.resourcemanager.address和fs.defaultFS之类的定位参数
当HDFS和Yarn均使用HA的时候,跨集群提交最少配置(依赖集群的具体搭建方法,比如如果在搭建过程中执行了yarn的classpath,那么yarn-site.xml中的参数yarn.application.classpath可以不要, 其他参数不可以少,必须存在!!!)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hdfs-cluster</value> </property> </configuration>
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.nameservices</name> <value>hdfs-cluster</value> </property> <property> <name>dfs.ha.namenodes.hdfs-cluster</name> <value>hdfs-cluster-1,hdfs-cluster-2</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs-cluster.hdfs-cluster-1</name> <value>hdfs-cluster-1:8020</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs-cluster.hdfs-cluster-2</name> <value>hdfs-cluster-2:8020</value> </property> <property> <name>dfs.client.failover.proxy.provider.hdfs-cluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> </configuration>
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.app-submission.cross-platform</name> <value>true</value> </property> </configuration>
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!-- RM Manager Configd --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yarn-cluster</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>yarn-cluster-1,yarn-cluster-2</value> </property> <!-- RM1 Configs--> <property> <name>yarn.resourcemanager.address.yarn-cluster-1</name> <value>yarn-cluster-1:8032</value> </property> <!-- RM2 Configs --> <property> <name>yarn.resourcemanager.address.yarn-cluster-2</name> <value>yarn-cluster-2:8032</value> </property> <property> <name>yarn.application.classpath</name> <value> $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/* </value> </property> </configuration>