2020年暑假 （7）

yarn集群配置启动

1.yarn-site.xml

<property>
<name>yarn.resourcemanager.hostname</name>
<value>hdp-01</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

2.将项目打包时报错

 

 maven的依赖出现问题,上面注释掉的部分改为下面的内容

    <!-- <plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> 
                <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> 
                <archive> <manifest> <mainClass>cn.edu360.mr.wc.JobSubmitterLinuxToYarn</mainClass> 
                </manifest> </archive> </configuration> <executions> <execution> <id>make-assembly</id> 
                this is used for inheritance merges <phase>package</phase> bind to the packaging 
                phase <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> -->
            <plugin>
    <artifactId>maven-assembly-plugin</artifactId>
    <executions>
        <execution>
            <id>make-assembly</id>
            <goals>
                <goal>single</goal>
            </goals>
            <configuration> 
                <descriptors> 
                     <descriptor>/assembly/assembly.xml</descriptor> 
                </descriptors> 
            </configuration>
        </execution>
    </executions>
</plugin>

3.

 

此依赖包并没有下载完整,删除本地的资源重新下载即可

4.

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ‪C:%5CUsers%5Cuser%5CDesktop%5Csrc%5Cmapreduce24-0.0.1-SNAPSHOT.jar

路径的问题,在jar包属性中复制的路径可能存在特殊字符,重新打一遍解决

 5.

 

 创建对应文件夹6.

 Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

找到hadoop安装目录mapred-site.xml,增加以下代码 

<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>