Cloudera Certified Associate Administrator案例之Troubleshoot篇

　　　　　　Cloudera Certified Associate Administrator案例之Troubleshoot篇

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

 

 

一.调整日志的进程级别

问题描述：
　　今天node103.yinzhengjie.org.cn节点的DataNode和NodeManager进程频繁死掉，你决定临时将该节点两个进程的日志级别调整为DEBUG，以便于进行维护排查。

解决方案：
　　操作时需注意要求将HDFS和YARN的所有节点日志界别都调整，还是只需要调整还是只需要调整个别节点，两者的调整方式是不一样的，对于个别节点日志级别的临时调整，可以使用yarn daemonlog -setlevel命令来进行，当然通过CM进行更为方便。

1>.点击主机，选择"所有主机"

2>.点击"HDFS DataNode"角色

3>.进入"node103.yinzhengjie.org.cn"节点的DataNode角色的配置界面（需要注意的是，此配置只针对该节点生效哟~）

4>.点击配置搜索关键字"DataNode 记录阈值"(英文页面为:"DataNode Logging Threshold")

5>.修改"node103.yinzhengjie.org.cn"节点的页面配置为"DEBUG"

6>.进入NodeManger的WebUI配置界面

7>.点击配置搜索关键字"NodeManager 记录阈值"(英文页面为:"NodeManager Logging Threshold")

 

二.运行wordcount

问题描述：
　　公司的某个开发人员尝试在集群上运行wordcount程序，但作业执行发生错误，请你帮助解决。
　　请将gateway机器的input.txxt文件上传到HDFS上的“/yinzhengjie/debug/mapreduce/data/input”目录中，并执行"wordcount /yinzhengjie/debug/mapreduce/data/input /yinzhengjie/debug/mapreduce/data/output"来测试是否可以运行。

解决方案：
　　通常作业失败类型只有目录存在，内存不足，权限部队等简单的几种，难度较低，一般不需要Java语言的知识，只要能看懂英文就可以解决。

1>.上传文件

[root@node101.yinzhengjie.org.cn ~]# ll
total 4
-rw-r--r-- 1 root root 553 Jun 15 17:38 input.txt
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# cat input.txt 
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -mkdir -p /yinzhengjie/debug/mapreduce/data/input
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -copyFromLocal input.txt /yinzhengjie/debug/mapreduce/data/input
[root@node101.yinzhengjie.org.cn ~]#
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -ls /yinzhengjie/debug/mapreduce/data/input
Found 1 items
-rw-r--r--   3 root supergroup        553 2019-06-15 17:53 /yinzhengjie/debug/mapreduce/data/input/input.txt
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 

2>.执行wordcount

[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# su hdfs
[hdfs@node101.yinzhengjie.org.cn /root]$ 
[hdfs@node101.yinzhengjie.org.cn /root]$ cd /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ 
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ hadoop jar hadoop-mapreduce-examples.jar wordcount /yinzhengjie/debug/mapreduce/data/input  /yinzhengjie/debug/mapreduce/data/output
19/06/15 18:00:46 INFO client.RMProxy: Connecting to ResourceManager at node101.yinzhengjie.org.cn/172.30.1.101:8032
19/06/15 18:00:47 INFO input.FileInputFormat: Total input paths to process : 1
19/06/15 18:00:47 INFO mapreduce.JobSubmitter: number of splits:1
19/06/15 18:00:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560591816714_0001
19/06/15 18:00:47 INFO impl.YarnClientImpl: Submitted application application_1560591816714_0001
19/06/15 18:00:47 INFO mapreduce.Job: The url to track the job: http://node101.yinzhengjie.org.cn:8088/proxy/application_1560591816714_0001/
19/06/15 18:00:47 INFO mapreduce.Job: Running job: job_1560591816714_0001
19/06/15 18:00:52 INFO mapreduce.Job: Job job_1560591816714_0001 running in uber mode : false
19/06/15 18:00:52 INFO mapreduce.Job:  map 0% reduce 0%
19/06/15 18:00:58 INFO mapreduce.Job:  map 100% reduce 0%
19/06/15 18:01:03 INFO mapreduce.Job:  map 100% reduce 13%
19/06/15 18:01:05 INFO mapreduce.Job:  map 100% reduce 38%
19/06/15 18:01:06 INFO mapreduce.Job:  map 100% reduce 50%
19/06/15 18:01:08 INFO mapreduce.Job:  map 100% reduce 63%
19/06/15 18:01:09 INFO mapreduce.Job:  map 100% reduce 88%
19/06/15 18:01:11 INFO mapreduce.Job:  map 100% reduce 100%
19/06/15 18:01:12 INFO mapreduce.Job: Job job_1560591816714_0001 completed successfully
19/06/15 18:01:12 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=976
        FILE: Number of bytes written=1349146
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=705
        HDFS: Number of bytes written=613
        HDFS: Number of read operations=27
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=16
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=8
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=3402
        Total time spent by all reduces in occupied slots (ms)=22251
        Total time spent by all map tasks (ms)=3402
        Total time spent by all reduce tasks (ms)=22251
        Total vcore-milliseconds taken by all map tasks=3402
        Total vcore-milliseconds taken by all reduce tasks=22251
        Total megabyte-milliseconds taken by all map tasks=3483648
        Total megabyte-milliseconds taken by all reduce tasks=22785024
    Map-Reduce Framework
        Map input records=1
        Map output records=87
        Map output bytes=901
        Map output materialized bytes=944
        Input split bytes=152
        Combine input records=87
        Combine output records=67
        Reduce input groups=67
        Reduce shuffle bytes=944
        Reduce input records=67
        Reduce output records=67
        Spilled Records=134
        Shuffled Maps =8
        Failed Shuffles=0
        Merged Map outputs=8
        GC time elapsed (ms)=453
        CPU time spent (ms)=9890
        Physical memory (bytes) snapshot=2097287168
        Virtual memory (bytes) snapshot=25250430976
        Total committed heap usage (bytes)=1470103552
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=553
    File Output Format Counters 
        Bytes Written=613
[hdfs@node101.yinzhengjie.org.cn /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce]$ 

 

三.