大数据之Hadoop集群中Yarn常用命令
Yarn状态的查询,除了可以在hadoop103:8088页面查看以外,还可以通过命令操作。常见的命令操作如下所示。
需求:执行WordCount案例,并通过Yarn命令查看任务运行情况。原文:sw-code
# 启动集群
[hadoop@hadoop102 bin]$ myhadoop.sh start
# 运行wordcount案例
[hadoop@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
yar application查看任务#
1、列出所有Application:
[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -list
2022-06-06 16:51:19,896 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
2、根据Application状态过滤:yarn application -list -appStates [ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED]
[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
2022-06-06 16:53:49,886 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):2
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
application_1654496324557_0001 word count MAPREDUCE hadoop default FINISHED FAILED 100% http://hadoop102:19888/jobhistory/job/job_1654496324557_0001
application_1654496324557_0002 word count MAPREDUCE hadoop default FINISHED FAILED 100% http://hadoop102:19888/jobhistory/job/job_1654496324557_0002
2、Kill掉Application
[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1654496324557_0001
2022-06-06 16:57:34,082 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Application application_1654496324557_0001 has already finished
yarn logs查看日志#
1、查询Application日志:yarn logs -applicationId <ApplicationId>
[hadoop@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1654496324557_0001
2、查询Container日志:yarn logs -applicationId <ApplicationId> -containerId <ContainerId>
[hadoop@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1654496324557_0001 -containerId container_1654496324557_0001_01_000001
yarn applicationattempt查看尝试的任务#
1、列出所有Application尝试的列表:yarn applicationattempt -list <ApplicationId>
[hadoop@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1654496324557_0001
2022-06-06 17:05:10,914 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of application attempts :1
ApplicationAttempt-Id State AM-Container-Id Tracking-URL
appattempt_1654496324557_0001_000001 FINISHED container_1654496324557_0001_01_000001 http://hadoop103:8088/proxy/application_1654496324557_0001/
2、打印ApplicationAttempt状态:yarn applicationattempt -status <ApplicationAttemptId>
[hadoop@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1654496324557_0001_000001
2022-06-06 17:08:56,570 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Application Attempt Report :
ApplicationAttempt-Id : appattempt_1654496324557_0001_000001
State : FINISHED
AMContainer : container_1654496324557_0001_01_000001
Tracking-URL : http://hadoop103:8088/proxy/application_1654496324557_0001/
RPC Port : 32771
AM Host : hadoop103
Diagnostics : Task failed task_1654496324557_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
yarn container查看容器#
1、列出所有Container:yarn container -list <ApplicationAttemptId>
[hadoop@hadoop103 ~]$ yarn container -list appattempt_1654496324557_0004_000001
2022-06-06 17:14:40,313 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of containers :2
Container-Id Start Time Finish Time State Host Node Http Address LOG-URL
container_1654496324557_0004_01_000001 星期一 六月 06 17:13:55 +0800 2022 N/A RUNNING hadoop102:46450 http://hadoop102:8042 http://hadoop102:8042/node/containerlogs/container_1654496324557_0004_01_000001/hadoop
container_1654496324557_0004_01_000004 星期一 六月 06 17:14:38 +0800 2022 N/A RUNNING hadoop103:44058 http://hadoop103:8042 http://hadoop103:8042/node/containerlogs/container_1654496324557_0004_01_000004/hadoop
2、打印Container状态:yarn container -status <ContainerId>
[hadoop@hadoop103 ~]$ yarn container -status container_1654496324557_0006_01_000001
2022-06-06 17:20:51,357 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Container Report :
Container-Id : container_1654496324557_0006_01_000001
Start-Time : 1654507199188
Finish-Time : 0
State : RUNNING
Execution-Type : GUARANTEED
LOG-URL : http://hadoop103:8042/node/containerlogs/container_1654496324557_0006_01_000001/hadoop
Host : hadoop103:44058
NodeHttpAddress : http://hadoop103:8042
Diagnostics : null
注:只有任务跑的途中才能看到container的状态
yarn node查看节点状态#
[hadoop@hadoop103 ~]$ yarn node -list -all
2022-06-06 17:25:20,336 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total Nodes:3
Node-Id Node-State Node-Http-Address Number-of-Running-Containers
hadoop104:38945 RUNNING hadoop104:8042 0
hadoop103:44058 RUNNING hadoop103:8042 0
hadoop102:46450 RUNNING hadoop102:8042 0
yarn rmadmin更新配置#
加载队列配置:yarn rmadmin -refreshQueues
[hadoop@hadoop103 ~]$ yarn rmadmin -refreshQueues
2022-06-06 17:27:10,830 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8033
yarn queue查看队列#
打印队列信息:yarn queue -status <QueueName>
[hadoop@hadoop103 ~]$ yarn queue -status default
2022-06-06 17:28:54,282 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Queue Information :
Queue Name : default
State : RUNNING
Capacity : 100.0%
Current Capacity : .0%
Maximum Capacity : 100.0%
Default Node Label expression : <DEFAULT_PARTITION>
Accessible Node Labels : *
Preemption : disabled
Intra-queue Preemption : disabled
下篇文章:
相关文章:
大数据之Hadoop集群中MapReduce的Join操作
大数据之Hadoop集群的HDFS压力测试
作者:sw-code
出处:https://www.cnblogs.com/sw-code/p/16387038.html
版权:本作品采用「署名-非商业性使用-相同方式共享 4.0 国际」许可协议进行许可。
未经作者同意,请勿转载;若经同意转载,请在文章明显位置注明作者和出处。
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 一个费力不讨好的项目,让我损失了近一半的绩效!
· 清华大学推出第四讲使用 DeepSeek + DeepResearch 让科研像聊天一样简单!
· 实操Deepseek接入个人知识库
· CSnakes vs Python.NET:高效嵌入与灵活互通的跨语言方案对比
· Plotly.NET 一个为 .NET 打造的强大开源交互式图表库