DolphinScheduler集成Arthas实现接口调用监控，提升调度任务可靠性
本文介绍了在Apache DolphinScheduler中嵌入Arthas的方法，以实现对接口调用的监控。Arthas是一款强大的 Java诊断工具，能够帮助开发者实时查看应用程序的运行状态、性能瓶颈和方法调用情况。在DolphinScheduler中集成Arthas，可以方便地捕获任务调度时的关键调用信息，及时发现并解决性能问题，提高系统的稳定性。本文将详细说明如何在DolphinScheduler环境下启动Arthas，监控特定接口的调用，并分析收集到的性能数据，从而提升任务调度的可靠性和可维护性。
手动安装
https://arthas.aliyun.com/download/latest_version?mirror=aliyun
arthas-packaging-3.7.2-bin.zip
cp arthas-packaging-3.7.2-bin.zip /opt/arthas
cd /opt/arthas
unzip arthas-packaging-3.7.2-bin.zip
java -jar arthas-boot.jar
选择对应的进程号
报错解决
报错1
[ERROR] Start arthas failed, exception stack trace:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:102)
at com.taobao.arthas.core.Arthas.<init>(Arthas.java:27)
at com.taobao.arthas.core.Arthas.main(Arthas.java:161)
解决 :
进入 ${DOLPINSCHEUDLER_HOME}/api-server/bin下，在 jvm_args_env.sh 中添加如下 :
-XX:+StartAttachListener
报错2
Picked up JAVA_TOOL_OPTIONS:
java.io.IOException: well-known file /tmp/.java_pid731688 is not secure: file should be owned by the current user (which is 0) but is owned by 989
at sun.tools.attach.LinuxVirtualMachine.checkPermissions(Native Method)
at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:117)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:102)
at com.taobao.arthas.core.Arthas.<init>(Arthas.java:27)
at com.taobao.arthas.core.Arthas.main(Arthas.java:161)
[ERROR] Start arthas failed, exception stack trace:
[ERROR] attach fail, targetPid: 731688
解决
arthas启动的服务和dolpinscheduler启动服务所属的用户要一样，不然有如上的报错
Watch
Watch 用于监控方法的具体执行细节，如参数、返回值等
watch org.apache.dolphinscheduler.api.controller.UsersController queryUserList returnObj
[arthas@731688]$ watch org.apache.dolphinscheduler.api.controller.UsersController queryUserList returnObj
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 126 ms, listenerId: 2
method=org.apache.dolphinscheduler.api.controller.UsersController.queryUserList location=AtExit
ts=2024-08-27 02:04:01; [cost=4.918943ms] result=@Result[
code=@Integer[0],
msg=@String[成功],
data=@PageInfo[PageInfo(totalList=[User(id=1, userName=admin, userPassword=null, email=825193156@qq.com, phone=, userType=ADMIN_USER, tenantId=1, state=1, tenantCode=hdfs, queueName=default, alertGroup=null, queue=default, timeZone=null, createTime=Fri Jul 19 04:19:31 GMT-05:00 2024, updateTime=Mon Aug 12 22:15:58 GMT-05:00 2024)], total=1, totalPage=1, pageSize=10, currentPage=1, pageNo=0)],
]
method=org.apache.dolphinscheduler.api.controller.UsersController.queryUserList location=AtExit
ts=2024-08-27 02:04:18; [cost=6.905345ms] result=@Result[
code=@Integer[0],
msg=@String[成功],
data=@PageInfo[PageInfo(totalList=[User(id=1, userName=admin, userPassword=null, email=825193156@qq.com, phone=, userType=ADMIN_USER, tenantId=1, state=1, tenantCode=hdfs, queueName=default, alertGroup=null, queue=default, timeZone=null, createTime=Fri Jul 19 04:19:31 GMT-05:00 2024, updateTime=Mon Aug 12 22:15:58 GMT-05:00 2024)], total=1, totalPage=1, pageSize=10, currentPage=1, pageNo=0)],
]
method=org.apache.dolphinscheduler.api.controller.UsersController.queryUserList location=AtExit
ts=2024-08-27 02:04:27; [cost=5.803269ms] result=@Result[
code=@Integer[0],
msg=@String[成功],
data=@PageInfo[PageInfo(totalList=[User(id=1, userName=admin, userPassword=null, email=825193156@qq.com, phone=, userType=ADMIN_USER, tenantId=1, state=1, tenantCode=hdfs, queueName=default, alertGroup=null, queue=default, timeZone=null, createTime=Fri Jul 19 04:19:31 GMT-05:00 2024, updateTime=Mon Aug 12 22:15:58 GMT-05:00 2024)], total=1, totalPage=1, pageSize=10, currentPage=1, pageNo=0)],
]
Trace
Trace 用于监控方法调用的深度，包括调用了哪些方法以及每个方法的执行时间。
[arthas@973263]$ trace org.apache.dolphinscheduler.api.controller.UsersController queryUserList
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 319 ms, listenerId: 1
`---ts=2024-08-27 10:33:08;thread_name=qtp1836984213-26;id=26;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@439f5b3d
`---[13.962731ms] org.apache.dolphinscheduler.api.controller.UsersController:queryUserList()
+---[0.18% 0.025123ms ] org.apache.dolphinscheduler.api.controller.UsersController:checkPageParams() #130
+---[0.09% 0.012549ms ] org.apache.dolphinscheduler.plugin.task.api.utils.ParameterUtils:handleEscapes() #131
`---[96.47% 13.469876ms ] org.apache.dolphinscheduler.api.service.UsersService:queryUserList() #132
Dump
heapdump arthas-output/dump.hprof 生成堆转储文件：
[arthas@973263]$ heapdump arthas-output/dump.hprof
Dumping heap to arthas-output/dump.hprof ...
Heap dump file created
使用MAT进行内存泄漏分析。
查看jvm内存变化
memory查看JVM内存
[arthas@973263]$ memory
Memory used total max usage
heap 485M 900M 900M 53.91%
ps_eden_space 277M 327M 358M 77.61%
ps_survivor_space 61M 61M 61M 99.98%
ps_old_gen 146M 512M 512M 28.54%
nonheap 162M 188M -1 85.96%
code_cache 11M 32M 240M 4.89%
metaspace 135M 140M -1 96.67%
compressed_class_space 14M 15M 1024M 1.43%
direct 949K 949K - 100.00%
mapped 0K 0K - 0.00%
查看CPU使用率
dashboard 可以查看CPU使用率，查看是哪个线程的，通过 thread -n 线程id查看：
转载自Journey
原文链接：https://segmentfault.com/a/1190000045219355
本文由 白鲸开源 提供发布支持！
