python下的jstack - pystack

背景

• python 多进程任务，卡在某个地方没有继续执行也没有报出异常，进程被hang住
• 日志没有捕获到相关信息，需要知道进程阻塞在哪里，为什么阻塞
• jvm提供了jstack、jmap类工具进行性能分析，python有没有类似于pystack的工具
• python jstack pystack 调试工具
• 依赖gdb
• 结论：http接口访问忘记设置超时，导致接口响应缓慢情况下任务死等。

安装

yum install gdb
pip install pystack-debugger

内网安装

pip download pystack-debugger
wget -c -t 10 10.222.77.121:12306/pystack_debugger-0.9.0-py2-none-any.whl
wget -c -t 10 10.222.77.121:12306/click-5.1-py2.py3-none-any.whl
pip install ./click-5.1-py2.py3-none-any.whl
pip install ./pystack_debugger-0.9.0-py2-none-any.whl

pystack 使用（其他作业，背景中debug作业信息已丢失）

• threads

pystack 22374

pystack 22374
Dumping Threads....


  File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 157, in wait
    time.sleep(self.watch_dog_duration)

---------------

  File "/data0/control_center/weibox/yarn-weibox/tools/waic_weibox_submit.py", line 17, in <module>
    kimy.submit()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 175, in submit
    self.job_execute()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 521, in job_execute
    (response_code, response_out, response_err) = helpers.run_shell("bash " + self.weibox_submit_name)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/helpers.py", line 218, in run_shell
    stdout, stderr = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>

---

• greenlet

pystack 22374 --include-greenlet

pystack 22374 --include-greenlet
Dumping Threads....


  File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 157, in wait
    time.sleep(self.watch_dog_duration)

---------------

  File "/data0/control_center/weibox/yarn-weibox/tools/waic_weibox_submit.py", line 17, in <module>
    kimy.submit()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 175, in submit
    self.job_execute()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 521, in job_execute
    (response_code, response_out, response_err) = helpers.run_shell("bash " + self.weibox_submit_name)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/helpers.py", line 218, in run_shell
    stdout, stderr = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>

Dumping Greenlets....


  File "/data0/control_center/weibox/yarn-weibox/tools/waic_weibox_submit.py", line 17, in <module>
    kimy.submit()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 175, in submit
    self.job_execute()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 521, in job_execute
    (response_code, response_out, response_err) = helpers.run_shell("bash " + self.weibox_submit_name)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/helpers.py", line 218, in run_shell
    stdout, stderr = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <genexpr>