Hung task

[  480.725351] INFO: task insmod:521 blocked for more than 120 seconds.
[  480.731759]       Tainted: G           O    4.1.15 #5
[  480.736917] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  480.744842] [<c05278e8>] (__schedule) from [<c0527d34>] (schedule+0x40/0xa4)
[  480.752029] [<c0527d34>] (schedule) from [<c0527ec8>] (schedule_preempt_disabled+0x18/0x1c)
[  480.760479] [<c0527ec8>] (schedule_preempt_disabled) from [<c0529200>] (__mutex_lock_slowpath+0x6c/0xe4)
[  480.770066] [<c0529200>] (__mutex_lock_slowpath) from [<c05292bc>] (mutex_lock+0x44/0x48)
[  480.778363] [<c05292bc>] (mutex_lock) from [<bf026020>] (dlock_init+0x20/0x2c [dlock])
[  480.786402] [<bf026020>] (dlock_init [dlock]) from [<c0009558>] (do_one_initcall+0x90/0x1e8)
[  480.794897] [<c0009558>] (do_one_initcall) from [<c007ac4c>] (do_init_module+0x6c/0x1c0)
[  480.803085] [<c007ac4c>] (do_init_module) from [<c007c568>] (load_module+0x1690/0x1d34)
[  480.811188] [<c007c568>] (load_module) from [<c007cce8>] (SyS_init_module+0xdc/0x130)
[  480.819113] [<c007cce8>] (SyS_init_module) from [<c000f800>] (ret_fast_sys

linux 内核线程的3种常见hung panic检测机制 一、Hung task

1.1 原理

核心思想为创建一个内核监测进程(khungtaskd)循环(每隔CONFIG_DEFAULT_HUNG_TASK_TIMEOUT时间)监测处于D状态的每一个进程（任务），统计它们在两次检测之间的调度次数，如果发现有任务在两次监测之间没有发生任何的调度，则可判断该进程一直处于D状态，很有可能已经死锁，因此触发报警日志打印，输出进程的基本信息，栈回溯以及寄存器保存信息以供内核开发人员定位。

D状态：线程的一种等待状态TASK_UNINTERRUPTIBLE，该种状态下进程不接收信号，只能通过wake_up唤醒。

内核源码： /kernel/hung_task.c

1.2 可能原因

相关routine或内核线程使用的completion, mutex, wait even等同步机制，且超过CONFIG_DEFAULT_HUNG_TASK_TIMEOUT没被唤醒。二. Soft lockup三. Hard lockup

Linux的进程存在多种状态，如TASK_RUNNING的运行态、EXIT_DEAD的停止态和TASK_INTERRUPTIBLE的接收信号的等待状态等等（可在include/linux/sched.h中查看）。其中有一种状态等待为TASK_UNINTERRUPTIBLE，称为D状态，该种状态下进程不接收信号，只能通过wake_up唤醒。处于这种状态的情况有很多，例如mutex锁就可能会设置进程于该状态，有时候进程在等待某种IO资源就绪时(wait_event机制)会设置进程进入该状态。一般情况下，进程处于该状态的时间不会太久，但若IO设备出现故障或者出现进程死锁等情况，进程就可能长期处于该状态而无法再返回到TASK_RUNNING态。因此，内核为了便于发现这类情况设计出了hung task机制专门用于检测长期处于D状态的进程并发出告警。本文分析内核hung task机制的源码并给出一个示例演示。其核心思想为创建一个内核监测进程循环监测处于D状态的每一个进程（任务），统计它们在两次检测之间的调度次数，如果发现有任务在两次监测之间没有发生任何的调度则可判断该进程一直处于D状态，很有可能已经死锁，因此触发报警日志打印，输出进程的基本信息，栈回溯以及寄存器保存信息以供内核开发人员定位。D状态死锁一般在驱动开发的过程中比较常见，且不太容易定位，内核提供这种hung task机制，开发人员只需要将这些输出的定位信息抓取并保留下来就可以快速的进行定位。[ 480.725351] INFO: task insmod:521 blocked for more than 120 seconds. [ 480.725351] INFO: task insmod:521 blocked for more than 120 seconds.
[ 480.731759] Tainted: G O 4.1.15 #5
[ 480.736917] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.744842] [<c05278e8>] (__schedule) from [<c0527d34>] (schedule+0x40/0xa4)
[ 480.752029] [<c0527d34>] (schedule) from [<c0527ec8>] (schedule_preempt_disabled+0x18/0x1c)
[ 480.760479] [<c0527ec8>] (schedule_preempt_disabled) from [<c0529200>] (__mutex_lock_slowpath+0x6c/0xe4)
[ 480.770066] [<c0529200>] (__mutex_lock_slowpath) from [<c05292bc>] (mutex_lock+0x44/0x48)
[ 480.778363] [<c05292bc>] (mutex_lock) from [<bf026020>] (dlock_init+0x20/0x2c [dlock])
[ 480.786402] [<bf026020>] (dlock_init [dlock]) from [<c0009558>] (do_one_initcall+0x90/0x1e8)
[ 480.794897] [<c0009558>] (do_one_initcall) from [<c007ac4c>] (do_init_module+0x6c/0x1c0)
[ 480.803085] [<c007ac4c>] (do_init_module) from [<c007c568>] (load_module+0x1690/0x1d34)
[ 480.811188] [<c007c568>] (load_module) from [<c007cce8>] (SyS_init_module+0xdc/0x130)
[ 480.819113] [<c007cce8>] (SyS_init_module) from [<c000f800>] (ret_fast_syscall+0x0/0x54)