一个rcu回调导致的简单死锁

在自有模块的处理中，我们设计了一个内核线程去做gc，

但同时，我们又用到了rcu，rcu中也会去抢gc的锁，由于该锁用的spin_lock，而不是spin_lock_bh,并没有关软中断，所以在rcu上下文中拿不到锁，造成死锁。

[106251.128106] NMI watchdog: BUG: soft lockup - CPU#45 stuck for 23s! [kflow_gcd:2791]
[106251.129425] Modules linked in: newsendfile(OE) witdriver(OE) mysendmsg(OE) xfs libcrc32c fuse tipc(OE) ossmod(OE) iptable_filter mptctl mptbase bonding dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure joydev sg mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_cpufreq acpi_power_meter tcp_bbr sch_fq binfmt_misc ip_tables ext4 mbcache jbd2 raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ahci libahci libata
[106251.129481]  ixgbe(OE) i40e(OE) mpt3sas(OE) ptp raid_class pps_core scsi_transport_sas i2c_core dca [last unloaded: witdriver]
[106251.129492] CPU: 45 PID: 2791 Comm: kflow_gcd Tainted: G        W  OEL ------------   3.10.0-693.21.1.el7.x86_64 #1
[106251.129493] Hardware name: ZTE ZXCDN/SC621DI-16F, BIOS 1.0b 09/21/2017
[106251.129496] task: ffff8839a2e70000 ti: ffff884f3fb34000 task.ti: ffff884f3fb34000
[106251.129498] RIP: 0010:[<ffffffff811005ce>]  [<ffffffff811005ce>] native_queued_spin_lock_slowpath+0x1ce/0x200
[106251.129508] RSP: 0018:ffff885fbdf43dd0  EFLAGS: 00000202
[106251.129510] RAX: 0000000000000001 RBX: 000000000000060c RCX: 0000000000000001
[106251.129512] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff885fbdf59b60
[106251.129513] RBP: ffff885fbdf43dd0 R08: 0000000000000101 R09: 0000000000000000
[106251.129515] R10: ffff88607fbc84a0 R11: ffffea00fba2ae00 R12: ffff885fbdf43d48
[106251.129517] R13: ffffffff816c6732 R14: ffff885fbdf43dd0 R15: ffff8852e87ccf40
[106251.129519] FS:  0000000000000000(0000) GS:ffff885fbdf40000(0000) knlGS:0000000000000000
[106251.129521] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[106251.129523] CR2: 00007f582cb9816c CR3: 0000000001a0a000 CR4: 00000000003607e0
[106251.129525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[106251.129527] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[106251.129528] Call Trace:
[106251.129530]  <IRQ>

[106251.129539]  [<ffffffff816adeee>] queued_spin_lock_slowpath+0xb/0xf
[106251.129546]  [<ffffffff816bb080>] _raw_spin_lock+0x20/0x30
[106251.129555]  [<ffffffffc055a3e5>] wit_inode_replace+0x3b5/0x400 [witdriver]
[106251.129561]  [<ffffffff8119243f>] ? free_pages.part.80+0x3f/0x50
[106251.129567]  [<ffffffffc055a534>] free_wit_flow+0x104/0x150 [witdriver]----------------rcu中调用
[106251.129571]  [<ffffffffc055a592>] release_wit_flow+0x12/0x20 [witdriver]
[106251.129577]  [<ffffffff81140340>] rcu_process_callbacks+0x1e0/0x580
[106251.129583]  [<ffffffff8109404d>] __do_softirq+0xfd/0x290
[106251.129588]  [<ffffffff816c8afc>] call_softirq+0x1c/0x30
[106251.129594]  [<ffffffff8102d435>] do_softirq+0x65/0xa0
[106251.129597]  [<ffffffff81094495>] irq_exit+0x175/0x180
[106251.129600]  [<ffffffff816c9e88>] smp_apic_timer_interrupt+0x48/0x60
[106251.129603]  [<ffffffff816c6732>] apic_timer_interrupt+0x162/0x170
[106251.129604]  <EOI>

[106251.129613]  [<ffffffff81347655>] ? __list_del_entry+0x35/0xd0
[106251.129615]  [<ffffffff813476fd>] list_del+0xd/0x30
[106251.129620]  [<ffffffffc055fdea>] prune_wait_free_list+0x9a/0x1c0 [witdriver]----------sys中调用
[106251.129624]  [<ffffffffc055ff65>] kflow_gcd_fn+0x55/0x1e0 [witdriver]
[106251.129628]  [<ffffffffc055ff10>] ? prune_wait_free_list+0x1c0/0x1c0 [witdriver]
[106251.129634]  [<ffffffff810b5241>] kthread+0xd1/0xe0
[106251.129637]  [<ffffffff810b5170>] ? insert_kthread_work+0x40/0x40
[106251.129641]  [<ffffffff816c5577>] ret_from_fork+0x77/0xb0
[106251.129644]  [<ffffffff810b5170>] ? insert_kthread_work+0x40/0x40

 

void wit_unref_flow(void *flow)
{
    struct wit_fq_flow *f = (struct wit_fq_flow*)flow;
    if ( !witdriver_init_done || !f ) return;
    if ( atomic_dec_and_test(&(f->users)) )
    {
        call_rcu(&f->rcu_head, release_wit_flow);
    }    
}

 

由于call_rcu其实是在软中断中完成的，

crash> irq -b
SOFTIRQ_VEC      ACTION
    [0]     ffffffff81092e40  <tasklet_hi_action>
    [1]     ffffffff8109bcf0  <run_timer_softirq>
    [2]     ffffffff81590d60  <net_tx_action>
    [3]     ffffffff81592b80  <net_rx_action>
    [4]     ffffffff81308250  <blk_done_softirq>
    [5]     ffffffff81358f00  <irq_poll_softirq>
    [6]     ffffffff81092d00  <tasklet_action>
    [7]     ffffffff810d6610  <run_rebalance_domains>
    [9]     ffffffff8113ea70  <rcu_process_callbacks>

release_wit_flow 会和gc流程抢一把锁，当gc流程拿了锁又被中断之后，执行到rcu的流程，执行 release_wit_flow 去抢锁，造成死锁。

所以需要加bh。

需要注意的是，这个软中断打印，很明显缺少[8],这个8对应的是啥呢？

crash> p softirq_to_name
softirq_to_name = $5 =
 {0xffffffff8195816a "HI", 0xffffffff818f79c0 "TIMER", 0xffffffff81914a4c "NET_TX", 0xffffffff81914a53 "NET_RX", 0xffffffff819547d4 "BLOCK", 0xffffffff81914a8a "BLOCK_IOPOLL", 0xffffffff81914a63 "TASKLET", 0xffffffff81914a6b "SCHED", 0xffffffff81914a71 "HRTIMER", 0xffffffff81914a79 "RCU"}

看起来【8】对应的是 HRTIMER ，而 HRTIMER 其实是在硬中断中实现的，而不是在软中断中实现的，所以这个看起来应该是废弃了。

hrtimers - High-resolution kernel timers，在rh的3.10内核中，高精度定时器是通过硬中断来实现的，
void __init hrtimers_init(void)
{
        hrtimer_cpu_notify(&hrtimers_nb, (unsigned long)CPU_UP_PREPARE,
                          (void *)(long)smp_processor_id());
        register_cpu_notifier(&hrtimers_nb);
}
对应的中断处理堆栈可以参考如下：
[ 6401.459859]  [<ffffffff816b9d53>] _raw_spin_lock_bh+0x33/0x40
[ 6401.466087]  [<ffffffffc0358971>] wit_hrtimer_pool_notify+0x181/0x3c0 [witdriver]-----这个是我们的timer的func
[ 6401.474060]  [<ffffffffc03587f0>] ? wit_send_tasklet+0x590/0x590 [witdriver]
[ 6401.481600]  [<ffffffff810b9096>] __hrtimer_run_queues+0xd6/0x260
[ 6401.488187]  [<ffffffff810b962f>] hrtimer_interrupt+0xaf/0x1d0
[ 6401.494566]  [<ffffffff8105467b>] local_apic_timer_interrupt+0x3b/0x60
[ 6401.501625]  [<ffffffff816c8e83>] smp_apic_timer_interrupt+0x43/0x60
[ 6401.508511]  [<ffffffff816c5732>] apic_timer_interrupt+0x162/0x170