中国移动网站控件引发的蓝屏问题分析

本周四（2月23日），我接到了我们同事的一个奇怪的蓝屏case，据他回忆，他最近没有安装任何软件和驱动，也没有更改计算机的硬件配置，除了Windows后台进行的自动更新之外，他实在想不起来到底对计算机有什么额外的改变。可是突然，就从前一天23日周三晚上起，他的计算机就开始蓝屏，重启之后，进系统之前就会蓝屏，或者进了系统用不到一会儿也会蓝屏。因此，他怀疑是硬件（如内存）故障导致的，或者是 Windows Update 导致的问题。

 

照说，例如内存条松动的这种突发硬件故障的确有可能引发蓝屏，但是由于 Windows Update 推送的补丁缺陷导致的蓝屏可实属少见，在排查蓝屏问题时，我们一般应该遵从默认信任微软自身组件的原则。

 

据了解，他的蓝屏一般有几个随机的错误代码，查询 Debugging Help 之后，得到的解释如下：

蓝屏代码 (Bug Check)	关键参数 (Parameter)	说明
PFN_LIST_CORRUPT (0x4E)		This is typically caused by a driver passing a bad memory descriptor list. For example, the driver might have called MmUnlockPages twice with the same list. Stack trace examination is needed.
MEMORY_MANAGEMENT (0x1A)	P1: 0x41287	Internal memory management structures are corrupted. To further investigate the cause, a kernel memory dump file is needed.
NTFS_FILE_SYSTEM (0x24)		One possible cause of this bug check is disk corruption. Corruption in the NTFS file system or bad blocks (sectors) on the hard disk can induce this error. Corrupted SCSI and IDE drivers can also adversely affect the system's ability to read and write to disk, thus causing the error. Another possible cause is depletion of nonpaged pool memory. If the nonpaged pool memory is completely depleted, this error can stop the system. However, during the indexing process, if the amount of available nonpaged pool memory is very low, another kernel-mode driver requiring nonpaged pool memory can also trigger this error.
SYSTEM_SERVICE_EXCEPTION (0x3B)		This error has been linked to excessive paged pool usage and may occur due to user-mode graphics drivers crossing over and passing bad data to the kernel code.
SYSTEM_THREAD_EXCEPTION _NOT_HANDLED_M (1000007e)		This indicates that a system thread generated an exception which the error handler did not catch.

 

在以上错误中，前两种出现的频率最高。如果您百度一下以上错误，已经有足够的理由开始拆下内存条，擦拭金手指了。但是我个人认为，这一定不是一个硬件产生的错误。在我看来，这样的错误看似随机，其实应该有一种共性的可能性——系统中存在一个写的很烂的驱动。为什么这么讲呢，我们可以从查到的描述中看见"bad""depletion""nonpaged pool"出现的频率很高；另外值得注意的是，对于0x24 NTFS文件系统的 bug check，在很多时候容易产生磁盘损坏的误导，殊不知，它还有一种可能就是非换页池耗尽，如上表中我加粗了的部分。

 

对于如此随机的错误，我们往往是无法通过分析栈去找到凶手的。例如，我在这里给出一个栈的示例：

MEMORY_MANAGEMENT (1a)
# Any other values for parameter 1 must be individually examined.
Arguments:
Arg1: 0000000000041287, The subtype of the bugcheck.
Arg2: 0000000000000030
Arg3: 0000000000000000
Arg4: 0000000000000000

Debugging Details:
------------------

BUGCHECK_STR: 0x1a_41287
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
PROCESS_NAME: WmiPrvSE.exe
CURRENT_IRQL: 0
TRAP_FRAME: fffff88007e6d6e0 -- (.trap 0xfffff88007e6d6e0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.

STACK_TEXT:
fffff880`07e6d578 fffff800`02c62d7e : 00000000`0000001a 00000000`00041287 00000000`00000030 00000000`00000000 : nt!KeBugCheckEx
fffff880`07e6d580 fffff800`02ccdd6e : 00000000`00000000 00000000`00000030 00000000`00000000 00000000`fffffa80 : nt! ?? ::FNODOBFM::`string'+0x46485
fffff880`07e6d6e0 fffff800`02dadbc5 : 00000000`000af94a 00000000`00000000 ffffffff`ffffffff 00000000`01464000 : nt!KiPageFault+0x16e
fffff880`07e6d870 fffff800`02d426b0 : fffffa80`098d5058 fffff6fd`4004c6a8 fffff800`02f055c0 fffff880`07e6db11 : nt!MiResolvePageFileFault+0x1115
fffff880`07e6d9b0 fffff800`02cdea07 : 00000000`00000000 00000000`01440004 00000000`0240f3c4 fffff800`00000000 : nt! ?? ::FNODOBFM::`string'+0x399d4
fffff880`07e6dac0 fffff800`02ccdd6e : 00000000`00000001 00000000`01440004 00000000`023ae701 00000000`00000460 : nt!MmAccessFault+0x1e47
fffff880`07e6dc20 00000000`76b87222 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiPageFault+0x16e
00000000`0240f394 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x76b87222

可以看见，除了ntkrnlmp.exe里面的函数，最前面发生的0x76b87222根本无法解析出来。dds 命令也是不能够解析出具体名称的。

 

那么，究竟该怎么样才能找到问题的元凶呢？

其实，将随机的蓝屏错误通过启用特殊池来转化为明显的错误是比较好的选择。对于特殊池（special pool）的概念，我并不是第一次介绍了，关于这个神奇的特殊内存区域的调试方法，请参见我早些时候的文章《启用特殊池解读 0x000000c5 蓝屏》，或者其英文版《Enable "Special Pool" to Interpret 0x000000c5 Blue Screen》。

 

从安全模式启动系统，启动 verifier，配置启用 special pool. 当然，安全模式下，可能引发问题的驱动也许并未加载，因此，我们最好选择"从一个列表选择驱动程序名"，然后继续选择"将目前没有加载的驱动程序添加到列表中…"，在弹出的选择文件对话框中，浏览 %systemroot%\system32\derivers , 然后增加"版权"和"产品名称"两列文件属性，并按照它们排序。选择所有不是微软的程序驱动，或者选择没有数字签名/版权和产品信息不完整的看似不专业的驱动，添加进来选中应用特殊池即可。

 

 

这里需要说明一下，其实特殊池的设置保存在注册表之中，具体是在内存管理器的分支里：

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

它靠 DWord 值 VerifyDriverLevel 和 String 值 VerifyDrivers 控制。有兴趣大家可以去窥探一下 J

 

启用特殊池之后，我们就重启计算机，正常进入系统尝试 repro 这个问题。没一会儿，还没登录果然就又蓝了。这回直接进入安全模式，获得内存转储文件进行分析：

首先我们可以看见，特殊池生效了，而且成功进行了内存池分配：

4: kd> !verifier

Verify Level 1 ... enabled options are:
    Special pool

Summary of All Verifier Statistics

RaiseIrqls 0x0
AcquireSpinLocks 0x0
Synch Executions 0x0
Trims 0x0

Pool Allocations Attempted 0x2
Pool Allocations Succeeded 0x2
Pool Allocations Succeeded SpecialPool 0x2
Pool Allocations With NO TAG 0x0
Pool Allocations Failed 0x0
Resource Allocations Failed Deliberately 0x0

Current paged pool allocations 0x0 for 00000000 bytes
Peak paged pool allocations 0x0 for 00000000 bytes
Current nonpaged pool allocations 0x0 for 00000000 bytes
Peak nonpaged pool allocations 0x0 for 00000000 bytes

 

然后，我们可以直接看到问题驱动究竟是谁了：

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 00000000000000b2, MmMapLockedPages called on an MDL having incorrect flags.
    For example, calling MmMapLockedPages for an MDL
    that is already mapped to a system address is incorrect.
Arg2: fffffa800a4e71b0, MDL address.
Arg3: 0000000000000005, MDL flags.
Arg4: 0000000000000005, Incorrect MDL flags.

Debugging Details:
------------------
BUGCHECK_STR: 0xc4_b2
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
PROCESS_NAME: System
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from fffff8000311f3dc to fffff80002c95c40
STACK_TEXT:
fffff880`033697e8 fffff800`0311f3dc : 00000000`000000c4 00000000`000000b2 fffffa80`0a4e71b0 00000000`00000005 : nt!KeBugCheckEx
fffff880`033697f0 fffff800`0311ffb3 : fffff880`05926f60 fffff880`05926f60 fffffa80`069ce700 fffff800`0312e09a : nt!VerifierBugCheckIfAppropriate+0x3c
fffff880`03369830 fffff800`031327bb : fffffa80`0a4e71b0 fffffa80`09b69000 fffffa80`0a4e71b0 fffff880`05926f60 : nt!ViMmMapLockedPagesSanityChecks+0xa3
fffff880`03369870 fffff880`06220009 : fffffa80`0a4e72c0 ffffffff`8000069c fffffa80`0a4e72c0 00000000`00000000 : nt!VerifierMmMapLockedPages+0x1b
fffff880`033698b0 fffff880`0624c93a : fffff880`03369970 fffff880`05926f60 fffffa80`00000032 00000000`0000001c : PassGuard_x64!distorm_version+0x6809
fffff880`033698f0 fffff880`03369970 : fffff880`05926f60 fffffa80`00000032 00000000`0000001c fffffa80`06768f30 : PassGuard_x64!distorm_version+0x3313a
fffff880`033698f8 fffff880`05926f5f : fffffa80`00000032 00000000`0000001c fffffa80`06768f30 00000000`00000200 : 0xfffff880`03369970
fffff880`03369900 fffffa80`00000032 : 00000000`0000001c fffffa80`06768f30 00000000`00000200 00000000`00000000 : usbhub!UsbhSyncSendCommand+0x327
fffff880`03369908 00000000`0000001c : fffffa80`06768f30 00000000`00000200 00000000`00000000 fffff880`06232040 : 0xfffffa80`00000032
fffff880`03369910 fffffa80`06768f30 : 00000000`00000200 00000000`00000000 fffff880`06232040 00000000`001e001c : 0x1c
fffff880`03369918 00000000`00000200 : 00000000`00000000 fffff880`06232040 00000000`001e001c fffff880`062563f8 : 0xfffffa80`06768f30
fffff880`03369920 00000000`00000000 : fffff880`06232040 00000000`001e001c fffff880`062563f8 00000000`00220020 : 0x200

STACK_COMMAND: kb
FOLLOWUP_IP:
PassGuard_x64!distorm_version+6809
fffff880`06220009 4889442428 mov qword ptr [rsp+28h],rax
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: PassGuard_x64!distorm_version+6809
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: PassGuard_x64
IMAGE_NAME: PassGuard_x64.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 4e2fb9f4
FAILURE_BUCKET_ID: X64_0xc4_b2_VRF_PassGuard_x64!distorm_version+6809
BUCKET_ID: X64_0xc4_b2_VRF_PassGuard_x64!distorm_version+6809
Followup: MachineOwner
---------

4: kd> lmvm PassGuard_x64
start end module name
fffff880`06218000 fffff880`06261000 PassGuard_x64 (export symbols) PassGuard_x64.sys
Loaded symbol image file: PassGuard_x64.sys
Image path: \??\C:\windows\system32\drivers\PassGuard_x64.sys
Image name: PassGuard_x64.sys
Timestamp: Wed Jul 27 15:10:44 2011 (4E2FB9F4)
CheckSum: 0004A5F0
ImageSize: 00049000
File version: 1.0.0.6
Product version: 1.0.0.6
File flags: 0 (Mask 17)
File OS: 4 Unknown Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0804.04b0
ProductName: SysEnter Application
InternalName: SysEnter
OriginalFilename: SysEnter.exe
ProductVersion: 1, 0, 0, 6
FileVersion: 1, 0, 0, 6
FileDescription: SysEnter Application
LegalCopyright: Copyright (C) 2011

 

好了，知道了这个叫 PassGuard_x64.sys 的驱动是罪魁祸首之后，那我们就该移除它的启动加载了。直接在安全模式打开注册表编辑器，删除HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services 下面的 PassGuard 整个键，当然，你还需要找到 ControlSet001 / 002 下面的同样的键删除。这里，我顺便把这个 PassGuard 键的内容展示出来：

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PassGuard]
"Type"=dword:00000001
"Start"=dword:00000002
"ErrorControl"=dword:00000001
"ImagePath"=hex(2):5c,00,3f,00,3f,00,5c,00,43,00,3a,00,5c,00,77,00,69,00,6e,00,\
64,00,6f,00,77,00,73,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,\
00,5c,00,64,00,72,00,69,00,76,00,65,00,72,00,73,00,5c,00,50,00,61,00,73,00,\
73,00,47,00,75,00,61,00,72,00,64,00,5f,00,78,00,36,00,34,00,2e,00,73,00,79,\
00,73,00,00,00
"DisplayName"="PassGuard"
"WOW64"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\PassGuard\Enum]
"0"="Root\\LEGACY_PASSGUARD\\0000"
"Count"=dword:00000001
"NextInstance"=dword:00000001

删除这个键，如果没有其他的多重问题，那么系统就能正常运行了。在重启进入正常模式之前，记得删除 verifier 中的特殊池配置。

 

为了进一步删除该驱动所关联的程序或者其他文件（如果有的话），我请这位同事仔细回忆这是什么。于是，我叫他回忆一下任何可能的情况，例如IE插件的变化、某些程序里面的捆绑、恶意程序等等。结果，他想起周三下午有去中国移动的 10086.cn 充值话费，安装了一个安全控件。

 

我们打开IE加载项管理器，选择所有加载项，果然发现了这一个：

 

 

还好不是恶意软件，而且可以通过下面的 remove 按钮删除关联的控件文件。好了，不多说了，看看中国移动为64位操作系统写的驱动有多烂你就明白了。

 

从这个案例中，我想我们看到的不仅仅是特殊池的使用方法和排错思路，而且还应看到现在很多信息服务公司所面临的一个问题，自己的产品的驱动外包出去，而且承包开发驱动的公司没有足够的驱动撰写经验和规范，或者没有经过测试就投入使用，受损的不仅是客户，更是这个服务公司自身的品牌。这样的例子太多了，延伸起来不仅有 badly written driver, 还有 badly written software, badly written website… 铁道部12306网站说多了就没意思了，说句实话，现在中国联通在营业厅推行的银行卡缴费机我真是不敢用。想想要把银行卡插进联通外包商开发的机器中去，还要输入银行卡密码我就觉得胆战心惊。。。