Debugging a Stack Overflow
OSR Staff | Published: 16-Jul-03| Modified: 22-Jul-03
We recently were testing the latest version of our File System Filter Driver Development Kit and in the process of doing so the system crashed with a bugcheck 0x7F (UNEXPECTED_KERNEL_MODE_TRAP). In our experience this generally indicates a stack overflow condition (there are other potential causes as well, but we normally see stack overflows in the file system filter driver business.
This particular stack overflow was interesting because it appears to have occured in an Interrupt Service Routine. Thus, the purpose of this article is to demonstrate how we unraveled this to figure out what was on the REAL stack.
This crash occurred in a test scenario, so a debugger was actually attached to the machine. Thus we had a complete trace of the running system at the point of the crash.
Naturally, we started with the usual !analyze -v in order to figure out what had happened:
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind
that the kernel isn't allowed to have/catch (bound trap) or that
is always instant death (double fault). The first number in the
bugcheck parens is the number of the trap (8 = double fault, etc)
Consult an Intel x86 family manual to learn more about what these
traps are. Here is a *portion* of those codes:
If kv shows a taskGate
use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
use .trap on that value
Else
.trap on the appropriate frame will show where the trap was taken
(on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 00000008, EXCEPTION_DOUBLE_FAULT
Arg2: 80042000
Arg3: 00000000
Arg4: 00000000
Debugging Details:
------------------
Database SolnDb not connected
BUGCHECK_STR: 0x7f_8
TSS: 00000028 -- (.tss 28)
eax=00000008 ebx=f3b8e0a0 ecx=f4500fe8 edx=f3b8e0cc esi=f3b8eb5c edi=f3b8eb5c
eip=806c4fb4 esp=f3b8dff4 ebp=f3b8dff4 iopl=0 nv up di pl zr na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010046
hal!HalpIpiHandler+84:
806c4fb4 68e1000000 push 0xe1
Resetting default context
DEFAULT_BUCKET_ID: DRIVER_FAULT
LAST_CONTROL_TRANSFER: from 00000000 to 806c4fb4
STACK_TEXT:
f3b8dff4 00000000 00000000 00000000 00000000 hal!HalpIpiHandler+0x84
FOLLOWUP_IP:
hal!HalpIpiHandler+84
806c4fb4 68e1000000 push 0xe1
FOLLOWUP_NAME: MachineOwner
SYMBOL_NAME: hal!HalpIpiHandler+84
IMAGE_NAME: Unknown_Image
DEBUG_FLR_IMAGE_TIMESTAMP: 0
STACK_COMMAND: .tss 28 ; kb
BUCKET_ID: ZEROED_STACK
MODULE_NAME: Unknown_Module
Followup: MachineOwner
---------
This was actually a rather interesting analysis because it appears that the fault actually occurred in the IPI handler, which is a specialized interrupt routine that indicates the OS running on one processor was sending a message to itself on THIS processor!
So we proceeded to examine the stack:
0: kd> kv
ChildEBP RetAddr Args to Child
805452a8 804fe44e 00000003 805455d8 00000008 nt!RtlpBreakWithStatusInstruction (FPO: [1,0,0])
805452f4 804feed3 00000003 00000000 00000000 nt!KiBugCheckDebugBreak+0x19 (FPO: [Non-Fpo])
805456c0 804ff471 0000007f 00000008 80042000 nt!KeBugCheck2+0x46d (FPO: [Non-Fpo])
805456e0 8053c3e8 0000007f 00000008 80042000 nt!KeBugCheckEx+0x19 (FPO: [Non-Fpo])
805456e0 806c4fb4 0000007f 00000008 80042000 nt!KiTrap08+0x57 (FPO: TaskGate 28:0)
f3b8dff4 00000000 00000000 00000000 00000000 hal!HalpIpiHandler+0x84 (FPO: [0,2] TrapFrame-EDITED @ 00000000)
This did not yield much useful information for us. Note that the trap frame present on the stack is invalid (more on that later) and thus we already have a clue that this isn't going to be straight-forward.
The other point the analyze command made is that this is on a task gate, so we can try using the .tss command to switch to that task gate. Windows uses a task gate for the double fault handler because a double fault might indicate a stack overflow - requiring that the double fault handler have its own. On the x86 platform, this is achieved by indicating in the double fault handler's IDT entry that it is a task gate - and the task gate specifies the location for the new stack.
This gave us:
0: kd> .tss 28
eax=00000008 ebx=f3b8e0a0 ecx=f4500fe8 edx=f3b8e0cc esi=f3b8eb5c edi=f3b8eb5c
eip=806c4fb4 esp=f3b8dff4 ebp=f3b8dff4 iopl=0 nv up di pl zr na po nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010046
hal!HalpIpiHandler+84:
806c4fb4 68e1000000 push 0xe1
0: kd> kv
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
f3b8dff4 00000000 00000000 00000000 00000000 hal!HalpIpiHandler+0x84 (FPO: [0,2] TrapFrame-EDITED @ 00000000)
This doesn't really help us any more than the PREVIOUS stack frame (although we are at least on the original stack). From this, we concluded that it would be necessary to spend a little bit more time trying to unravel the stack.
We started by finding the actual stack boundaries using the !thread command:
0: kd> !thread
THREAD 811a4da8 Cid 078c.07a4 Teb: 7ffda000 Win32Thread: 00000000 RUNNING on processor 0
IRP List:
826e2e00: (0006,01fc) Flags: 40000900 Mdl: 00000000
8323cf68: (0006,0094) Flags: 40000000 Mdl: 00000000
Not impersonating
DeviceMap e1001940
Owning Process 81739448
Wait Start TickCount 814497 Elapsed Ticks: 0
Context Switch Count 224021
UserTime 00:00:00.0000
KernelTime 00:00:19.0406
Start Address kernel32 (0x77e7d342)
*** ERROR: Module load completed but symbols could not be loaded for MSVCRT.dll
Win32 Start Address MSVCRT (0x77c37de7)
Stack Init f3b91000 Current f3b8e80c Base f3b91000 Limit f3b8e000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 1 DecrementCount 16
ChildEBP RetAddr Args to Child
805452a8 804fe44e 00000003 805455d8 00000008 nt!RtlpBreakWithStatusInstruction (FPO: [1,0,0])
805452f4 804feed3 00000003 00000000 00000000 nt!KiBugCheckDebugBreak+0x19 (FPO: [Non-Fpo])
805456c0 804ff471 0000007f 00000008 80042000 nt!KeBugCheck2+0x46d (FPO: [Non-Fpo])
805456e0 8053c3e8 0000007f 00000008 80042000 nt!KeBugCheckEx+0x19 (FPO: [Non-Fpo])
805456e0 806c4fb4 0000007f 00000008 80042000 nt!KiTrap08+0x57 (FPO: TaskGate 28:0)
f3b8dff4 00000000 00000000 00000000 00000000 hal!HalpIpiHandler+0x84 (FPO: [0,2] TrapFrame-EDITED @ 00000000)
Thus, the stack base is at 0xf3b8e000. By examining the stack there, we can attempt to find the location at the point the interrupt occurred. From that, we can then feed the information into the kv command. We are looking for the location of the interrupt information: first EFLAGS, then CS and finally EIP. Of course, since the stack grows down we will observe these parameters in the opposite order. So we start by looking at the end of the stack (starting at the limit) trying to find something that looks "right" for an interrupt frame:
0: kd> dd f3b8e000
f3b8e000 00000000 00000000 00000000 00000000
f3b8e010 00000000 00000000 00000000 00000000
f3b8e020 00000000 00000000 00000000 00000000
f3b8e030 f3b8e0cc f4500fe8 00000008 00000000
f3b8e040 f3b8e0a0 00000000 f3b8eb5c f3b8eb5c
f3b8e050 00000000 f3b8e0b0 00000000 80528ef9
f3b8e060 00000008 00000286 00000005 00000003
f3b8e070 826e2e00 f4500fe8 f3b8eb5c 00000008
Note that the value at 0xf3b8e034 appears to have the correct format, since we expect to see 0x8 for the CS selector (that's the standard selector in Windows.) An EFLAGS value of zero is certainly not unreasonable, and 0xf4500fe8 seems plausible (interestingly, we never confirmed this was a valid EIP!)
Given that, we know the state of the stack at the time of entry to hal!HalpIpiHandler. That's because the stack pointer would have been f3b8e030 (just past the EIP argument). We know the new EIP (the interrupt routine) and thus only need to know the value to use for EBP. Normally this requires a bit more digging on the stack to find the EBP, but in this case we gave it the ESP address and (surprise!) it worked enough for the debugger to unravel the stack from that point (part of this has to do with the routines that are present on the end of the stack here).
Thus we had:
0: kd> kv = f3b8e030 f3b8e000 806c4fb4
ChildEBP RetAddr Args to Child
f3b8e030 f3b8e068 00000008 00000000 f3b8e0a0 hal!HalpIpiHandler+0x84 (FPO: [0,2] TrapFrame @ f3b8e030)
WARNING: Frame IP not in any known module. Following frames may be wrong.
f3b8e0a0 805337af 804e2038 ffffffff f3b8e1cc 0xf3b8e068
f3b8e0b0 80528f2b f3b8e0cc 00000008 00000000 nt!_except_handler2+0xb7
f3b8e1cc 8063e852 00000002 00000005 8197f32c nt!RtlCaptureStackBackTrace+0x29 (FPO: [Non-Fpo])
f3b8e1f0 8063eff3 f9ab8202 8f8f8f00 f9ab8202 nt!ViTrackIrqlLog+0x6e (FPO: [Non-Fpo])
f3b8e204 8063f180 8f8f8f00 817b4318 8f8f8f00 nt!KfSanityCheckLowerIrql+0x71 (FPO: [Non-Fpo])
f3b8e21c f9ab498f 81358958 817b4318 826e2e00 nt!VerifierKfReleaseSpinLock+0x36 (FPO: [Non-Fpo])
f3b8e278 f9ab5376 81333ee8 f3b8e4a8 81358958 osrgui!GuimonGetFullPath+0x9f (CONV: stdcall)
[e:\projects\fddk\samples\guimon\dd\guimon.c @ 1118]
f3b8e9d8 f44ff48d 818be258 826e2e00 f3b8ea34 osrgui!PreDoNothing+0x96 (CONV: stdcall)
[e:\projects\fddk\samples\guimon\dd\guimon.c @ 1556]
f3b8ea98 f4500fe8 81358958 826e2e00 81358958 osrfiltr!FilterDispatch+0x36d (CONV: stdcall)
[e:\projects\fddk\filter\fdispatch.cpp @ 311]
f3b8eb5c 804f53c1 81358958 826e2e00 806c52e4 osrfiltr!FilterReadWriteDispatch+0x148 (CONV: stdcall)
[e:\projects\fddk\filter\fdispatch.cpp @ 1709]
f3b8eb6c 8063d110 8137a720 826e2e00 826e2e00 nt!IopfCallDriver+0x31 (FPO: [0,0,1])
f3b8eb90 f36751cb 812f4cf0 81995160 804f53c1 nt!IovCallDriver+0x9e (FPO: [Non-Fpo])
*** ERROR: Symbol file could not be found. Defaulted to export symbols for SYMEVENT.SYS -
f3b8ebd0 8056e4d2 826e2fd8 826e2e00 81333ee8 SYMEVENT+0x61cb
f3b8ebe4 8056b822 812f4cf0 826e2e00 81333ee8 nt!IopSynchronousServiceTail+0x5e (FPO: [Non-Fpo])
f3b8ec7c 8053a6a4 00000218 00000000 00000000 nt!NtReadFile+0x536
f3b8ec7c 80504f19 00000218 00000000 00000000 nt!KiSystemService+0xc9 (FPO: [0,0] TrapFrame @ f3b8eca8)
*** ERROR: Module load completed but symbols could not be loaded for SAVRT.SYS
f3b8ed18 f3591d33 00000218 00000000 00000000 nt!ZwReadFile+0x11 (FPO: [9,0,0])
f3b8ed4c f3596365 e195d238 00000000 00000000 SAVRT+0x2ed33
*** ERROR: Module load completed but symbols could not be loaded for NAVEX15.SYS
f3b8ed84 f3316527 e195d238 f3b8edc4 00000013 SAVRT+0x33365
This is nothing more than a busy stack on a system where an inter-processor interrupt caused a transfer to the IPI handler! Once we had seen this, we decided to use the heuristic stack walker (from kdex2x86):
0: kd> !kdex2x86.stack
T. Address RetAddr Called Procedure
*0 F3B8E034 F4500FE8 osrfiltr!FilterDispatch(00000008, 00000000, F3B8E0A0,...);
*0 F3B8E074 F4500FE8 osrfiltr!FilterDispatch(F3B8EB5C, 00000008, 00000008,...);
*1 F3B8E0B4 80528F2B nt!RtlWalkFrameChain(F3B8E0CC, 00000008, 00000000,...);
*1 F3B8E0CC 80528F2B nt!RtlWalkFrameChain(8063E852, 8063EFF3, 8063F180,...);
*1 F3B8E0D0 8063E852 nt!RtlCaptureStackBackTrace(8063EFF3, 8063F180, F9AB498F,...);
*1 F3B8E0D4 8063EFF3 nt!ViTrackIrqlLog(8063F180, F9AB498F, F9AB5376,...);
*1 F3B8E0D8 8063F180 nt!KfSanityCheckLowerIrql(F9AB498F, F9AB5376, F44FF48D,...);
*1 F3B8E0DC F9AB498F nt!VerifierKfReleaseSpinLock(F9AB5376, F44FF48D, F4500FE8,...);
*1 F3B8E0E0 F9AB5376 osrgui!GuimonGetFullPath(F44FF48D, F4500FE8, F4500FE8,...);
*0 F3B8E0E8 F4500FE8 osrfiltr!FilterDispatch(F4500FE8, 826E0E00, 81358958,...);
*0 F3B8E0EC F4500FE8 osrfiltr!FilterDispatch(826E0E00, 81358958, 8197F54C,...);
*1 F3B8E0FC 80528F2B nt!RtlWalkFrameChain(8063E852, 8063EFF3, 805006BA,...);
*1 F3B8E100 8063E852 nt!RtlCaptureStackBackTrace(8063EFF3, 805006BA, 80503EF2,...);
*1 F3B8E104 8063EFF3 nt!ViTrackIrqlLog(805006BA, 80503EF2, F9AAC121,...);
*0 F3B8E108 805006BA nt!KiFlushSingleTb(80503EF2, F9AAC121, 00000206,...);
*1 F3B8E10C 80503EF2 dword ptr EAX(F9AAC121, 00000206, 80504025,...);
*1 F3B8E118 80504025 hal!HalRequestIpi(00000002, 0F380963, 00000002,...);
*1 F3B8E128 805008D8 nt!KiIpiSendPacket(00000002, 805006A4, 00000002,...);
*1 F3B8E150 8052072F nt!MiDecrementReferenceCount(0F380882, 0000F380, 0000F380,...);
*1 F3B8E164 80508D67 nt!MiDecrementShareCount(805549A0, C0C00000, 00000005,...);
*1 F3B8E178 80503F9B hal!HalRequestIpi(00000002, FFFFFFFD, 00000002,...);
*1 F3B8E188 805073B3 nt!KiIpiSend(811A4DA8, 811A4640, 80554801,...);
*1 F3B8E1A0 8053E5E4 nt!KeReleaseQueuedSpinLockFromDpcLevel(00000001, 804FF832, 80645612,...);
*1 F3B8E1A8 804FF832 nt!KiUnlockDispatcherDatabase(80645612, F9AB8228, F9AB4931,...);
*0 F3B8E1AC 80645612 nt!ViDeadlockCanProceed(F9AB8228, F9AB4931, 00000004,...);
*1 F3B8E1B4 F9AB4931 nt!VerifierKfAcquireSpinLock(00000004, 81358958, 805549C4,...);
*1 F3B8E1D0 8063E852 nt!RtlCaptureStackBackTrace(00000002, 00000005, 8197F32C,...);
*1 F3B8E1F4 8063EFF3 nt!ViTrackIrqlLog(F9AB8202, 8F8F8F00, F9AB8202,...);
*1 F3B8E208 8063F180 nt!KfSanityCheckLowerIrql(8F8F8F00, 817B4318, 8F8F8F00,...);
*1 F3B8E220 F9AB498F nt!VerifierKfReleaseSpinLock(81358958, 817B4318, 826E2E00,...);
*1 F3B8E27C F9AB5376 osrgui!GuimonGetFullPath(81333EE8, F3B8E4A8, 81358958,...);
*1 F3B8E2AC F9AB5376 osrgui!GuimonGetFullPath(81333EE8, F3B8E4D8, 81358958,...);
*1 F3B8E2D8 F9AB5376 osrgui!GuimonGetFullPath(81333EE8, F3B8E504, 81358958,...);
*1 F3B8E2F0 80528E27 nt!RtlpCaptureStackLimits(F3B8E348, 00000003, 80528EFE,...);
*1 F3B8E2FC 80528EFE nt!_SEH_epilog(00000005, 00000003, 81706F01,...);
*0 F3B8E30C F4500FE8 osrfiltr!FilterDispatch(F3B8EDF4, 000C6D3F, 00000000,...);
*1 F3B8E34C 80528E27 nt!RtlpCaptureStackLimits(F3B8E3A4, F3B8E384, 80528EFE,...);
*1 F3B8E358 80528EFE nt!_SEH_epilog(00000005, 00000003, 815A8601,...);
*0 F3B8E368 F4500FE8 osrfiltr!FilterDispatch(F3B8EE50, 00000008, 00000008,...);
*1 F3B8E378 F9AB5376 osrgui!GuimonGetFullPath(F3B91000, 00B91000, F3B8E3A4,...);
*1 F3B8E3A4 80528EFE nt!_SEH_epilog(80528F2B, F3B8E3C0, 00000008,...);
*1 F3B8E3A8 80528F2B nt!RtlWalkFrameChain(F3B8E3C0, 00000008, 815A8601,...);
*1 F3B8E3C0 80528F2B nt!RtlWalkFrameChain(8063E852, 8063EFF3, 8063F180,...);
*1 F3B8E3C4 8063E852 nt!RtlCaptureStackBackTrace(8063EFF3, 8063F180, F9AB498F,...);
*1 F3B8E3C8 8063EFF3 nt!ViTrackIrqlLog(8063F180, F9AB498F, F9AB5376,...);
*1 F3B8E3CC 8063F180 nt!KfSanityCheckLowerIrql(F9AB498F, F9AB5376, F44FF48D,...);
*1 F3B8E3D0 F9AB498F nt!VerifierKfReleaseSpinLock(F9AB5376, F44FF48D, F4500FE8,...);
*1 F3B8E3D4 F9AB5376 osrgui!GuimonGetFullPath(F44FF48D, F4500FE8, F4500FE8,...);
*0 F3B8E3DC F4500FE8 osrfiltr!FilterDispatch(F4500FE8, 805006BA, 80503EF2,...);
*0 F3B8E3E0 F4500FE8 osrfiltr!FilterDispatch(805006BA, 80503EF2, F9AAC121,...);
*0 F3B8E3E4 805006BA nt!KiFlushSingleTb(80503EF2, F9AAC121, 00000001,...);
*1 F3B8E3E8 80503EF2 dword ptr EAX(F9AAC121, 00000001, BAEBB201,...);
*1 F3B8E408 806C4FD2 nt!KiIpiServiceRoutine(F3B8E41C, 806C4FD8, 805006BA,...);
*0 F3B8E410 806C4FD8 hal!HalEndSystemInterrupt(805006BA, 80503EF2, F9AAC121,...);
*0 F3B8E414 805006BA nt!KiFlushSingleTb(80503EF2, F9AAC121, 00000001,...);
*1 F3B8E418 80503EF2 dword ptr EAX(F9AAC121, 00000001, E1A31201,...);
*1 F3B8E438 806C4FD2 nt!KiIpiServiceRoutine(F3B8E44C, 806C4FD8, 81358901,...);
*0 F3B8E440 806C4FD8 hal!HalEndSystemInterrupt(81358901, 000000E1, F3B8E55C,...);
*1 F3B8E450 806C4FD2 nt!KiIpiServiceRoutine(F3B8E464, 806C4FD8, 80503E02,...);
*0 F3B8E458 806C4FD8 hal!HalEndSystemInterrupt(80503E02, 000000E1, 00000001,...);
*0 F3B8E46C 805006BA nt!KiFlushSingleTb(80503EF2, F9AAC121, 00000001,...);
*1 F3B8E470 80503EF2 dword ptr EAX(F9AAC121, 00000001, 8056B201,...);
*1 F3B8E548 80528E27 nt!RtlpCaptureStackLimits(F3B8E5A0, F3B8E580, 80528EFE,...);
*1 F3B8E554 80528EFE nt!_SEH_epilog(00000005, 00000003, 8173AE01,...);
*0 F3B8E564 F4500FE8 osrfiltr!FilterDispatch(F3B8F04C, 00000008, 00000008,...);
*1 F3B8E5A0 80528EFE nt!_SEH_epilog(80528F2B, F3B8E5BC, 00000008,...);
*1 F3B8E5A4 80528F2B nt!RtlWalkFrameChain(F3B8E5BC, 00000008, 8173AE01,...);
*1 F3B8E5BC 80528F2B nt!RtlWalkFrameChain(8063E852, 8063EFF3, 8063F180,...);
*1 F3B8E5C0 8063E852 nt!RtlCaptureStackBackTrace(8063EFF3, 8063F180, F9AB498F,...);
*1 F3B8E5C4 8063EFF3 nt!ViTrackIrqlLog(8063F180, F9AB498F, F9AB5376,...);
*1 F3B8E5C8 8063F180 nt!KfSanityCheckLowerIrql(F9AB498F, F9AB5376, F44FF48D,...);
*1 F3B8E5CC F9AB498F nt!VerifierKfReleaseSpinLock(F9AB5376, F44FF48D, F4500FE8,...);
*1 F3B8E5D0 F9AB5376 osrgui!GuimonGetFullPath(F44FF48D, F4500FE8, F4500FE8,...);
*0 F3B8E5D8 F4500FE8 osrfiltr!FilterDispatch(F4500FE8, 8111ED58, 805006BA,...);
*0 F3B8E5DC F4500FE8 osrfiltr!FilterDispatch(8111ED58, 805006BA, 80503EF2,...);
*0 F3B8E5E4 805006BA nt!KiFlushSingleTb(80503EF2, F3B8E648, 80528E27,...);
*1 F3B8E5E8 80503EF2 dword ptr EAX(F3B8E648, 80528E27, F3B8E648,...);
*1 F3B8E5F0 80528E27 nt!RtlpCaptureStackLimits(F3B8E648, F3B8E628, 80528EFE,...);
*1 F3B8E5FC 80528EFE nt!_SEH_epilog(00000001, 00000003, 8320EE00,...);
*0 F3B8E60C 804FA8EE nt!IovFreeIrpPrivate(F3B8E7F0, 00000004, 000000E1,...);
*1 F3B8E648 80528EFE nt!_SEH_epilog(80528F2B, F3B8E664, 00000004,...);
*1 F3B8E64C 80528F2B nt!RtlWalkFrameChain(F3B8E664, 00000004, 8320EE00,...);
*1 F3B8E664 80528F2B nt!RtlWalkFrameChain(80643E54, 8063D061, 804FA8EE,...);
*1 F3B8E668 80643E54 nt!RtlCaptureStackBackTrace(8063D061, 804FA8EE, FFFFFFFF,...);
*1 F3B8E66C 8063D061 nt!VerifierIoFreeIrp(804FA8EE, FFFFFFFF, 80528EFE,...);
*1 F3B8E670 804FA8EE nt!IovFreeIrpPrivate(FFFFFFFF, 80528EFE, 80528F2B,...);
*1 F3B8E678 80528EFE nt!_SEH_epilog(80528F2B, F3B8E694, 00000004,...);
*1 F3B8E67C 80528F2B nt!RtlWalkFrameChain(F3B8E694, 00000004, 826E0E00,...);
*1 F3B8E694 80528F2B nt!RtlWalkFrameChain(80643E54, 8063D061, 804FA8EE,...);
*1 F3B8E698 80643E54 nt!RtlCaptureStackBackTrace(8063D061, 804FA8EE, 80528EFE,...);
*1 F3B8E69C 8063D061 nt!VerifierIoFreeIrp(804FA8EE, 80528EFE, C0C00000,...);
*1 F3B8E6A0 804FA8EE nt!IovFreeIrpPrivate(80528EFE, C0C00000, 805549A0,...);
*0 F3B8E6A4 80528EFE nt!_SEH_epilog(C0C00000, 805549A0, E19FA604,...);
*0 F3B8E6B4 805006BA nt!KiFlushSingleTb(00000206, 80504025, 00000002,...);
*0 F3B8E6BC 80504025 hal!HalRequestIpi(00000002, 036C5963, 00000002,...);
*0 F3B8E6DC 805008ED nt!KiIpiStallOnPacketTargets(00000001, 00001873, C0C0003C,...);
*1 F3B8E6FC 8051449D nt!MiUpdateWsle(F3B8E724, 811A4DA8, C0C00000,...);
*1 F3B8E720 805162FD nt!MiAddValidPageToWorkingSet(00000006, C0386924, 8116D400,...);
*1 F3B8E74C 805165C2 nt!MiResolveTransitionFault(E1A490D0, C0386924, C0300E18,...);
*0 F3B8E75C 805006BA nt!KiFlushSingleTb(80503EF2, F9AAC121, 00000001,...);
*1 F3B8E760 80503EF2 dword ptr EAX(F9AAC121, 00000001, 805F3201,...);
*1 F3B8E780 806C4FD2 nt!KiIpiServiceRoutine(F3B8E794, 806C4FD8, 804FA81C,...);
*0 F3B8E788 806C4FD8 hal!HalEndSystemInterrupt(804FA81C, 000000E1, 8063D061,...);
*1 F3B8E794 8063D061 nt!VerifierIoFreeIrp(81333EE8, 00000000, F3B8E7F0,...);
*1 F3B8E7E0 8051449D nt!MiUpdateWsle(F3B8E808, 811A4DA8, C0C00000,...);
*1 F3B8E804 805162FD nt!MiAddValidPageToWorkingSet(00000002, C0386924, 8116D400,...);
*1 F3B8E830 805165C2 nt!MiResolveTransitionFault(E1A490D6, C0386924, C0300E18,...);
*0 F3B8E860 8052FF49 nt!KeWaitForSingleObject(00000000, 00000000, 00000000,...);
*0 F3B8E89C 8052FAD3 hal!KeReleaseInStackQueuedSpinLock(03107FFF, 805549A0, 00000000,...);
*1 F3B8E8BC 8051E5E2 hal!KfLowerIrql(E1A490D6, 817B4318, F3B8EA88,...);
*1 F3B8E8F8 8053D2B8 nt!MmAccessFault(00000000, E1A490D6, 00000000,...);
*1 F3B8E91C 806C4FD2 nt!KiIpiServiceRoutine(F3B8E930, 806C4FD8, F3B8E902,...);
*0 F3B8E924 806C4FD8 hal!HalEndSystemInterrupt(F3B8E902, 000000E1, 80648ADD,...);
*1 F3B8E930 80648ADD hal!KfReleaseSpinLock(8123A008, 813F2BA0, F3B80000,...);
*1 F3B8E960 804F3747 hal!ExReleaseFastMutex(81358958, 817B4318, 826E2E00,...);
*1 F3B8E970 F44F5FDD nt!FsRtlLookupPerStreamContextInternal(E1A490D0, 817B40E8, E1A490D0,...);
*0 F3B8E9A0 80528EFE nt!_SEH_epilog(83254FC8, 00000004, 826E2FB4,...);
*1 F3B8E9B0 8056E4D2 nt!IovCallDriver(81358958, 826E2FB4, 81333EE8,...);
*1 F3B8EA10 8063D0FE nt!IovpCallDriver1(F36751CB, 8056E4D2, 00000000,...);
*1 F3B8EA14 F36751CB nt!IovCallDriver(8056E4D2, 00000000, 805337B0,...);
*1 F3B8EA18 8056E4D2 nt!IovCallDriver(00000000, 805337B0, 804E2038,...);
*1 F3B8EA4C 8064A1CC nt!RtlCaptureStackBackTrace(805006BA, 03503EF2, F9AAC121,...);
*0 F3B8EA50 805006BA nt!KiFlushSingleTb(03503EF2, F9AAC121, 00000001,...);
*0 F3B8EA60 805006BA nt!KiFlushSingleTb(80503EF2, F9AAC121, 005006BA,...);
*0 F3B8EA64 80503EF2 dword ptr EAX(F9AAC121, 005006BA, 81358A10,...);
*0 F3B8EA9C F4500FE8 osrfiltr!FilterDispatch(81358958, 826E2E00, 81358958,...);
*0 F3B8EABC 806476D5 nt!VfMajorIsValidIrpStatus(826E2FB4, 8123A350, 81358958,...);
*1 F3B8EAF4 8056E4D2 nt!IovCallDriver(826E2E00, 80648ADD, 826E2E00,...);
*1 F3B8EAFC 80648ADD hal!KfReleaseSpinLock(826E2E00, 8123A350, F3B8EB68,...);
*1 F3B8EB1C 8064A585 nt!VfIrpDatabaseEntryReleaseLock(81358958, 826E2E00, 806C52E4,...);
*1 F3B8EB60 804F53C1 dword ptr [ESI+EAX*4+38](81358958, 826E2E00, 806C52E4,...);
*1 F3B8EB70 8063D110 nt!IopfCallDriver(8137A720, 826E2E00, 826E2E00,...);
*1 F3B8EB94 F36751CB nt!IovCallDriver(812F4CF0, 81995160, 804F53C1,...);
*1 F3B8EBA0 804F53C1 dword ptr [ESI+EAX*4+38](812F4CF0, 826E2E00, 806C52E4,...);
*1 F3B8EBB0 8063D110 nt!IopfCallDriver(8323CF78, 806C52CC, 826E2E00,...);
*1 F3B8EBD4 8056E4D2 nt!IovCallDriver(826E2FD8, 826E2E00, 81333EE8,...);
*1 F3B8EBE8 8056B822 nt!IopSynchronousServiceTail(812F4CF0, 826E2E00, 81333EE8,...);
*1 F3B8EC1C 80503F9B hal!HalRequestIpi(826E2E00, FFFFFF01, 00000002,...);
*1 F3B8EC2C 805073B3 nt!KiIpiSend(811A4DA8, 811A4640, 00000101,...);
*1 F3B8EC80 8053A6A4 dword ptr EBX(00000218, 00000000, 00000000,...);
*1 F3B8ECBC 8053D2B8 nt!MmAccessFault(00000000, E195D238, 00000000,...);
*1 F3B8ED10 80504F19 nt!KiSystemService(00000008, 00000202, F3591D33,...);
*1 F3B8ED1C F3591D33 nt!ZwReadFile(00000218, 00000000, 00000000,...);
*1 F3B8ED50 F3596365 SAVRT+0002ECF0(E195D238, 00000000, 00000000,...);
*1 F3B8ED88 F3316527 dword ptr [ESI+08](E195D238, F3B8EDC4, 00000013,...);
*1 F3B8EDDC F33166D3 NAVEX15+000114D7(F359CA80, E195D238, 0000026C,...);
*1 F3B8FC84 F3317218 NAVEX15+00011695(00000001, 00016E68, C0C0003C,...);
*1 F3B8FCA4 8051449D nt!MiUpdateWsle(F3B8FCCC, 811A4DA8, C0C00000,...);
*1 F3B8FCC8 805162FD nt!MiAddValidPageToWorkingSet(00000016, C03CCDF4, 810DD478,...);
*1 F3B8FCF4 805165C2 nt!MiResolveTransitionFault(F337D3AC, C03CCDF4, C0300F30,...);
*0 F3B8FD24 8052FF49 nt!KeWaitForSingleObject(00000000, 00000000, 00000000,...);
*0 F3B8FD60 8052FAD3 hal!KeReleaseInStackQueuedSpinLock(03107F31, 805549A0, 00000000,...);
*1 F3B8FD80 8051E5E2 hal!KfLowerIrql(F337D3AC, F337D3B0, F3B90CF0,...);
*1 F3B8FDBC 8053D2B8 nt!MmAccessFault(00000000, F337D3AC, 00000000,...);
*1 F3B8FE30 F3317218 NAVEX15+00011695(F359CA80, E195D238, 0000026C,...);
*0 F3B8FEF4 F44FF6FC osrfiltr!CallAndRelease(81358A10, 8279CE00, 000001DE,...);
*1 F3B8FF2C 8063D0FE nt!IovpCallDriver1(00000000, 804F5CBB, 00000000,...);
*1 F3B8FF34 804F5CBB nt!IovCallDriver(00000000, 805337B0, 8279CF90,...);
*1 F3B8FF68 8064A1CC nt!RtlCaptureStackBackTrace(82456FC0, 034F5CBB, 00000000,...);
*1 F3B8FF94 8064A1CC nt!RtlCaptureStackBackTrace(8279CF90, F3B8FF80, 80530AF9,...);
*0 F3B8FFA0 80530AF9 hal!KfAcquireSpinLock(F3B90068, F4505198, F45091D8,...);
*0 F3B8FFB8 F4500FF9 osrfiltr!DereferenceExtension(81358A10, 000006B2, 81358958,...);
*0 F3B8FFD8 806476D5 nt!VfMajorIsValidIrpStatus(8279CFB4, 8139B350, 81358958,...);
*0 F3B8FFFC 80646112 dword ptr EAX(00000000, 00000000, 0039B3A0,...);
*1 F3B90010 804F5CBB nt!IovCallDriver(8279CE00, 80648ADD, 8279CE00,...);
*1 F3B90018 80648ADD hal!KfReleaseSpinLock(8279CE00, 8139B350, F3B90084,...);
*0 F3B90040 80506492 nt!KiComputeTimerTableIndex(FFB3B4C0, FFFFFFFF, A16ADFEC,...);
*1 F3B90058 8053E88C nt!KeReleaseQueuedSpinLockFromDpcLevel(F3B90CF0, 00000202, 8053E6CB,...);
*1 F3B90064 8053E6CB nt!SwapContext(F3B900AC, 811A4DA8, FFDFF120,...);
*1 F3B90078 805077AE dword ptr ESI(811A4E18, 811A4DA8, 8194B098,...);
*1 F3B90088 805004E8 nt!KiSwapThread(F3B900C0, 000ADE68, 80532E3B,...);
*1 F3B90094 80532E3B nt!ExfInterlockedCompareExchange64(F3B900A8, C03E7B70, 00000000,...);
*1 F3B900C4 8054138E nt!ExInsertTailNBQueue(81980008, C03E7B70, 00002F29,...);
*1 F3B90104 8051449D nt!MiUpdateWsle(F3B9012C, 811A4DA8, C0C00000,...);
*1 F3B90128 805146CD nt!InterlockedPushEntrySList(8108A108, 815B1DD0, 80516B1A,...);
*1 F3B90134 80516B1A nt!MiFreeInPageSupportBlock(815B1DD0, 806C58CC, C03CCC5C,...);
*1 F3B90158 8053E88C nt!KeReleaseQueuedSpinLockFromDpcLevel(F3B90CF0, 00000202, 8053E666,...);
*1 F3B90164 8053E666 nt!SwapContext(F3B901A8, 811A4DA8, 811A4DA8,...);
*1 F3B90178 804FF832 nt!KiUnlockDispatcherDatabase(00000000, 00000114, 8052FB66,...);
*1 F3B90184 8052FB66 nt!KeSetEventBoostPriority(8194B098, 805546B8, 03105248,...);
*1 F3B901AC 8051E5E2 hal!KfLowerIrql(F33171BC, 00000002, F3B90CF0,...);
*1 F3B901C8 8053E88C nt!KeReleaseQueuedSpinLockFromDpcLevel(FFFFFFFF, 00000202, 8053E787,...);
*1 F3B901D4 8053E787 nt!SwapContext(F3B901FC, F3378848, 00000002,...);
*1 F3B901E4 806C5DAB nt!KiDispatchInterrupt(00000000, F3B90CF0, 806C4FD8,...);
*1 F3B901F0 806C4FD8 hal!HalEndSystemInterrupt(00000000, 000000E1, F3378894,...);
*1 F3B9020C 8053D2B8 nt!MmAccessFault(00000000, F33137D6, 00000000,...);
*1 F3B90214 F33137D6 NAVEX15+00010213(00000000, F3B90220, 00000000,...);
*1 F3B90270 F3313B59 dword ptr [EDI-08](F359CA80, F337677C, F359CA80,...);
*1 F3B90288 F33137DD NAVEX15+0000EB2F(F359CA80, F3B90758, E1A027A8,...);
*0 F3B902A8 8064896F dword ptr EDI(813581B8, 813581B8, 806494CE,...);
*1 F3B902B4 806494CE nt!VfIrpDatabaseEntryReleaseLock(818BE258, 806C52E4, 815A8601,...);
*1 F3B902D0 8063D134 nt!VfDeadlockAfterCallDriver(818BE258, 82AA8E00, 8063D136,...);
*1 F3B902DC 8063D136 dword ptr ESI(81358958, 817B4318, 815A8601,...);
*1 F3B90300 F450457D nt!IovCallDriver(81358958, 817B4318, 815A8601,...);
*1 F3B90314 804F5CBB nt!IovCallDriver(00000103, 818BE258, F3B90304,...);
*1 F3B9033C F44FF6FC osrfiltr!CallAndRelease(81358A10, 82AA8E00, 000001DE,...);
*1 F3B90374 8063D0FE nt!IovpCallDriver1(00000000, 804F5CBB, 00000000,...);
*1 F3B9037C 804F5CBB nt!IovCallDriver(00000000, 805337B0, 82AA8F90,...);
*1 F3B903B0 8064A1CC nt!RtlCaptureStackBackTrace(81F7CFC0, 034F5CBB, 00000000,...);
*1 F3B903DC 8064A1CC nt!RtlCaptureStackBackTrace(82AA8F90, F3B903C8, 80530AF9,...);
*0 F3B903E8 80530AF9 hal!KfAcquireSpinLock(F3B904B0, F4505198, F45091D8,...);
*0 F3B90400 F4500FF9 osrfiltr!DereferenceExtension(81358A10, 000006B2, 81358958,...);
*0 F3B90420 806476D5 nt!VfMajorIsValidIrpStatus(82AA8FB4, 81336350, 81358958,...);
*0 F3B90444 80646112 dword ptr EAX(00000000, 00000000, 003363A0,...);
*1 F3B90458 804F5CBB nt!IovCallDriver(82AA8E00, 80648ADD, 82AA8E00,...);
*0 F3B90460 80648ADD hal!KfReleaseSpinLock(82AA8E00, 81336350, F3B904CC,...);
*0 F3B90488 80506492 nt!KiComputeTimerTableIndex(FFB3B4C0, FFFFFFFF, A06BC44E,...);
*1 F3B904A0 8053E88C nt!KeReleaseQueuedSpinLockFromDpcLevel(F3B90CF0, 00000202, 8053E6CB,...);
*1 F3B904AC 8053E6CB nt!SwapContext(F3B904F4, 811A4DA8, FFDFF120,...);
*1 F3B904C0 805077AE dword ptr ESI(811A4E18, 811A4DA8, 8194B098,...);
*1 F3B904D0 805004E8 nt!KiSwapThread(F3B90508, 000ADCDB, 80532E3B,...);
*1 F3B904DC 80532E3B nt!ExfInterlockedCompareExchange64(F3B904F0, C03E7BF4, 00000000,...);
*1 F3B9050C 8054138E nt!ExInsertTailNBQueue(81980008, C03E7BF4, 00002F21,...);
*1 F3B9054C 8051449D nt!MiUpdateWsle(F3B90574, 811A4DA8, C0C00000,...);
*1 F3B90570 805146CD nt!InterlockedPushEntrySList(810D3020, 815A86E0, 80516B1A,...);
*1 F3B9057C 80516B1A nt!MiFreeInPageSupportBlock(815A86E0, 806C58CC, C03CCC4C,...);
*1 F3B90590 80503F9B hal!HalRequestIpi(00000002, 00000000, F9AAC120,...);
*1 F3B905A0 805073B3 nt!KiIpiSend(811A4DA8, 811A4640, 00000101,...);
*1 F3B905B8 8053E5E4 nt!KeReleaseQueuedSpinLockFromDpcLevel(00000001, 804FF832, 00000000,...);
*1 F3B905C0 804FF832 nt!KiUnlockDispatcherDatabase(00000000, 00000114, 8052FB66,...);
*1 F3B905CC 8052FB66 nt!KeSetEventBoostPriority(8194B098, 805546B8, 030F7E06,...);
*1 F3B905F4 8051E5E2 hal!KfLowerIrql(F331373A, E1A027A8, F3B90CF0,...);
*1 F3B90630 8053D2B8 nt!MmAccessFault(00000000, F331373A, 00000000,...);
*1 F3B90644 8053D2B8 nt!MmAccessFault(00000000, E1A027AC, 00000000,...);
*1 F3B90664 F35752DE SAVRT+0002F190(00000000, 83668FD8, 813FA8F8,...);
*1 F3B906B8 F357534E dword ptr [ECX+20](F359CA80, 00000000, F3303288,...);
*1 F3B907F8 8051449D nt!MiUpdateWsle(F3B90820, 811A4DA8, C0C00000,...);
*1 F3B9081C 805146CD nt!InterlockedPushEntrySList(810DB5A0, 815A86E0, 80516B1A,...);
*1 F3B90828 80516B1A nt!MiFreeInPageSupportBlock(815A86E0, 806C58CC, C03CD5D4,...);
*0 F3B90880 8052FAD3 hal!KeReleaseInStackQueuedSpinLock(030F76AF, 805549A0, 00000000,...);
*1 F3B908A0 8051E5E2 hal!KfLowerIrql(F3575130, E1A027A8, F3B90CF0,...);
*1 F3B908DC 8053D2B8 nt!MmAccessFault(00000000, F3575130, 00000000,...);
*1 F3B90964 F3573558 SAVRT+00012130(E1A027A8, E1B54808, E152DD18,...);
*1 F3B90A04 805162FD nt!MiAddValidPageToWorkingSet(0000016A, C03CD5CC, 810FF6D8,...);
*1 F3B90A30 805165C2 nt!MiResolveTransitionFault(F3573410, C03CD5CC, C0300F34,...);
*1 F3B90A64 8053D2B8 nt!MmAccessFault(00000000, E19B4A78, 00000000,...);
*0 F3B90A9C 8052FAD3 hal!KeReleaseInStackQueuedSpinLock(02FC86A9, 805549A0, 00000000,...);
*1 F3B90ABC 8051E5E2 hal!KfLowerIrql(F3573410, E152DD18, F3B90CF0,...);
*1 F3B90AF8 8053D2B8 nt!MmAccessFault(00000000, F3573410, 00000000,...);
*1 F3B90B80 F356BA7D SAVRT+00010410(E1A027A8, E1DAC5F0, E19B4A80,...);
*1 F3B90BAC F356C81F SAVRT+000088B0(00000000, 00A2FF08, F356CAA4,...);
*1 F3B90BB8 F356CAA4 SAVRT+000097D0(F3B90BF4, F3580FAE, 00000006,...);
*1 F3B90BC0 F3580FAE dword ptr ECX(00000006, 00000000, 00000000,...);
*1 F3B90BDC F3581417 SAVRT+0001DF60(8323CF68, F3581521, F3B90BF4,...);
*1 F3B90BE4 F3581521 SAVRT+0001E380(F3B90BF4, 812FAEF8, 8323CF68,...);
*1 F3B90C0C F356BFE9 SAVRT+0001E4E0(812FAEF8, 8323CF68, 813F99F0,...);
*1 F3B90C20 804F53C1 dword ptr [ESI+EAX*4+38](812FAEF8, 8323CF68, 806C52E4,...);
*1 F3B90C30 8063D110 nt!IopfCallDriver(811A4FB8, 806C52CC, 8323CF68,...);
*0 F3B90C44 8063D553 nt!VerifierIoAllocateIrp1(8162F318, 8323CF68, F3B90C64,...);
*1 F3B90C54 8056E4D2 nt!IovCallDriver(8323CFD8, 817364A8, 8323CF68,...);
*1 F3B90C68 8056F1F6 nt!IopSynchronousServiceTail(812FAEF8, 8323CF68, 817364A8,...);
*1 F3B90CA8 805655FD nt!_SEH_epilog(80565C2E, 00A2FE70, C0100080,...);
*1 F3B90CAC 80565C2E nt!IopCreateFile(00A2FE70, C0100080, 812FAEF8,...);
*1 F3B90D04 80568288 nt!IopXxxControlFile(000001F8, 00000000, 00000000,...);
*0 F3B90D38 8053A6A4 dword ptr EBX(000001F8, 00000000, 00000000,...);
*0 F3B90DE0 8053EDE6 dword ptr EAX(BAE7398E, 813C4960, 00000000,...);
This (essentially) confirmed much of the stack (although, interestingly it does not confirm that initial part of the stack.) If we had been truly dedicated we would have finished confirming the initial stack, but instead stopped our analysis at this point because we were clear - the stack was blown, we understood WHY the stack was blown, and we concluded that in the future we needed to find ways to further minimize stack utilization, as well as reject I/O requests where there is not a sufficient amount of stack space to process the request.
The next time you see an 0x7F bugcheck, see if you can also reconstruct the stack!
