windbg

http://www.codemachine.com/article_x64deepdive.html

https://sites.google.com/site/jozsefbekes/Home/windows-programming/windbg

http://www.codeproject.com/Articles/Toby-Opferman#Article

http://www.nynaeve.net/?cat=2

https://blogs.msdn.microsoft.com/alejacma/2009/07/24/managed-debugging-with-windbg-call-stacks-part-1/

https://blogs.msdn.microsoft.com/jmstall/page/2/

https://blogs.msdn.microsoft.com/alejacma

Debugging Tools for Windows (WinDbg, KD, CDB, NTSD)

https://msdn.microsoft.com/zh-cn/library/windows/hardware/ff551063(v=vs.85).aspx

 

Yeah, that’s not very helpful. Here’s how to get from that terse description to something useful.

As an example, let’s use the following trivial program:

using System;

public class Simple
{
   public static int Main()
   {
       Console.WriteLine( "Hello World!" );
       Console.ReadLine();
       
       return 0;
   }
}

Dump this snippet into Notepad, save as simple.cs file and compile it. We want to see what it’ll be like when the user runs it, so we’ll do a release build. From the VS2005 Command Prompt, run csc /debug:pdbonly /o+ simple.cs. (I find it’s quicker to do command-line compiles for simple test programs than firing up Visual Studio. It’s not as if we’re designing a Form.) If you’re doing this on a 64–bit computer, like me, add /platform:x86 so we get consistent output across platforms. (Marking as x86 means that the executable headers are set to indicate a 32–bit program. The IL is the same whatever you set here, but if you leave it as anycpu, the default, the .NET Framework will create a 64–bit process which obviously gives different output.)

Why call Console.ReadLine? Well, the JIT compiler is helpful (or at least it thinks it is). If you start a program under the debugger, it will generate less optimized code because it thinks you want to debug it, but that changes the code from what the user will see. So I want to add a stop point in the program so I can attach the debugger after the process has started.

The next thing you’ll need is the Debugging Tools for Windows kit. Grab the 32–bit kit – the 64–bit kit can debug 32–bit processes, but SOS (the debugger extension DLL which implements the commands we’re going to use) doesn’t appear to work. We’re going to use WinDBG, which is slightly friendlier than the other debuggers although not a lot!

Open WinDBG. Run simple.exe and, when it’s waiting for input, go to File/Attach to a Process in WinDBG. You’ll see a list of other processes (and if running Windows Vista with UAC enabled, or on XP as a standard user, a load of access denied errors). Select simple.exe from the list and hit OK. WinDBG automatically stops once you’ve attached to a process so you can start manipulating the program straight away. This is the output I got:

Microsoft (R) Windows Debugger Version 6.8.0004.0 X86
Copyright (c) Microsoft Corporation. All rights reserved.

*** wait with pending attach
Symbol search path is: C:\Windows;C:\Windows\System32;C:\Windows\SysWOW64;SRV*C:\WebSymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
ModLoad: 008c0000 008c8000   C:\Users\Mike\Documents\Programming\simple.exe
ModLoad: 776a0000 77800000   C:\Windows\SysWOW64\ntdll.dll
ModLoad: 79000000 79046000   C:\Windows\system32\mscoree.dll
ModLoad: 75b70000 75c80000   C:\Windows\syswow64\KERNEL32.dll
ModLoad: 76ce0000 76da6000   C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 76e10000 76f00000   C:\Windows\syswow64\RPCRT4.dll
ModLoad: 75850000 758b0000   C:\Windows\syswow64\Secur32.dll
ModLoad: 75a80000 75ad8000   C:\Windows\syswow64\SHLWAPI.dll
ModLoad: 77110000 771a0000   C:\Windows\syswow64\GDI32.dll
ModLoad: 77040000 77110000   C:\Windows\syswow64\USER32.dll
ModLoad: 76c30000 76cda000   C:\Windows\syswow64\msvcrt.dll
ModLoad: 75d00000 75d60000   C:\Windows\system32\IMM32.DLL
ModLoad: 76940000 76a08000   C:\Windows\syswow64\MSCTF.dll
ModLoad: 75a70000 75a79000   C:\Windows\syswow64\LPK.DLL
ModLoad: 759c0000 75a3d000   C:\Windows\syswow64\USP10.dll
ModLoad: 746c0000 7485e000   C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6001.18000_none_5cdbaa5a083979cc\comctl32.dll
ModLoad: 79e70000 7a3ff000   C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
ModLoad: 75390000 7542b000   C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.1434_none_d08b6002442c891f\MSVCR80.dll
ModLoad: 75e30000 7693f000   C:\Windows\syswow64\shell32.dll
ModLoad: 771b0000 772f4000   C:\Windows\syswow64\ole32.dll
ModLoad: 790c0000 79bf6000   C:\Windows\assembly\NativeImages_v2.0.50727_32\mscorlib\5b3e3b0551bcaa722c27dbb089c431e4\mscorlib.ni.dll
ModLoad: 79060000 790b6000   C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorjit.dll
(1dc.12b4): Break instruction exception - code 80000003 (first chance)
eax=7efaf000 ebx=00000000 ecx=00000000 edx=7770d2d4 esi=00000000 edi=00000000
eip=776b0004 esp=04c6fa00 ebp=04c6fa2c iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!DbgBreakPoint:
776b0004 cc              int     3

You enter your debugging commands in the edit box at the bottom, next to the prompt 0:003>. This means we’re debugging the 0th process we’re attached to, and our commands by default apply to thread number 3. Four threads? Well, thread 0 is our main thread, thread 1 was created by the CLR’s debugging support in case we attached a debugger, thread 2 is the finalizer thread, and thread 3 was just created by WinDBG so it could stop the process safely. (You can see this by running ~* k, although you’ll need to be set up to get debugging symbols from the symbol server to get good stack traces.) The Debugging Tools debuggers automatically stop the process when you attach, so that you can start manipulating the process straight away.

The first thing we need to do is ask WinDBG to load the SOS extension. This is installed with the CLR, so we ask it to load from the same folder that mscorwks.dll (the DLL which implements the virtual machine, effectively the guts of the CLR) lives in:

0:003> .loadby sos.dll mscorwks

Now we can see what state of the managed threads are in by running !threads. Any command beginning ! comes from a debugger extension DLL, and they can be disambiguated if necessary, but they’re searched in reverse order loaded (last loaded = first searched) so anything we want will be found in SOS anyway.

Right. To see what we can do, run !help.

0:003> !help
-------------------------------------------------------------------------------
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
Type "!help " for detailed info on that function. 

Object Inspection                  Examining code and stacks
-----------------------------      -----------------------------
DumpObj (do)                       Threads
DumpArray (da)                     CLRStack
DumpStackObjects (dso)             IP2MD
DumpHeap                           U
DumpVC                             DumpStack
GCRoot                             EEStack
ObjSize                            GCInfo
FinalizeQueue                      EHInfo
PrintException (pe)                COMState
TraverseHeap                       BPMD 

Examining CLR data structures      Diagnostic Utilities
-----------------------------      -----------------------------
DumpDomain                         VerifyHeap
EEHeap                             DumpLog
Name2EE                            FindAppDomain
SyncBlk                            SaveModule
DumpMT                             GCHandles
DumpClass                          GCHandleLeaks
DumpMD                             VMMap
Token2EE                           VMStat
EEVersion                          ProcInfo 
DumpModule                         StopOnException (soe)
ThreadPool                         MinidumpMode 
DumpAssembly                       
DumpMethodSig                      Other
DumpRuntimeTypes                   -----------------------------
DumpSig                            FAQ
RCWCleanupList
DumpIL

To find out a little more about !U, let’s run !help U:

0:003> !help U
-------------------------------------------------------------------------------
!U [-gcinfo] [-ehinfo] <MethodDesc address> | <Code address> 

Presents an annotated disassembly of a managed method when given a MethodDesc
pointer for the method, or a code address within the method body. Unlike the
debugger "U" function, the entire method from start to finish is printed,
with annotations that convert metadata tokens to names.


...
03ef015d b901000000       mov     ecx,0x1
03ef0162 ff156477a25b     call   dword ptr [mscorlib_dll+0x3c7764 (5ba27764)] (System.Console.InitializeStdOutError(Boolean), mdToken: 06000713)
03ef0168 a17c20a701       mov     eax,[01a7207c] (Object: SyncTextWriter)
03ef016d 89442414         mov     [esp+0x14],eax

If you pass the -gcinfo flag, you'll get inline display of the GCInfo for
the method. You can also obtain this information with the !GCInfo command.

If you pass the -ehinfo flag, you'll get inline display of exception info
for the method. (Beginning and end of try/finally/catch handlers, etc.).
You can also obtain this information with the !EHInfo command.

Right, so we need the address of a MethodDesc structure, or the address of the code itself. How can we get one of these? There’s a command called DumpMD…

0:003> !help dumpmd
-------------------------------------------------------------------------------
!DumpMD 

This command lists information about a MethodDesc. You can use !IP2MD to turn 
a code address in a managed function into a MethodDesc:

0:000> !dumpmd 902f40
Method Name: Mainy.Main()
Class: 03ee1424
MethodTable: 009032d8
mdToken: 0600000d
Module: 001caa78
IsJitted: yes
m_CodeOrIL: 03ef00b8

If IsJitted is "yes," you can run !U on the m_CodeOrIL pointer to see a 
disassembly of the JITTED code. You can also call !DumpClass, !DumpMT, 
!DumpModule on the Class, MethodTable and Module fields above.

We’re getting a bit closer. Let’s try DumpModule…

0:003> !help DumpModule
-------------------------------------------------------------------------------
!DumpModule [-mt] 

You can get a Module address from !DumpDomain, !DumpAssembly and other 
functions. Here is sample output:

0:000> !dumpmodule 1caa50
Name: C:\pub\unittest.exe
Attributes: PEFile
Assembly: 001ca248
LoaderHeap: 001cab3c
TypeDefToMethodTableMap: 03ec0010
TypeRefToMethodTableMap: 03ec0024
MethodDefToDescMap: 03ec0064
FieldDefToDescMap: 03ec00a4
MemberRefToDescMap: 03ec00e8
FileReferencesMap: 03ec0128
AssemblyReferencesMap: 03ec012c
MetaData start address: 00402230 (1888 bytes)

The Maps listed map metadata tokens to CLR data structures. Without going into 
too much detail, you can examine memory at those addresses to find the 
appropriate structures. For example, the TypeDefToMethodTableMap above can be 
examined:

0:000> dd 3ec0010
03ec0010  00000000 00000000 0090320c 0090375c
03ec0020  009038ec ...

This means TypeDef token 2 maps to a MethodTable with the value 0090320c. You 
can run !DumpMT to verify that. The MethodDefToDescMap takes a MethodDef token 
and maps it to a MethodDesc, which can be passed to !DumpMD.

There is a new option "-mt", which will display the types defined in a module,
and the types referenced by the module. For example:

0:000> !dumpmodule -mt 1aa580
Name: C:\pub\unittest.exe
......
MetaData start address: 0040220c (1696 bytes)

Types defined in this module

      MT    TypeDef Name
------------------------------------------------------------------------------
030d115c 0x02000002 Funny
030d1228 0x02000003 Mainy

Types referenced in this module

      MT    TypeRef Name
------------------------------------------------------------------------------
030b6420 0x01000001 System.ValueType
030b5cb0 0x01000002 System.Object
030fceb4 0x01000003 System.Exception
0334e374 0x0100000c System.Console
03167a50 0x0100000e System.Runtime.InteropServices.GCHandle
0336a048 0x0100000f System.GC

We still need a Module to pass to this command, but we’re getting there. Maybe DumpDomain can help us? I’ll skip the help text this time.

0:003> !DumpDomain
--------------------------------------
System Domain: 7a3bc8b8
LowFrequencyHeap: 7a3bc8dc
HighFrequencyHeap: 7a3bc934
StubHeap: 7a3bc98c
Stage: OPEN
Name: None
--------------------------------------
Shared Domain: 7a3bc560
LowFrequencyHeap: 7a3bc584
HighFrequencyHeap: 7a3bc5dc
StubHeap: 7a3bc634
Stage: OPEN
Name: None
Assembly: 0052f848
--------------------------------------
Domain 1: 00516800
LowFrequencyHeap: 00516824
HighFrequencyHeap: 0051687c
StubHeap: 005168d4
Stage: OPEN
SecurityDescriptor: 00517d30
Name: simple.exe
Assembly: 0052f848 [C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll]
ClassLoader: 0050e470
SecurityDescriptor: 0051f5d0
  Module Name
790c2000 C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll

Assembly: 00535ac0 [C:\Users\Mike\Documents\Programming\simple.exe]
ClassLoader: 0050e630
SecurityDescriptor: 00535a38
  Module Name
000f2c3c C:\Users\Mike\Documents\Programming\simple.exe

At last, we have the address of something we can use. Let’s get the module information for simple.exe. For that we need the address next to it in the module list for the simple.exe assembly.

0:003> !dumpmodule -mt f2c3c
Name: C:\Users\Mike\Documents\Programming\simple.exe
Attributes: PEFile 
Assembly: 00535ac0
LoaderHeap: 00000000
TypeDefToMethodTableMap: 000f0038
TypeRefToMethodTableMap: 000f0040
MethodDefToDescMap: 000f005c
FieldDefToDescMap: 000f0068
MemberRefToDescMap: 000f006c
FileReferencesMap: 000f0088
AssemblyReferencesMap: 000f008c
MetaData start address: 008c206c (740 bytes)

Types defined in this module

      MT    TypeDef Name
------------------------------------------------------------------------------
000f3030 0x02000002 Simple

Types referenced in this module

      MT    TypeRef Name
------------------------------------------------------------------------------
790fd0f0 0x01000001 System.Object
79101118 0x01000006 System.Console

Now we can get the MethodTable for the Simple class:

0:003> !dumpmt -md f3030
EEClass: 000f1174
Module: 000f2c3c
Name: Simple
mdToken: 02000002  (C:\Users\Mike\Documents\Programming\simple.exe)
BaseSize: 0xc
ComponentSize: 0x0
Number of IFaces in IFaceMap: 0
Slots in VTable: 6
--------------------------------------
MethodDesc Table
   Entry MethodDesc      JIT Name
79371278   7914b928   PreJIT System.Object.ToString()
7936b3b0   7914b930   PreJIT System.Object.Equals(System.Object)
7936b3d0   7914b948   PreJIT System.Object.GetHashCode()
793624d0   7914b950   PreJIT System.Object.Finalize()
00310070   000f3020      JIT Simple.Main()
000fc01c   000f3028     NONE Simple..ctor()

Finally we have something we can pass to !U. Note the JIT column. Here, PreJIT means that the code has come from a native image generated by ngen, JIT means that the JIT compiler in this process has generated the code, and NONE means that the method hasn’t been compiled yet. We didn’t override any of System.Object’s virtual methods, so they occupy the first four places in our MethodTable.

I placed the call to Console.ReadLine at the end of the program so everything has already been compiled – remember, the JIT compiles code on-demand, one method at a time, when that method is called (barring very simple methods which might be inlined into their callers). Let’s just get on and show the code for Main.

0:003> !U f3020
Normal JIT generated code
Simple.Main()
Begin 00310070, size 36
00310070 833d8c10920300  cmp     dword ptr ds:[392108Ch],0
00310077 750a            jne     00310083
00310079 b901000000      mov     ecx,1
0031007e e8e1580579      call    mscorlib_ni+0x2a5964 (79365964) (System.Console.InitializeStdOutError(Boolean), mdToken: 06000770)
00310083 8b0d8c109203    mov     ecx,dword ptr ds:[392108Ch] (Object: System.IO.TextWriter+SyncTextWriter)
00310089 8b153c309203    mov     edx,dword ptr ds:[392303Ch] ("Hello World!")
0031008f 8b01            mov     eax,dword ptr [ecx]
00310091 ff90d8000000    call    dword ptr [eax+0D8h]
00310097 e84c750d79      call    mscorlib_ni+0x3275e8 (793e75e8) (System.Console.get_In(), mdToken: 0600076e)
0031009c 8bc8            mov     ecx,eax
0031009e 8b01            mov     eax,dword ptr [ecx]
003100a0 ff5064          call    dword ptr [eax+64h]
003100a3 33c0            xor     eax,eax
003100a5 c3              ret

Interesting. The JIT clearly sees Console.WriteLine(string) and Console.ReadLine as trivial and has inlined the calls. In turn it’s inlined Console.get_Out. The reference to SyncTextWriter isn’t the JIT being clever – it’s SOS telling us that the address 0x0392108C is the base of SyncTextWriter’s vtable, as this call to TextWriter.WriteLine(string) is virtual.

I realise, and hope, that this isn’t everyday usage. More often than not you’ll have crashed somewhere and want to find out where (for which see !IP2MD). But it can be useful to see just how your C# code turns into machine instructions.

 

========================

Introduction 

WinDbg is the most powerful debugger for windows, and it is licensed with the OS, so once you paid money for the OS, no extra money has to be paid for windbg.

Article about WinDbg: http://www.debuginfo.com/articles/easywindbg.html

Technique for analysing exceptions: http://blogs.msdn.com/jmstall/archive/2005/01/18/355697.aspx

Collection of debugging techniques: http://www.dumpanalysis.org/

This is another good tutorial: http://www.codeproject.com/KB/debug/windbg_part1.aspx

Memory leak detection using windbg http://www.codeproject.com/Articles/31382/Memory-Leak-Detection-Using-Windbg

Installing Windbg

#  Download windbg and install (http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx)

#  Add the installation folder to the path (for me it is "c:\Program Files\Debugging Tools for Windows")
#  Create a new folder for symbols (I chose to use c:\symbols, this is going to be used in this document)
# Create an environment variable called _NT_SYMBOL_PATH and set _NT_SYMBOL_PATH to the following: CACHE*c:\Symbols;c:\CustomSymbols;srv*http://msdl.microsoft.com/download/symbols  (substitute your folder for c:\symbols and c:\customsymbols. CustomSymbols can be used to manually copy pdb files into - if you run !sym noisy you will be able to see how it is used.)

_NT_SYMBOL_PATH is used for getting the pdb files for windows dlls, you will see the function names in the call-stack for windows dlls if you configure this setting properly.

You can also download symbol files for  windows dlls manually using the command (issue in the command prompt):

symchk /r c:\windows\system32 /s SRV*c:\symbols\*http://msdl.microsoft.com/download/symbols

where symchk.exe is installed as part of the WinDbg software into c:\Program Files\Debugging Tools for Windows. You can also issue commands to download symbol files from within windbg, see the documentation of WinDbg for details, and this address: http://support.microsoft.com/kb/311503.
 
You can set windbg to be the post-mortem debugger by running
windbg -I 

Using built in help

.hh <search str>

General stuff

!wow64exts.sw                 - Switch between 32 and 64 bit modes (obviously only if you have a 64 bit windbg)
!verifier                              - verifier has a built in command in windbg, useful when you investigate a bsod caused by verifier
!verifier FF <drivername> - turn on full info about a driver

Debugging unmanaged (native, ordinary c++) code

When a software crashes, or asserts on a machine that does not have Visual Studio installed it is possible to locate the crash or assert by using WinDbg to debug the application, see threads running and examine the call stack. To achieve this you will need to make sure that the application was built in debug mode, you have the generated symbol (pdb) files and the source files.

 
#    Start "c:\Program Files\Debugging Tools for Windows\windbg.exe" (or from the location where it has been installed)
#    Push Ctrl+P, add the relevant source folders to the list ( select the folder of the solution)
#    Push F6, select the process
#    Push Alt+9 (threads window pops up) and Alt+6 (callstack window pops up)

If you see a message like this: *** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\system32\RPCRT4.dll, the symbol files cannot be found. If the error mentions one of your dlls,  make sure that the pdb file is in the same folder as the dll that was mentioned in the error message. If a windows dll is mentioned, the _NT_SYMBOL_PATH is not configured correctly. 

Using the watch

The shortcut to the watch window is alt+2.

You can use the watch just like in VS, but you should try to prefix your variables, otherwise the debugging process might become too long. For a local variable you should try the prefix $!. So for getting the value if a variable i, you should write $!i into the watch window. A member variable of a class can be referenced as a local variable if the debugger is positioned on a line of a non-static member function.

For a global variable you could try <modulename>!variablename. 

Investigating the call stack in managed (vb.net, C#) code

#  Map a drive to the shared src drive

# Start "c:\Program Files\Debugging Tools for Windows\windbg.exe" (or from the location where it has been installed)
# Push Ctrl+P, add the relevant source folders to the list (from the mapped drive)
# Push F6, select LCShell.exe
# Push Alt+1 for the command window
# Type: .load c:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\SOS.dll (dll location might be different, depends on .net version)
# Instead of the previous step this might also work: .loadby sos mscorwks (tells WinDbg to look up which directory mscorwks.dll was loaded from and load sos.dll from there)
# Type: !threads (this will list the threads with managed code)
# Type: ~*e !clrstack (this will list the callstack in the managed code)

Useful commands

(some from http://www.codeproject.com/KB/debug/windbg_part1.aspx)

Common windbg commands: http://windbg.info/doc/1-common-cmds.html

NATIVE + MANAGED

Start DML (more user friendly display):
.dml_start

Start a new temporary command window (with command history):

.browse

Add a new folder to the list of symbol paths:

.sympath +c:\SomeFolder\SomeSubfolder
.reload
!sym noisy # turn on detailed logging about symbols
  
loads all the necessary symbols
.reload /f
 
.reload /f mydr.sys # reload all symbols for mydr
.reload /unl /f mydr.sys # reload all symbols for mydr even though mydr has been unloaded at this stage. Useful is e.g. worker threads are not shut down during driver unload and the crash is due to unloading the module with active calls in place.
 
Which is the current address of execution
ln
 
Which is the current function of execution
ln eip

Automatic helper

!analyze -v

If the PC is hanging and you're attached to it remotely with windbg, this is useful:

!analyze -v -hang

Get a description of the handles held by the process

!handle
 

NATIVE

Processes
 
 
!dml_proc            - list all processes & PID and process address & threads and switch context to this process etc.
!peb <address>    - display process environment  block
.attach PID          - attach to a process
.detach                - ends the debugging session
.process <address>  - you can attach to process and dump callstacks etc. works only in kdb mode (not in user mode)
!process <address> - displays info (threads etc.) about the process specified by address
!process 0 7         - dumps all callstacks in all processes (takes ages), better to attach to a process: !process <id> where id can be retrieved from !dml_proc. Then you can run !process -1 7, it will only dump the process you're attached to.
!findstack xxx       - finds all callstacks that contain xxx in the symbols. !Uext.help  for more info. Works only in user mode (not in kdb mode)
~*kb                    - display all callstacks in all threads in the attached process
 
This is interesting about attaching to processes: http://analyze-v.com/?p=336
 
Threads
!teb                     - dump thread environment  block
!peb                     - dump process environment  block
~                         - list threads (only in user mode)
 

!stacks 2 bla        - list all callstacks that contain bla (kernel mode only?)

.thread <address>         - switch to the context of this thread
!runaway -           - for performance, in usermode (or usermode dump) sorts threads according to how busy they are. Not tested yet.
 
Stack trace (Displays stack trace of current thread (x frames))
K x
KB x # (also shows the first 3 words of params)
KV x # (also shows the calling convention) kP n # (the most useful, prints variable types and values as well) ~*kL # prints stacktrace from all treads kf # Automatically do the ChildEBP - Previous ChildEBP calculation to see which function call is eating up all the stack .ecxr to switch the current context to an exceptions callstack
When the callstack seems wrong this can be useful: http://www.dumpanalysis.org/blog/index.php/2007/04/03/crash-dump-analysis-patterns-part-11/ Good explanation about what is the stack & how it works: http://www.codeproject.com/KB/debug/cdbntsd2.aspx !stack - to display stacks, see example below in hang analysis
!stacks 2 <drivername> - will list stacks that contain code from that driver
If you have a code address and have proper symbols, source file location all that, you can open the source file that has the code at that code address by .open -a <address>
To see the assembly of a function implementation, you can do .browse uf <func> e.g. .browse uf ExFreePoolWithTag
 
When analyzing a hang, use these
 
!stacks
You can do !stacks 2 <drivername> and it will list stacks that contain code from that driver
!locks
          To see the callstack for who is holding a lock: 
!locks <drivername>!<global lock variable name>
 
Will say
 
Resource @ <drivername>!<global lock variable name> (0xfffff801948d2d80)    Exclusively owned
     Threads: ffffe001b38b2840-01<*>
1 total locks, 1 locks currently held
 
So you can do
 
!thread ffffe001b38b2840
 
Which will print the stack.
 
!analyze -hang
To see threads and all that
 
Variables
 
With some commands here if you do not specify an address, they will move forward. So if you do db <address>, you will see the byte at the specified memory at address. Further db commands w/o an address will display bytes following the previous byte and the "window will move forward".
dv
dv /t - prints all variables on the currently selected stack-frame
dv - without arguments prints all locals
dv /v - locals with their addresses
 
dt <variablename> -  print, -r will print structures recursively
in case the variablename does not work (e.g. for a structure member of a structure) then you can do dt <TypeName> <Address>
dd <module>!<variablename> # dump  data - this one will print the value of the variable
db <address> - dump the contents of memory at address (dumps a byte), repeatedly issue db and it will print the next bytes
du <address> - display unicode string at address. Usually dt returns the address of a string buffer if the variable is a wchar_t* or char*.
!for_each_frame dv /t - prints all variables on all stack-frames in current thread
ed < module>!<variablename>  0x02000FFF # ed means edit data so this one will overwrite the value of  the variable in memory
 
!stl <variablename> - displays std::wstring (possibly also std::string) contents
 
Memory
!memusage # Overview of OS memory info
!vm # virtual memory usage details !poolused # displays pool usage summaries, report includes the tag  used for each pool allocation. See help for sorting capabilities. !pool <address> # Whether a particular block of memory is allocated correctly or is just some random bit of memory. !poolfind # finds all instances of a specific pool tag in either nonpaged or paged memory pools. Takes ages! !heap -s # will print heap overview info about allocations. Then the address of an entry can be fed to the !heap command again to get further details. !vadump # dump list of memory pages and info !handle 0 0 0 # displays all handles with owning process r # show registers. Can see function return value after call in eax. Last item (in time) placed onto the stack is in esi. See this for more: http://unixwiz.net/techtips/win32-callconv-asm.html s # can be used to search memory. s -u 0 L?4000000000 blabla # In a user mode (one process) dump it searches 4GB from start for the word blable as unicode string in memory
!search # search memory in context only.
dc <address> # will print the memory at that address dds, dps, and dqs # To display the contents of memory in the given range, treat the numbers as code addresses and display the corresponding symbols.
 
Breakpoints
bl # list
bp,  bc, be, bd # (put, clear, enable, disable)
bc * # clears all breakpoints
bs 4 "dt varname;g;" # with bl you can list breakpoints. You can get any of them to run stuff by executing bs against its number. In this example we want to run "dt varname;g;" whenever breakpoint 4 is hit. bl will show the execute info as well.
  ----
#If you do not have sources but you do have pdb files, you can still set breakpoints to functions. Here is how it goes.
#Use the X command to search for symbols. E.g.
X mymodule!functionname
#You can use asterisks (*) for partial match. For all matches you will see addresses of functions. You can use those addresses with the bp function.
----

Modules and symbols

 
lm                                            # show loaded modules
lm v m <modulename>*             # show info about a module
!lmi  modulename                       # dump the info for module modulename               
X < module>!*mynameportion*    # search for every symbol (variable name) that matches *mynameportion*
x <module>!*                            # list all symbols exported by <module>
.symopt +0x40                         # this will make windbg load any pdb file that it can find for a dll, based on name matching. If you do not have the proper pdb can be useful as a last resort.
.reload /f <dll/exe name>           # this forces a symbol (re)load for the module
!symfix                                     # probably this will fix windows symbols?
!sym noisy                               # display detailed info about loading (or failing to load) symbols
List of symbol options: https://msdn.microsoft.com/en-us/library/windows/hardware/ff558827%28v=vs.85%29.aspx#symopt_load_lines

Others

b # if a debug assertion is shown, break into the code
.reboot # reboots the pc windbg is currently connected to (useful if we connect to the PC remotely)
.dump # creates a dump file from what's in windbg at the moment
 
# Combining windbg with unixtools like grep (https://sourceforge.net/projects/unxutils/) (or any other shell commands but grep is the one that I miss a lot from windbg)
.shell -ci "!threads" grep 236 # this will grep 236 from the text output printed by !threads
Scripting, .foreach https://msdn.microsoft.com/en-us/library/windows/hardware/ff552157%28v=vs.85%29.aspx
# Here is to list threads with !threads, get the address of the teb for each (3rd column in printout) then call !teb on each address and filter out the TEB address and the LastErrorValue for each. .foreach (obj {.shell -ci "!threads" egrep -v Total | gawk "{print $3}" }) {.shell -ci "!teb obj" egrep "LastErrorValue|TEB at"}
# Will print sg. like
TEB at 0000000000270000     LastErrorValue:       0 .shell: Process exited TEB at 00000000002d6000     LastErrorValue:       997 .shell: Process exited TEB at 00000000002da000     LastErrorValue:       0 .shell: Process exited TEB at 00000000002e2000     LastErrorValue:       0 .shell: Process exited TEB at 00000000002e4000     LastErrorValue:       0 .shell: Process exited
# This approach is not perfect as there is some garbage returned at the end of the first command execution and we run !teb against the garbage which times out slowly. To improve that we'll try an if statement. # .if ($spat("${obj}","0x[0-9]+*")) - spat is a pattern matching thing. The pattern is a weak regular-expression-like syntax (https://msdn.microsoft.com/en-us/library/windows/hardware/ff558819%28v=vs.85%29.aspx).
# What we're saying here is that the string has to start with 0x and then it should have one or more digits and anything after that (we have extra spaces...) and then the .if condition is true.
# so this command works well (we can print what is passed in object with .echo):
.foreach (obj {.shell -ci "!threads" egrep -v TEB | gawk "{print $3}" }) { .if ($spat("${obj}","0x[0-9]+*")) { .echo "${obj}" ; .shell -ci "!teb obj" egrep "LastErrorValue" } }
 
# here is another similar example: .foreach (obj {.shell -ci "!heap -flt s 178" egrep -v HEAP | egrep -v ole32 | gawk "{print $5}" }) {!heap -p -a obj}
# (needs gflags.exe /i MemoryLeak.exe +ust executed from the command prompt for your exe for it to make any sense, see http://www.codeproject.com/Articles/31382/Memory-Leak-Detection-Using-Windbg )
If you have executed gflags and run your process etc. here is how to dump all the callstacks for all memory reservations

# change filename as needed… .foreach (obj2 { .foreach (obj {.shell -ci "!heap -s" egrep "^0.*" | gawk "{print $1}"}) { .if ($spat("${obj}","[0-9]+*")) { .shell -ci "!heap -stat -h obj" grep .*-.*(.*) | gawk "{print $1}" } } } ) { .if ($spat("${obj2}","[0-9]+*")) { .shell -ci "!heap -flt s obj2" grep (busy) >> d:\lmhost (2).1.txt } }

# from cmd prompt run this: cat "lmhost (2).1.txt" | gawk "{print $5}" | sort | uniq > "lmhost (2).2.txt"

# back to windbg, again change filenames as needed .foreach /f ( obj "d:\ lmhost (2).2.txt" ) { .shell -ci "!heap -p -a obj" grep .* >> "d:\lmhost (2).3.txt" }
#This last step can run for days if not weeks, be prepared...

Usual way to investigate calls in a process

!dml_proc  - to see all processes, pick the one you're interested in

                  then click "select user mode state" (this will load pdb files and set up the process context)
                  then click Full Details (this will list all threads with thread addresses and ids and callstacks)
.thread <thread address> - this will switch the context to this thread. Now you can use all the callstack commands.
                                       Also you can use alt+2 and alt+6 to walk up & down on the call stack and see the values
                                       of variables. At this stage if you have the source file open the current line of execution
                                       will be shown as you click in the calls window.
 

MANAGED

 
You need to load the sos.dll for these to work (see above)
 
List the threads that have managed code:
!threads
 
Select current thread:
~[n]s, e.g. ~11s
 
Get the callstack in the active thread of a managed app:
!clrstack
 
The same for all threads:
~*e !clrstack
 
Print exception in current thread
!pe
 
Print exceptions in all threads
~*e !pe
 
Get the whole stack of a managed app (current thread):
!dumpstack
 

Dump all the managed objects that are currently on the stack:

!dumpstackobjects
 

Dump all the managed objects that are currently in the heap:

!dumpheap -stat

Getting the value of a member variable

find the address of the container object using !dumpstackobjects, e.g.

0:008> !dumpstackobjects

Thread 8
ESP/REG    Object     Name
edx        0x44a7d98 System.Single[]
0x6bafaec 0x444c4c0 System.Threading.Thread
0x6bafaf4 0x444c4c0 System.Threading.Thread
0x6bafafc 0x444c46c Project1.ThreadManager
0x6bafb00 0x44a7d80 Project1.USBHardware/DownloadPointsToUsbParams
0x6bafd6c 0x444c4a4 System.Threading.ThreadStart

and then type

!dumpobj 0x444c46c

(for the ThreadManager object)

 

print the help for the sos extension

!clr10\sos.help  
 

Setting a managed breakpoint to a function call:

!bpmd <modulename> <functionname> 

Memory leaks

C++: http://www.codeproject.com/Articles/31382/Memory-Leak-Detection-Using-Windbg

Here is a similar approach:

 

gflags.exe /i <leaking exe> +ust gflags.exe /i <leaking exe> /tracedb 50

 

(re)start your exe, let it run and leak and then create a dump (i.e. through task manager), open the dump in windbg, load all symbols and then

!heap -l

will print the callstack for memory addresses that are potentially results of a leak.

C#: http://blogs.msdn.com/b/ricom/archive/2004/12/10/279612.aspx

Creating and debugging a memory dump

 
The usual method of debugging a problem that occurs on a machine in production is that in case an error happens a dump file is generated and this dump file is debugged on the developer's desktop, where the code and the debugging environment is installed. This technology is built up on exceptions.
 
In general, windows shows two types of error dialogs.
 
The first type error dialog says:
 
Runtime Error! This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
 
This dialog just has an OK button.
 
The second type error dialog can be configured in Control Panel -> System -> Advanced -> Error reporting, and it lets the user select to debug the application.
 
These dialogs appear if an exception is not handled in the application, in other words in the case of a second chance exception. (A first chance exception is one that the application can/should catch; if the application does not catch, it becomes a second chance exception that the operating system catches, and in response to that the OS shows the above described dialogs.)
 
Most automatic memory dump generators will start when the debug button is pushed on the second type error dialog, it means that they do not do anything for the first type error dialog. Our aim now is to create a dump file for any second chance exception that happens no matter which dialog is shown.
 
Also keep in mind that these tools can produce a dump file in case a first chance exception happens, and they can be configured for which first case exceptions to produce the memory dump, but that is not our aim now in general.
  
Creating a memory dump
 
The most typical ways to achieve this are the followings:
 
  1. Configure Dr. Watson or WER so that if a crash happens, and the user selects to debug the problem, instead of the debugger Dr. Watson starts and it creates a dump file. Dr. Watson is part of the XP pro install. http://support.microsoft.com/kb/308538 and WER replaces it in Vista.
     
  2. (Use NTSD.exe. http://support.citrix.com/article/CTX105888)
  3. Install the "Debugging tools for Windows" package and use ADPlus.vbs. http://support.microsoft.com/kb/286350
  4. Install the "Debugging tools for Windows" package and use Userdump.exe. http://support.microsoft.com/kb/241215
  5. Start WinDbg, attach to the process and execute this command: .dump
  6. Use Sysinternals Procdump utility
     

I have tested #3 and #4 with two scenarios: I have created a small Application that generated a division by zero exception and could also throw a user defined exception. I have also tested what throwing and catching an exception would cause in any of the above tools (that is part of the normal operation in our software). The result of this investigation is that if the application does not just die, but displays a dialog before, it is best to start ADPlus.vbs when the dialog is displayed and generate a snapshot of the process memory (for details see the comments just at the end of this subchapter). This is explained step by step in "Using windbg to create a crash dump" below. Nr. 1 is described in "Using built-in windows tools to generate crash reports".

 

* Using windbg to create a crash dump

 
For setting up the environment:
 
  1. Install the "Debugging tools for Windows" package
  2. Create a batch file with the following content:
 
        pushd "c:\Program Files\WinDbg"
        cscript ADPlus.vbs -hang -NoDumpOnFirst -pn CrashDump.exe -o c:\temp\dump -quiet
 
Of course c:\Program Files\WinDbg should be changed to the folder where you installed the "Debugging tools for Windows" package, and CrashDump.exe should be the process you are investigating. When any error dialog appears, execute the batch file. You will not be able to access the application until the error reporting process finishes.
 
If the process dies without any error dialogs, there is a way to attach ADPlus.vbs from the start of the execution, this however slows down the process a noticeable amount. This is only useful if we want to investigate silent finish of the process, then we do not have any other choice (we could also use Userdump.exe for this purpose, but it has the same disadvantages).
 
Comments:
 
Userdump.exe produces a report on all (configured) first chance exceptions that is not useful for us and slows down the software a lot. With this tool there is no way to catch a second chance exception but not catch a first chance exception of the same type.
 
For debugging a silent crash ADPlus.vbs would be executed just after the application starts, and it would be called like this: ADPlus.vbs -crash -NoDumpOnFirst -pn CrashDump.exe -o c:\temp\dump -quiet
 
There is a good description of ADPlus.vbs in debugger.chm (in the same folder where ADPlus.vbs is)

* The firefox way

https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg

 

* Using built-in windows tools to generate crash reports

 
Dr Watson on XP
 
http://msdn.microsoft.com/en-us/library/cc266343.aspx
 
To specify that you want dr watson to be the post-mortem debugger (auto invoke it if something crashes) execute this from console:
drwtsn32.exe -I
 
Once you set it up, if a crash happens, you will get the well known dialog which says:
 
yourappprocess.exe has encountered a problem and needs to close....
 
Here you should select "Don't Send."
 
Then open a console and type drwtsn32.exe, this will open up a dialog with your latest crashes. Here you can see an editbox on the top, it says "Crash Dump" and it specifies the path to a file, this is the file that developers will need to see what happened. Just attach the user.dmp file somehow to the bug you're reporting (it is big so probably you'll need to share it from your computer or something?).
 
Windows Error Reporting on Vista
 
http://msdn.microsoft.com/en-us/library/bb513638%28VS.85%29.aspx
 
 
Add/modify this registry entry:
 
under
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting
add
ForceQueue=1 (this value shouls be of type REG_DWORD)
 
Once you put this change to your registry in case a crash happens the process will die silently. In this case go to Control Panel->Problem Reports and Solutions, select "View problem history" and double-click the crash that you want to report to the developers. Here you should select "View a temporary copy of these files" under the section "Files that help describe the problem". Attach minidump.mdmp to your bug report (and any other files that you think might be helpful).
 

Creating a system memory dump (for drivers)

Forcing a System Crash from the Keyboard

 

Configure Windows to save full dump on BSOD

page file settings

 
Go to computer properties (Advanced system properties on Vista+) (WindowsKey + Break)
Advanced Tab
Settings Button in Performance section
Advanced Tab
Change button in Virtual memory section
Paging file should be either system managed or have the maximum size specified by the page file size guidelines above.  For Vista+, you can just check the 'Automatically manage page file size for all drives' checkbox
 
Log settings
 
Go to computer properties (Advanced system properties on Vista+) (WindowsKey + Break)
Advanced Tab
Settings Button in Startup and Recovery section
System failure section
Check - Write an event to the system log
Check - Send an administrative alert
Automatically restart (tester's discretion)
Set the 'Write debugging information' drop down to 'Complete memory dump'
Set the Dump file to: %SystemDrive%\MEMORY.DMP
Check - Overwrite any existing file
 

Configure Windows so that we can trigger a system crash via the keyboard

Install the appropriate hotfix. E.g. look at these for the corresponding OSs:

2003 SP1, XP x64 SP1: kb244139
Vista SP1, 2008 SP1:  kb971284
 
Create these registry keys
 
In the registry key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\i8042prt\Parameters, create a value named CrashOnCtrlScroll, and set it equal to a REG_DWORD value of 0x01.
In the registry key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\kbdhid\Parameters, create a value named CrashOnCtrlScroll, and set it equal to a REG_DWORD value of 0x01.
 
Reboot
 

Real-time checks

 

A common problem with C type code is that you can very easily overindex, meaning you can write into memory that you really shouldn't, and the code won't complain (unlike in c++ or C# or java where if you overindex a vector or an arraylist you'll get a well define exception). This problem might not cause a crash when it happens, until another area of the code tries to read and use the memory. Even worse, the code might not crash even in that case, maybe some data has just changed mysteriously, but the process will survive. So now you have a strange number in one of your structures or your application crashes for no apparent reason, how do you find the offending code? There are tools and techniques that can be used to detect such problems but they slow down the processes, so these are not enabled by default. However it can be extremely useful for error analysis or dev testing; when they are turned on, the process or code will crash/bluescreen when it performs the overindexing, so the dump file that you get will contain the offending code and not the victim. I have heard of 2 such tools that can achieve this, they are verifier (for drivers) and gflags. gflags is part of windbg, verifier is installed on all recent windows versions by default.

 

Debugging a memory dump

 
If you want to debug a memory dump, these are the steps:
 
  1. Start WinDbg
  2. Push ctrl+D, open the dump file
  3. Push alt+6 for callstack, and select the more button for seeing the whole list (required in most cases)
  4. You can also jump around threads, as if you were debugging, but of course you cannot go forward or backward in time - this is just a snapshot

Here is a good series of blogs about how complex crash-dump analysis can become: http://www.dumpanalysis.org/blog/index.php/2006/10/30/crash-dump-analysis-patterns-part-1/

 
Further readings
  

 

 

Apparently there is an extension that makes it possible to find out extra info about COM calls. Need to dig into this later. Here is the name of the dll: sieextpub.dll

 
 

posted on 2016-08-04 00:40  暖风的风  阅读(892)  评论(0编辑  收藏  举报

导航