[Tip: How to use memory dump to trace bug]
[From Code Project: http://www.codeproject.com/KB/debug/moomoo.aspx]
Introduction
Our software just published version 1.0 . My clients often encountered exceptions and blue screens that caused the program to terminate. Most of time, we can not reproduce the bug in our systems. So catching and analyzing a memory dump file is important. This article teaches you how to use hands-on tools to catch and read memory dump files.
The hands-on tools are :-
- Dr. Watson, which is included by almost all the windows systems.
- VC6 and VC7 compiler, we need to recompile the project to generate Map files and COM files for tracing the bug.
- Dumpchk can check dump file, before you are going to send it.
- Windbg and kd can give you detailed information about the spot when exception occurred. Symbol files are always necessary for debugging dump files.
Details
Before reproducing the bug, Dr. Watson must be installed and running for srwtsn32.log to be created. We also need to configure the project to generate debug information, MAPFILE and COD file, then recompile the project.
Configuring Dr. Watson
To start Dr. Watson and set it as default debugger, go to command line, type:
drwtsn32 �I
If you want to change the settings you can run drwtsn32 at command line any time. Default settings is ok. Then if a program crashes, Dr. Watson will write drwtsn32.log and dump memory into user.dmp.
Reconfiguring Project Settings in Visual C++ environment
You can prepare the source code to be ready for Dr. Watson and memory dump. You need to take three simple step as following:
- You need to Generate debug symbols for the Win32 Release version of your .EXE and/ or .DLL files in settings configuration for the project, for the linker, Open "Project Settings" window( click Settings item in Project menu), select Link tab, check Generate Mapfile, and check Generate debug info. For the compiler, check Debug Info: Program Database, under General in C/C++ tab.
- Creating and keeping a copy of your application's .MAP and .COD intermediate files for further reference. Under Link tab, in Project Options editbox, add switch "/mapinfo:exports" ( switch /mapinfo:exports causes the linker to describe the binary's exported functions.) Add switch "/mapinfo:lines" (The switch /mapinfo:lines will add line numbers in mapfile). A .COD file contains a mixture of source code, assembly code and memory addresses, and it will help you to locate the offending line of code within a function that has been identified via the .MAP file. Under General section in C/C++ tab, add switch "/FAcs" in project options.
- Specifying a fixed load base hexadecimal address for your .EXE and/or .DLL files. Under Output in the Link tab, input a hexadecimal address in Base address, such as 0x602f0000
Examples
Use the Visual C++ application wizard to create MFC win32 exe file names "MapDemo". Add following lines in InitInstance function in CMapDemoApp class.
char * pch1 = 0; char * pch2 = 0; pch1 = (char *)malloc( 20 ); pch2 = (char *)malloc( 20 ); pch1 = pch2; free( pch1 ); free( pch2 );
This code will eventually fail because the application will try to release the same block of memory twice. Run the program and it will crash. Open the file drwtsn32.log:
*----> Module List <----*(0000000000400000 - 000000000040b000: C:\WORK\CODEPROJ\MapDemo\Debug\MapDemo.exe(0000000010200000 - 000000001026c000: C:\WINDOWS\System32\MSVCRTD.dll(000000005f400000 - 000000005f4e5000: C:\WINDOWS\System32\MFC42D.DLL(00000000605d0000 - 00000000605d8000: C:\WINDOWS\System32\mslbui.dll(0000000070a70000 - 0000000070ad4000: C:\WINDOWS\system32\SHLWAPI.dll(0000000074720000 - 0000000074764000: C:\WINDOWS\System32\MSCTF.dll(0000000077120000 - 00000000771ab000: C:\WINDOWS\system32\OLEAUT32.DLL(00000000771b0000 - 00000000772d1000: C:\WINDOWS\system32\OLE32.DLL(0000000077c00000 - 0000000077c07000: C:\WINDOWS\system32\VERSION.dll(0000000077c10000 - 0000000077c63000: C:\WINDOWS\system32\msvcrt.dll(0000000077c70000 - 0000000077cb0000: C:\WINDOWS\system32\GDI32.dll(0000000077d40000 - 0000000077dc6000: C:\WINDOWS\system32\USER32.dll(0000000077dd0000 - 0000000077e5d000: C:\WINDOWS\system32\ADVAPI32.dll(0000000077e60000 - 0000000077f46000: C:\WINDOWS\system32\kernel32.dll(0000000077f50000 - 0000000077ff7000: C:\WINDOWS\System32\ntdll.dll(0000000078000000 - 000000007807f000: C:\WINDOWS\system32\RPCRT4.dll
*----> State Dump for Thread Id 0x3274 <----* eax=00000001 ebx=7ffdf000 ecx=10261558 edx=00010000 esi=0012febc edi=7ffdf000eip=10213638 esp=0012fe5c ebp=0012fe6c iopl=0 nv up ei pl zr na po nccs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
function: MSVCRTD!_free_dbg_lk 10213616 7521 jnz MSVCRTD!_free_dbg_lk+0xc9 (10213639) 10213618 6814732510 push 0x10257314 1021361d 6a00 push 0x0 1021361f 6814040000 push 0x414 10213624 68f4712510 push 0x102571f4 10213629 6a02 push 0x2 1021362b e810180000 call MSVCRTD!_CrtDbgReport (10214e40) 10213630 83c414 add esp,0x14 10213633 83f801 cmp eax,0x1 10213636 7501 jnz MSVCRTD!_free_dbg_lk+0xc9 (10213639)FAULT ->10213638 cc int 3 10213639 33c0 xor eax,eax 1021363b 85c0 test eax,eax 1021363d 75c9 jnz MSVCRTD!_free_dbg_lk+0x98 (10213608) 1021363f 8b4d08 mov ecx,[ebp+0x8] 10213642 83e920 sub ecx,0x20 10213645 894dfc mov [ebp-0x4],ecx 10213648 8b55fc mov edx,[ebp-0x4] 1021364b 8b4214 mov eax,[edx+0x14] 1021364e 25ffff0000 and eax,0xffff 10213653 83f804 cmp eax,0x4
*----> Stack Back Trace <----**** WARNING: Unable to verify checksum for C:\WORK\CODEPROJ\MapDemo\Debug\MapDemo.exe*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\kernel32.dll - WARNING: Stack unwind information not available. Following frames may be wrong.ChildEBP RetAddr Args to Child 0012fe6c 10213541 00324708 00000001 7ffdf000 MSVCRTD!_free_dbg_lk+0xc80012fea0 102134ce 00324708 00000001 0012fee8 MSVCRTD!_free_dbg+0x410012feb0 00401360 00324708 7ffde000 00324238 MSVCRTD!free+0xe0012fee8 5f4335c3 7ffdf000 7ffde000 7ffdf000 MapDemo!CMapDemoApp__InitInstance+0x1320012ff08 00402248 00400000 00000000 001423ac MFC42D!AfxWinMain+0x830012ff20 00402153 00400000 00000000 001423ac MapDemo!WinMain+0x180012ffc0 77e814c7 7ffdf000 7ffde000 7ffdf000 MapDemo!WinMainCRTStartup+0x1b30012fff0 00000000 00401fa0 00000000 78746341 kernel32!GetCurrentDirectoryW+0x44
Because the code was fault in the memory free function "_free_dbg_lk", it is called by function Initinstance(), so we search the stack-track-back part, on the fourth line, it indicates that the sub function was called by InitInstance function at the offset 0x132.
So, look at the .COD intermediate file, "MapDemo.COD", search for the module "InitInstance()", the offset address at the beginning of the module is 0x00ae, add the address of (0x132+0xae = 0x01e0 ) , so we know the program was crashed at line 79 in source code file "MapDemo.cpp".
Another Example - How to trace fault into DLL file( in � process, and out � process DLL file).
Use Application Wizard in Visual Studio to generate a DLL library project MapDemoDll, create a export function DllSquareRoot(), and put a fault code in that function :-
char* pChar = NULL; *pChar = 'a'; // This is fault line 70
The program will throw an exception that trying to access to a invalid memory address(0x0000). Use Application Wizard to generate an client program calling the DllSquareRoot function in MapDemoDll.dll. Before running the exe program with the DLL library, we need to set the MapDemoDll�s project�s settings. Generate MAP, COD file, specify the base address for loading at 0x60f20000. Compile the project and run the program.
Program crashed and screen displayed the message like that "DemoDllClient.exe has generated errors and will be closed by windows. You will need to restart the program." Following is the drwsnt32.log generated by Dr. Watson.
State Dump for Thread Id 0x7b0 eax=00000000 ebx=005e4b68 ecx=00135dd0 edx=60f35918 esi=0012f27c edi=0012f26ceip=60f21320 esp=0012f204 ebp=0012f26c iopl=0 nv up ei pl zr na po nccs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
function: DllSquareRoot 60f212f9 8d7da4 lea edi,[ebp+0xa4] ss:00c8c83e=???????? 60f212fc b917000000 mov ecx,0x17 60f21301 b8cccccccc mov eax,0xcccccccc 60f21306 f3ab rep stosd es:0012f26c=0012f2cc 60f21308 e8f3020000 call AfxGetStaticModuleState (60f21600) 60f2130d 50 push eax 60f2130e 8d4df8 lea ecx,[ebp+0xf8] ss:00c8c83e=???????? 60f21311 e8a4010000 call AFX_MAINTAIN_STATE2::AFX_MAINTAIN_STATE2 (60f214ba) 60f21316 c745f400000000 mov dword ptr [ebp+0xf4],0x0 ss:00c8c83e=???????? 60f2131d 8b45f4 mov eax,[ebp+0xf4] ss:00c8c83e=????????FAULT ->60f21320 c60078 mov byte ptr [eax],0x78 ds:00000000=?? 60f21323 dd4508 fld qword ptr [ebp+0x8] ss:00c8c83e=???????????????? 60f21326 dc1d4841f360 ds:60f34148=0000000000000000 fcomp qword ptr [_real (60f34148)] 60f2132c dfe0 fstsw 60f2132e f6c401 test ah,0x1 60f21331 7520 jnz _enc$textbss$begin+0x5f43 (60f29e53) 60f21333 8b4d0c mov ecx,[ebp+0xc] ss:00c8c83e=???????? 60f21336 51 push ecx 60f21337 8b5508 mov edx,[ebp+0x8] ss:00c8c83e=???????? 60f2133a 52 push edx 60f2133b e848060000 call sqrt (60f21988) 60f21340 83c408 add esp,0x8
The program crashed at address 0x60f21230, here is part of the mapfile
Address Publics by Value Rva+Base Lib:Object 0001:00000050 ?_GetBaseMessageMap@CMapDemoDllApp@@KGPBUAFX_MSGMAP@@XZ 60f21050 f MapDemoDll.obj 0001:00000080 ?GetMessageMap@CMapDemoDllApp@@MBEPBUAFX_MSGMAP@@XZ 60f21080 f MapDemoDll.obj 0001:000000c0 ??0CMapDemoDllApp@@QAE@XZ 60f210c0 f MapDemoDll.obj 0001:00000120 ??_GCMapDemoDllApp@@UAEPAXI@Z 60f21120 f i MapDemoDll.obj 0001:00000120 ??_ECMapDemoDllApp@@UAEPAXI@Z 60f21120 f i MapDemoDll.obj 0001:00000190 ??1CMapDemoDllApp@@UAE@XZ 60f21190 f i MapDemoDll.obj 0001:000002f0 _DllSquareRoot 60f212f0 f MapDemoDll.obj 0001:000003b2 ?WinHelpA@CWinApp@@UAEXKI@Z 60f213b2 f mfc42d:MFC42D.DLL 0001:000003b8 ?OnDDECommand@CWinApp@@UAEHPAD@Z 60f213b8 f mfc42d:MFC42D.DLL 0001:000003be ?DoWaitCursor@CWinApp@@UAEXH@Z 60f213be f mfc42d:MFC42D.DLL 0001:000003c4 ?DoMessageBox@CWinApp@@UAEHPBDII@Z 60f213c4 f mfc42d:MFC42D.DLL 0001:000003ca ?SaveAllModified@CWinApp@@UAEHXZ 60f213ca f mfc42d:MFC42D.DLL 0001:000003d0 ?InitApplication@CWinApp@@UAEHXZ 60f213d0 f mfc42d:MFC42D.DLL
The biggest function address that just less than the crashing address 0x60f21320 is 0x60f212f0, which belongs to DllSquareRoot function in MapDemoDll object. We also can follow that calculation: crash_address - preferred_load_address - 0x1000. Why do we need to subtract 0x1000, because the first part the binary is the Portable Executable (PE), which is 0x1000 bytes long. So the calculation is: 0x60f21320 � 0x60f20000 � 0x1000 = 0x320
Line numbers for .\Debug\MapDemoDll.obj (D:\work\MapDemoDll\MapDemoDll.cpp) segment .text 44 0001:00000050 44 0001:00000080 55 0001:000000c0 58 0001:000000f0 63 0001:00000220 66 0001:000002f0 67 0001:00000308 69 0001:00000316 70 0001:0000031d 72 0001:00000323 73 0001:00000333 74 0001:00000353 75 0001:00000361 76 0001:0000037a
Above is the information section of MAPFILE, like that: 44 0001:00000050, first number is line number, the second number is the offset from the beginning of the code section in which this line occurred. So we will know the program was crashed at line 70 in MapDemoDll.cpp. Which tried to write a number into an invalid address.
How to read and debug by memory dump file
- Get the symbol table or symbol server address. To prepare for debugging the dump file, you should get the Microsoft symbol server address or download the symbols from Microsoft (http://www.microsoft.com/whdc/ddk/debugging/symbolpkg.mspx#Windows%20symbol%20packages)
- Download the debugging tools for Windows. Those debugging tools can read the windows memory dump and simulate the original memory environment.
- Need to install the support tools from original installation CD. Before using memory dump to debug, we need to check if it is not corrupted by dumpchk.exe, which is in the windows installation CD. Go to command line and input: Dumpchk -a [dump file]
- If the memory dump file passes the check, now we can use windbg or kd to debug the memory dump. After the installation is complete, start a command prompt, change to the path where the debugging tools were installed. Use the one of following command to load the dump file into debugger. Windbg.exe is a windows-based debugger, kd.exe is a command line debugger.
windbg -y SymbolPath -i ImagePath -z DumpFilePathkd -y SymbolPath -i ImagePath -z DumpFilePath
The SymbolPath is the path of symbols, it is either a local path where the symbol files were downloaded, or the server path which is an URL. ImagePath is the path of windows image files, which are contained in the /I386 folder on the WINDOWS. Installation CD-ROM. DumpPath is the path and file name for the dump that you are examining.
Example
kd -y SRV*f:\symbols*http://msdl.microsoft.com/download/symbols -i e:\i386 -z f:\dumps\minidump.dmpkd -y f:\symbols -i e:\i386 -z "f:\my debug files\minidump.dmp"windbg -y f:\symbols -i e:\i386 -z "f:\dumps\minidump.dmp"
Both of the debuggers accept commands which gather information from dump file, such as:
The !analyze -show command displays the Stop error code (also known as the bug check code) and its parameters. The !analyze -v command displays verbose output. The !drivers command displays a list of the drivers that were loaded on the computer when the problem occurred.
