调试ASP.NET应用程序Security Token Handle Leak

问题现象

ASP.NET应用程序security token相关的句柄泄露一般都与impersonate机制相关。通常发生了这种问题会导致整个操作系统性能减慢，在系统日志中有可能记录2020错误。

Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Date: 12/24/2008
Time: 12:13:31 AM
User: N/A
Computer: MYW2K3
Description:
The server was unable to allocate from the system paged pool because the pool was empty.

如果在这种情况下抓系统kernel dump,查看dump中paged pool，可以发现大部分都被token占用。从进程管理器我们也可以看到w3wp.exe和lsass.exe的handle数量远大于其他进程，并且持续增长。

0: kd> !token ea698028
_TOKEN ea698028
TS Session ID: 0
User: S-1-5-21-606747145-527237240-1605580848-740255

如何在不同版本的操作系统中抓取kernel dump,可以参考如下文章。

How To Enable Windows XP to Capture a Complete or Kernel Memory Dump
How to generate a kernel dump file or a complete memory dump file in Windows Server 2003
How to generate a kernel or a complete memory dump file in Windows Server 2008 and Windows Server 2008 R2

另外关于paged/nonpaged pool, 可以参考Mark Russinovich的博客文章

Pushing the Limits of Windows: Paged and Nonpaged Pool

问题调试

在没有源代码的情况下如何找到什么地方调用了impersonate相关的方法呢？我们可以尝试以下两种方式。

1. 在应用程序dump文件中直接搜索该方法调用。 要实现impersonate就必须要调用相关的Windows APIs：LogonUser，DuplicateToken，我们可以在内存中搜索哪些assembly引用了这些API,然后通过ILSpy之类的IL反汇编工具来查看相应的代码。

0:018> s-a 0 L?7fffffff "LogonUserA" 
0e752a90  4c 6f 67 6f 6e 55 73 65-72 41 00 44 75 70 6c 69  LogonUserA.Dupli 
0e762a90  4c 6f 67 6f 6e 55 73 65-72 41 00 44 75 70 6c 69  LogonUserA.Dupli 
77f747e4  4c 6f 67 6f 6e 55 73 65-72 41 00 4c 6f 67 6f 6e  LogonUserA.Logon 

0:018> !address 0e752a90  
0e750000 : 0e752000 - 00002000 
                    Type     01000000 MEM_IMAGE 
                    Protect  00000020 PAGE_EXECUTE_READ 
                    State    00001000 MEM_COMMIT 
                    Usage    RegionUsageImage 
FullPath C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\securityhandleleak\91744b5d\ab584e94\App_Web_oeootcha.dll

如果该命令返回记录比较多，我们可以通过以下命令来查看

0:000> .foreach (place { s-[1]a 0 L?7fffffff "LogonUser" }) {!address place} 
    03530000 : 03532000 - 0000e000 
                    Type     01000000 MEM_IMAGE 
                    Protect  00000020 PAGE_EXECUTE_READ 
                    State    00001000 MEM_COMMIT 
                    Usage    RegionUsageImage 
                    FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\83aa2552\f8a67fd5_af8fc901\Assembly1.DLL
 
    03590000 : 03592000 - 0000e000 
                    Type     01000000 MEM_IMAGE 
                    Protect  00000020 PAGE_EXECUTE_READ 
                    State    00001000 MEM_COMMIT 
                    Usage    RegionUsageIsVAD 
    035b0000 : 035b2000 - 00011000 
                    Type     01000000 MEM_IMAGE 
                    Protect  00000020 PAGE_EXECUTE_READ 
                    State    00001000 MEM_COMMIT 
                    Usage    RegionUsageImage 
                    FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\73e6c81e\fcbd4e76_6f96c901\Approve.DLL
 
    03700000 : 03702000 - 00032000 
                    Type     01000000 MEM_IMAGE 
                    Protect  00000020 PAGE_EXECUTE_READ 
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageImage 
                    FullPath C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\erpapv\be46890c\b4f7eeae\assembly\dl3\19c4bca1\9a54ebdc_5a97c901\Assembly2.DLL
 
……………

2. 第二种方法是通过调试器在程序运行过程中记录相应API调用栈。

• 安装调试工具Debug Tool for Windows 将以下配置文件存为TokenLeak.cfg，将OutputDir中目录更改为你需要存log的目录。
• 打开命令行，转到调试器安装目录，运行以下命令
• Cscript.exe –p <PID of the w3wp.exe has token leak problem> -c <FullPathTo TokenLeak.cfg>
• 监测w3wp.exe handle数量增长情况并查看生成的日志文件，日志文件中记录了调用相应API调用栈，根据这些调用栈我们可以回头查看源代码来将没有关闭的handle释放掉。

# ChildEBP RetAddr  Args to Child
00 01f1ef08 0e49126a 01f1f128 01f1f028 01f1ef28 ADVAPI32!LogonUserA (FPO: [Non-Fpo]) (CONV: stdcall) 
01 01f1f244 0e790605 00000001 00000000 060e3178 CLRStub[StubLinkStub]@e49126a 
02 01f1f2a4 0e790505 00000000 00000000 00000000 App_Web_oeootcha!_Default.impersonateValidUser(System.String, System.String, System.String)+0x7d*** WARNING: Unable to verify checksum for C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\securityhandleleak\91744b5d\ab584e94\App_Web_oeootcha.dll  (Managed) 
03 00000000 66f12980 00000000 00000000 00000000 App_Web_oeootcha!_Default.Page_Load(System.Object, System.EventArgs)+0x3d (Managed) 
04 01f1f504 6628efd2 00000000 00000000 00000000 

Configuration File:
 
<ADPlus> 
      <Settings> 
                      <RunMode> CRASH </RunMode> 
                      <Sympath> http://msdl.microsoft.com/download/symbols </Sympath> 
                      <Option> Quiet </Option> 
                      <OutputDir> d:\HandleLeak </OutputDir> 
      </Settings> 
      <Breakpoints> 
                      <NewBP> 
                                      <Address> advapi32!LogonUserA </Address> 
                                      <Type> BP </Type> 
                                      <Actions> stack  </Actions> 
                                      <CustomActions>  </CustomActions> 
                                      <ReturnAction> G </ReturnAction> 
                      </NewBP>   
                      <NewBP>
 
                                      <Address> advapi32!LogonUserW </Address> 
                                      <Type> BP </Type> 
                                      <Actions> stack  </Actions> 
                                      <CustomActions>  </CustomActions> 
                                      <ReturnAction> G </ReturnAction> 
                        </NewBP>  
      </Breakpoints> 
</ADPlus>

 

希望以上内容对你有所帮助

 

weizhao