We have about 1000 separate sites that are the same .Net application (different codebase folders and app pools per site).
64 bit Windows and running .Net 2.0, application is compiled with 'AnyCPU' flag.
Since the same exact code works on the old server and never throws outofmemoryexceptions, we are holding off on spending major time profiling the application and examining dumps and performing code optimizations that would help avoid large object heap fragmentations (so we are hoping to get some hints on possible server configratuion issues that may be the culprit more than looking in the code base and optimizing it...).
Config 1 - Rackspace CloudSites (shared hosting, we can only FTP to it, no access to IIS settings):
1 IIS server, we don't have control over managing it but were told each app pool has a 250MB recycle limit.
Of our 1000 sites, many sites (20-50ish) apparently share the same app pool. We never get OutOfMemoryExceptions here and have the application running on it since years.
Config 2 - Rackspace Dedicated Server (full control):
Monster server with 128GB RAM, dedicated, each site has it's own app pool. All app pools have the same settings (350MB recycle limit). Not sure if this is of consequence but the page file size on this sever is 4GB (no idea what Config 1 has - does this need to be increased/addressed?).
Both configs are load balanced across 2 or 3 web servers, but that really doesn't matter here per se, since we are seing sites with NO TRAFFIC getting killed with OutOfMemoryExceptions.
长话短说，简单处理方式：
This post escalated while I wrote it, here's the take-away bullets
- Increase the size of your page file (shooting from the hip I'd say to at least 40GB, much more if you can afford the disk capacity and I/O, but read the article at the bottom)
- Increase the
frequentHitThresholdandfrequentHitTimePeriodvalues(review the Web Service Cache performance counters and adjust accordingly) - Lower the maxResponseSize value to 85KB or below to avoid cache entries in the large object heap
- Lower the Memory Limit for application pool Recycling, it doesn't make much sense
- Consider grouping applications with the same or similar codebase in application pools
Original Answer
Not sure if this is of consequence but the page file size on this sever is 4GB
This, here ^ right up here ^ look at it.
I'm willing to bet that this is exactly the reason why your applications throws a OutOfMemoryException in response to the seemingly most random and benign requests,
but to understand why, let's get one thing clear:
OutOfMemory does not mean that your server is out of memory!
I know it sounds like a bad joke, but it really isn't. It's not the Operating System complaining about memory depletion - it's the process.
If that last statement made no sense to you, please read on.
Memory Management 101
When a process allocates memory from the OS, it comes in a sequence of fragments called pages, 4 kilobytes a piece, that the process can treat as its own (this is commonly referred to as a Virtual Address Space).
Since an object (such as a string, an XML document, an image or whatever you need to keep in memory) can exceed the pagesize of 4KB, the process will need to allocate multiple successive pages from this memory from time to time.
Over time however, the memory space becomes fragmented, even using the .NET CLR. The Garbage Collector will try and do what it can to help your application make better use of the address space by rearranging the pages in the working set during collections (this is practically the same as a disk defragmentation), but pointers to the large object heap for example, will be left untouched.
How IIS 7.x plays a role
As recently explained in this answer, IIS will also try to store as many cacheable output objects (such as static files up to 256KB) as it can, in the same process that serves your application - in addition to the suggestion in that answer, you can also try to tune the caching frequency thresholds using the <serverRuntime> configuration element.
In any case, IIS 7.5 - in its default configuration - cares deeply about allocating enough memory for its worker processes, and even with "NO TRAFFIC", it's not uncommon to see a worker process claim the first 100MB when it spins up, even with a slightly smaller application codebase on disk.
What does this have to do with the pagefile?
It doesn't take graduate level math to see that 100MB * 1000 processes is not that far from the 128GB of RAM the OS has to offer. Even though IIS tries to allocate as much memory for its worker processes as needed, it stops at some point to leave some room for the operating system at around 85% of total memory installed, regardless of how many mega- or gigabytes of RAM that might be (this is not a fact I've seen documentation for, but drawn from first-hand experience with a large range of IIS installation with different hardware specs).
At this point, the operating system can help out freeing memory by allocating pages from the pagefile instead - pages stored in a file on a physical disk. Since disk capacity is often plentyful, allocating large chunks of disk storage is not that big a deal, but if it needs to page memory for a 1000 processes and is only allowed to do so on 4GB of space, it doesn't take long before the processes will not be able to allocate longer sequences of non-fragmented memory and puff!: the process throws an OutOfMemoryException, it simply means that it was not able to find enough adjacent pages for something in its accessible virtual address space.
It doesn't even have to be large objects. They just have to be larger than the maximum number of contiguous pages you have available at runtime. Theoretically, you could get an OutOfMemory exception for trying to append a single character to a string that's currently more than 2KB in size.
What should the Pagefile size be set to then?
Microsofts answer to this question has always been: "it depends", but at least 1 x RAM + 257MB (this is the amount of storage the system requires to be able to write a complete memory dump).
A rule-of-thumb seems to be around 1.5-2 x RAM but again, it depends, and a number of articles has been published on how to determine the right minimum and maximum page file size on a given system. I've included the most relevant one in the bottom.
Be sure to monitor the disk containing the pagefile as well, if Disk Queue Length counters start spiking, you might want to move it to a dedicated disk, or spread it out over multiple disks.
How to determine the appropriate page file size for 64-bit versions of Windows
You can Disable your Application Pool Memory limits (set it to 0) to quickly allow your application pool to take up as much as needed.
However, it sounds like something is leaking and will eventually take up all available memory (and recycle the app pool again).
Check your c:\windows\system32\LogFiles\HTTPERR log file for fatal error messages.
You should not have to modify your PageFile settings, that is only used when the 128GB of RAM is depleted.
To track down the source of the memory leak you'll need to do more digging with Trace Logs or DebugDiag (not the easiest task):
How to use the IIS Debug Diagnostics tool to troubleshoot a memory leak in an IIS process
Troubleshooting native memory leak in an IIS 7.x Application Pool
Our issue was resolved by switching the 1000+ app pools to 32-bit mode instead of the default 64 bit.
We were able to confirm this by flipping back to 64-bit and immediately seeing the outofmemory exceptions.
I believe Jessen's points on the page file do have weight. . . since upping the page file required a system restart,
we tried to flip to 32-bit mode app pools (which doesn't require restarting) and got lucky.
If anyone has some interesting comments on this please feel free!
Thanks!
我们有大约 1000 个独立的站点，它们都是相同的 .Net 应用程序（每个站点有不同的代码库文件夹和应用程序池）。
64 位 Windows 并运行 .Net 2.0，应用程序使用“AnyCPU”标志进行编译。
由于相同的代码在旧服务器上运行并且永远不会抛出内存不足异常，因此 我们推迟花费大量时间分析应用程序并检查转储并执行代码优化，这将有助于避免大型对象堆碎片（因此我们希望得到一些提示可能的服务器配置问题可能是罪魁祸首，而不仅仅是查看代码库并对其进行优化......）。
配置 1 - Rackspace CloudSites（共享主机，我们只能通过 FTP 访问它，无法访问 IIS 设置）：
1 IIS 服务器，我们无法控制管理它，但被告知每个应用程序池有 250MB 的回收限制。
在我们的 1000 个网站中，许多网站（20-50 左右）显然共享相同的应用程序池。我们在这里从未遇到过 OutOfMemoryExceptions，并且多年来一直在其上运行应用程序。
配置 2 - Rackspace 专用服务器（完全控制）：
拥有 128GB RAM 的 Monster 服务器，专用，每个站点都有自己的应用程序池。所有应用程序池都具有相同的设置（350MB 回收限制）。不确定这是否有影响，但该服务器上的页面文件大小为4GB（不知道配置 1 有什么 - 这是否需要增加/解决？）。
这两个配置都在 2 或 3 个 Web 服务器之间进行负载平衡，但这本身并不重要，因为我们看到没有流量的网站因 OutOfMemoryExceptions 被杀死。
长话短说，简单处理方式：
这篇文章在我写的时候升级了，这是要点
- 增加页面文件的大小（从臀部开始，我想说至少 40GB，如果您能负担得起磁盘容量和 I/O，则更多，但请阅读底部的文章）
- 增加
frequentHitThreshold和frequentHitTimePeriod值（查看 Web 服务缓存性能计数器并进行相应调整） - 将maxResponseSize值降低 到 85KB 或更低，以避免在大对象堆中缓存条目
- 降低应用程序池回收的内存限制，没有多大意义
- 考虑将具有相同或相似代码库的应用程序分组到应用程序池中
原答案
不确定这是否有影响，但该服务器上的页面文件大小为4GB
这个，就在 ^ 这里 ^ 看看吧。
我敢打赌，这正是您的应用程序抛出 a OutOfMemoryException 来响应看似最随机和良性的请求的原因，
但为了理解为什么，让我们弄清楚一件事：
OutOfMemory 并不意味着您的服务器内存不足！
我知道这听起来像是一个糟糕的笑话，但事实并非如此。 抱怨内存耗尽的不是操作系统，而是进程。
如果最后一句话对您来说没有意义，请继续阅读。
内存管理 101
当进程从操作系统分配内存时，它会出现在一系列称为 页面的片段中，每块 4 KB，进程可以将其视为自己的内存（通常称为虚拟地址空间）。
由于对象（例如字符串、XML 文档、图像或需要保存在内存中的任何内容）可能超过 4KB 的页面大小，因此进程需要不时地从该内存中分配多个连续的页面。
然而，随着时间的推移，即使使用 .NET CLR，内存空间也会变得碎片化。垃圾收集器将尝试通过在收集期间重新排列工作集中的页面（这实际上与磁盘碎片整理相同）来帮助您的应用程序更好地利用地址空间，但指针指向大对象堆例如，将保持不变。
IIS 7.x 如何发挥作用
正如此答案中最近 所解释的，IIS 还将尝试在为您的应用程序提供服务的同一进程中存储尽可能多的可缓存输出对象（例如高达 256KB 的静态文件） - 除了该答案中的建议之外，您还可以还可以尝试使用 <serverRuntime> 配置元素调整缓存频率阈值。
无论如何，IIS 7.5 - 在其默认配置中 - 非常关心为其工作进程分配足够的内存，即使在“无流量”的情况下，工作进程在启动时占用第一个 100MB 的情况也并不罕见，即使在磁盘上稍小的应用程序代码库。
这与页面文件有什么关系？
不需要研究生水平的数学知识就可以看出 100MB * 1000 个进程与操作系统提供的 128GB RAM 相差不远。尽管 IIS 尝试为其工作进程分配所需的尽可能多的内存，但它会在某个时刻停止，以便在安装的总内存的 85% 左右时为操作系统留出一些空间，无论可能需要多少兆或千兆字节的 RAM （这不是我见过的文档，而是根据不同硬件规格的大量 IIS 安装的第一手经验得出的）。
此时，操作系统可以通过从页面文件中分配页面（存储在物理磁盘上的文件中的页面）来帮助释放内存。由于磁盘容量通常很充足，分配大块磁盘存储并不是什么大问题，但如果需要为 1000 个进程分配内存并且只允许在 4GB 空间上这样做，那么很快就会出现问题。这些进程将无法分配更长的非碎片内存序列和 噗！：进程抛出一个 OutOfMemoryException，它仅仅意味着它无法在其可访问的虚拟地址空间中为某些内容找到足够的相邻页面。
它甚至不必是大物体。它们只需大于运行时可用的最大连续页面数即可。理论上，如果尝试将单个字符附加到当前大小超过 2KB 的字符串，可能会出现 OutOfMemory 异常。
那么页面文件大小应该设置为多少呢？
微软对这个问题的回答一直是：“这取决于”，但至少 1 x RAM + 257MB（这是系统能够写入完整内存转储所需的存储量）。
经验法则似乎约为1.5-2 x RAM，但同样，这取决于情况，并且已经发表了许多关于如何确定给定系统上正确的最小和最大页面文件大小的文章。我已将最相关的一个放在底部。
请务必监视包含页面文件的磁盘，如果磁盘队列长度计数器开始达到峰值，您可能需要将其移动到专用磁盘，或将其分布在多个磁盘上。
您可以禁用应用程序池内存限制（将其设置为 0），以快速允许应用程序池占用所需的内存。
然而，听起来有些东西正在泄漏，最终会占用所有可用内存（并再次回收应用程序池）。
检查 c:\windows\system32\LogFiles\HTTPERR 日志文件中是否有致命错误消息。
您不必修改页面文件设置，该设置仅在 128GB RAM 耗尽时使用。
要追踪内存泄漏的根源，您需要使用跟踪日志或 DebugDiag 进行更多挖掘（这不是最简单的任务）：
This is Windows server 2008 R2, 64 bit, 32gb RAM, I think its running IIS 7.5. We have set the application pool to use 4 worker process.
This is a ASP.NET 4 application but running in 32 bit compatability mode.
We are getting oSystem.OutOfMemoryException when the memory usage crosses more than 650-700MB/worker process.
I thought that it should be able to handle upto 2gb or atleast 1.5 gb with no issues?
Another thing, why does it not recycle the worker process when there is a System.OutOfMemoryException?
update: This application works perfectly fine on a 64bit windows server 2003 with IIS6.0. I have seen the max memory usage of it being around 700mb/worker process.
Update: The reason for high memory usage is XML processing using DOM. We are going to start work to fix that, but thats a long term plan. I just find it weird that it cannot go higher than 650 mb.
南来地,北往的,上班的,下岗的,走过路过不要错过!
======================个性签名=====================
之前认为Apple 的iOS 设计的要比 Android 稳定,我错了吗?
下载的许多客户端程序/游戏程序,经常会Crash,是程序写的不好(内存泄漏?刚启动也会吗?)还是iOS本身的不稳定!!!
如果在Android手机中可以简单联接到ddms,就可以查看系统log,很容易看到程序为什么出错,在iPhone中如何得知呢?试试Organizer吧,分析一下Device logs,也许有用.
