I already know the answer to this, but wanted to share with the community since it is NOT documented from Microsoft.
The scenario: A surge of traffic hits your IIS 7.5 ASP.NET website, and you notice that requests start queuing up. The performance of the site slows to a crawl, yet you have plenty of CPU and RAM available.
This is the problem we saw recently with a site that made a bunch of internal web-service calls. An internal health check would start timing-out, which would cause this server to drop out of our cluster. (However, this server is the most powerful hardware of the bunch ...)
After searching around the internet, I found the following articles from Microsoft that relate to the problem:
This article gives some great performance-tweaking tips, however it fails to mention some some VERY important ceilings that we ran in to.
The solution for us was to modify our machine.config, and populate the following XML nodes:
<system.web><processModelautoConfig="false"maxWorkerThreads="xxx"maxIoThreads="xxx"minWorkerThreads="xxx"minIoThreads="xxx"requestQueueLimit="5000"responseDeadlockInterval="00:03:00"/><httpRuntimeminFreeThreads="xxx"minLocalRequestFreeThreads="xxx"/></system.web>
I purposefully set some of these numbers to "xxx" since they are dependent upon your hardware.
From the KB article above, Microsoft suggests some equations for figuring out these values. However, they fail to mention that the MAXIMUM value for these numbers is the size of an INT, or 32767.
So, the CORRECT equations for figuring these out are as follows:
- maxWorkerThreads: 32767 / #Cores
- In our case, we have a 24-core server. So, our maxWorkerThreads value is correctly set to: 1365. Any number that results in an integer LARGER than 32767, the server will set the maxWorkerThreads to 32767.
- maxIoThreads: Same as maxWorkerThreads (32767 / #Cores)
- minWorkerThreads: maxWorkerThreads / 2
- This was a tricky one. If one exceeded an integer value LARGER than 32767 (and despite what the KB article says, this number IS multiplied by the number of cores you have), and unlike the "max" value, this defaults to the number of cores on your machine! In our case, this was getting set to 24 (because we set an arbitrarily high value for the min), and that was KILLING the performance on our server.
- minIoThreads: Same as minWorkerThreads
- minFreeThreads: 88 * #Cores (taken directly from the KB article)
- minLocalRequestFreeThreads: 76 * #Cores (taken directly from the KB article)
This solution is not for everyone, and should only be used if you meet the criteria in the KB article.
Another tool we used in helping us diagnose this was an .ASPX page with no code-behind that we could throw out on any server (without resetting the Application Pool). This page uses reflection to tell you what is actually going on in the thread pool, and what the values of these settings render to on your server.
<%@PageLanguage="C#" %>
<!DOCTYPE html><htmllang="en"><head><style>
body {margin:20pt;padding:0pt;font-family: Verdana,"san-serif";}
fieldset {border-radius:5px;border: none;background-color:#fff;margin:10pt;}
fieldset.parent {background-color:#f0f0f0;}
legend {font-size:10pt;color:#888;margin:5pt;}.ports div {padding:10pt0pt0pt0pt;clear: both;}.ports div:first-child {padding:0pt;}.ports div div {padding:0pt;clear: none;margin:1pt;background-color:#eef;display: block;float: left;border:5pt solid #eef;}.ports div div:first-child {border-top-left-radius:5pt;border-bottom-left-radius:5pt;background-color:#ccf;border-color:#ccf;}.ports div div:last-child {border-top-right-radius:5pt;border-bottom-right-radius:5pt;background-color:#ccf;border-color:#ccf;padding:0pt10pt0pt10pt;}</style></head><body><%Response.Cache.SetCacheability(HttpCacheability.NoCache);int worker, workerMIN, workerMAX;int port, portMIN, portMAX;System.Threading.ThreadPool.GetAvailableThreads(out worker,out port);System.Threading.ThreadPool.GetMinThreads(out workerMIN,out portMIN);System.Threading.ThreadPool.GetMaxThreads(out workerMAX,out portMAX);
%>
<fieldsetclass="parent"><legend>Thread Information</legend><fieldset><legend>Worker Threads</legend><divclass="ports"><div><div>Min: <%=workerMIN %></div><div>Current: <%=workerMAX - worker %></div><div>Max: <%=workerMAX %></div></div></div></fieldset><fieldset><legend>Completion Port Threads</legend><divclass="ports"><div><div>Min: <%=portMIN %></div><div>Current: <%=portMAX - port %></div><div>Max: <%=portMAX %></div></div></div></fieldset><fieldset><legend>Request Queue Information</legend><divclass="ports"><%varfi=typeof(HttpRuntime).GetField("_theRuntime",System.Reflection.BindingFlags.NonPublic|System.Reflection.BindingFlags.Instance|System.Reflection.BindingFlags.Static).GetValue(null);var rq =typeof(HttpRuntime).GetField("_requestQueue",System.Reflection.BindingFlags.NonPublic|System.Reflection.BindingFlags.Instance).GetValue(fi);var fields = rq.GetType().GetFields(System.Reflection.BindingFlags.NonPublic|System.Reflection.BindingFlags.Instance);foreach(var field in fields){string name = field.Name;string value ="";switch(name){case"_localQueue":case"_externQueue":System.Collections.Queue queue = field.GetValue(rq)asSystem.Collections.Queue;
value = queue.Count.ToString();break;default:
value = field.GetValue(rq).ToString();break;}
%>
<div><div><%=name %></div><div><%=value %></div></div><%//Response.Write(string.Format("{0}={1}<br/>", name, value));}
%>
</div></fieldset></fieldset></body></html>
