20130427遇到的2个问题：503错误与Couchbase集群CPU占用不均衡

（注：这2个问题与阿里云一点关系没有）

一、503错误

今天13:00~13:10左右，出现了503错误。出错原因是当时的并发请求数超出了IIS应用程序池的队列长度（Queue Length），当时用的是IIS的默认设置1000（见下图）。

我们将这里的Queue Length由1000改为2000解决了问题（最大可以设置为65535）。

后来发现可以通过 Performance Monitor 监测 "HTTP Service Request queue" -> "Arrival Rate" 来设定 Queue Length。

比如上图中显示"Arrival Rate"的最大值是400，那么Queue Length最好大于400。

看一下当时的负载均衡中一台Web服务器的CPU监控图：

（红色曲线表示%Processor Time，绿色曲线表示Request Execution Time）

不知当时这台云服务器发生了什么异常情况？看来503错误的根源是云服务器的CPU异常，已向阿里云提交工单了解情况。

更新：

经过仔细排查，503错误是当时应用程序池崩溃引起的，应用程序池崩溃是Couchbase客户端引起的，当时正在进行Couchbase集群增/减服务器的操作。

证据来自Windows事件日志：

Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace:    at Hammock.RestClient.CompleteWithQuery(WebQuery query, RestRequest request, RestCallback callback, WebQueryAsyncResult result)
   at Hammock.RestClient.<>c__DisplayClass18.<BeginRequestImpl>b__15(Object sender, WebQueryResponseEventArgs args)
   at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
   at Hammock.Web.WebQuery.OnQueryResponse(WebQueryResponseEventArgs args)
   at Hammock.Web.WebQuery.HandleWebException(WebException exception)
   at Hammock.Web.WebQuery.GetAsyncResponseCallback(IAsyncResult asyncResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.ContextAwareResult.Complete(IntPtr userToken)
   at System.Net.HttpWebRequest.SetResponse(Exception E)
   at System.Net.ConnectionReturnResult.SetResponses(ConnectionReturnResult returnResult)
   at System.Net.Connection.CompleteConnectionWrapper(Object request, Object state)
   at System.Net.PooledStream.ConnectionCallback(Object owningObject, Exception e, Socket socket, IPAddress address)
   at System.Net.ServicePoint.ConnectSocketCallback(IAsyncResult asyncResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.ContextAwareResult.Complete(IntPtr userToken)
   at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
Stack:
   at System.Net.ServicePoint.ConnectSocketCallback(System.IAsyncResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr)
   at System.Net.ContextAwareResult.Complete(IntPtr)
   at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x000007ff0033cbed
Faulting process id: 0x10b4
Faulting application start time: 0x01ce42fb6c5d3e18
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: unknown
Report Id: 30767fd7-aef7-11e2-8bf7-e5d3e0390d57

2. Couchbase集群CPU占用不均衡