Socket Server with .NET 3.5 using pooled buffers and SocketAsyncEventArgs
In a previous post I talked about the System.Net.Sockets enhancements in .NET 3.5, and if you haven't read it I suggest you do before tucking in to this as some of that code is important to understand what's happening here. Before I start, in essence this is just a result of my experimentation and while it seems it does a pretty good job, I'm not going to claim it's bullerproof or that it's a good example of how to write a socket server all it does is demonstrate the techniques of working with the new classes and methods.
The sample solution you can see on the right there contains three projects. FlawlessCode contains all the classes we'll need to build ourselves a socket server. TestLoadGenerator is a console application which generates load for us by connecting lots of sockets to our server and sending it random data. TestSocketServer is a small socket server implementation using the classes in FlawlessCode.
TcpSocketListener
We'll begin by looking at the FlawlessCode project and in particular, the TcpSocketListener. It should be fairly obvious from the name what this class is meant to achieve, it sits in a loop listening for socket connections and lets us know when one arrives. The public interface is very simple and looks like this:
public void Start();
public void Stop();
public event EventHandler<SocketEventArgs> SocketConnected;
The only thing we'll take a closer look at here is the internal loop which accepts the client sockets. Here you can see the first usage of the new SocketAsyncEventArgs and we're calling AcceptAsync, in our callback we check the SocketError property to see if we had any errors.
private void ListenForConnection(SocketAsyncEventArgs args)
{
args.AcceptSocket = null;
listenerSocket.InvokeAsyncMethod(
new SocketAsyncMethod(listenerSocket.AcceptAsync)
, SocketAccepted, args);
}
private void SocketAccepted(object sender, SocketAsyncEventArgs e)
{
SocketError error = e.SocketError;
if (e.SocketError == SocketError.OperationAborted)
return; //Server was stopped
if (e.SocketError == SocketError.Success)
{
Socket handler = e.AcceptSocket;
OnSocketConnected(handler);
}
lock (this)
{
ListenForConnection(e);
}
}
ServerConnection
Next we're going to take a look at the ServerConnection class, this class encapsulates the concept of a connected client. Depending on what you wanted to do with your server you may decide to extend this class, rewrite it or maybe completely replace it with something derived from NetworkStream. For our purposes today, this class when created will begin listening for data from the network, it has two public methods, one to disconnect the client and one to send data synchronously back to the client. ServerConnection also fires two callbacks, one when data is received and one when the client is disconnected. Here is a rundown of the interesting parts:
{
lock (this)
{
CloseConnection(eventArgs);
}
}
public void SendData(Byte[] data, Int32 offset, Int32 count)
{
lock (this)
{
State state = eventArgs.UserToken as State;
Socket socket = state.socket;
if (socket.Connected)
socket.Send(data, offset, count, SocketFlags.None);
}
}
private void ListenForData(SocketAsyncEventArgs args)
{
lock (this)
{
Socket socket = (args.UserToken as State).socket;
if (socket.Connected)
{
socket.InvokeAsyncMethod(socket.ReceiveAsync,
ReceivedCompleted, args);
}
}
}
private void ReceivedCompleted(Object sender,
SocketAsyncEventArgs args)
{
if (args.BytesTransferred == 0)
{
CloseConnection(args); //Graceful disconnect
return;
}
if (args.SocketError != SocketError.Success)
{
CloseConnection(args); //NOT graceful disconnect
return;
}
State state = args.UserToken as State;
Byte[] data = new Byte[args.BytesTransferred];
Array.Copy(args.Buffer, args.Offset, data, 0, data.Length);
OnDataReceived(data, args.RemoteEndPoint as IPEndPoint,
state.dataReceived);
ListenForData(args);
}
private void CloseConnection(SocketAsyncEventArgs args)
{
State state = args.UserToken as State;
Socket socket = state.socket;
try
{
socket.Shutdown(SocketShutdown.Both);
}
catch { } // throws if client process has already closed
socket.Close();
socket = null;
args.Completed -= ReceivedCompleted; //MUST Remember This!
OnDisconnected(args, state.disconnectedCallback);
}
Taking it from the top, we can see the public Disconnect method, this simply calls our internal CloseConnection method which shuts down the socket and fires our disconnected callback. An interesting point to note here is that when this class is instanciated we subscribe to the SocketAsyncEventArgs.Completed event, when a client disconnects we need to remember to unhook this event because when we're reusing objects and pooling resources like this it's a bad idea to leave these references hanging around. Moving down we have the public SendData method, nothing interesting here really, just a standard synchrounous call. Next we get to the internal loop which listens for data from the client, notice how we check SocketAsyncEventArgs.BytesTransferred, if this is zero, the client has closed the connection and disconnected gracefully. We check the value of SocketError here also to make sure there was no error anywhere, after that we make a copy of the bytes we received and inform any interested parties we have new data.
BufferPool and SocketArgsPool
These two classes help us with pooling our resources and are not really very interesting, they're also almost identical to the MSDN examples so you can either look there or just check out the code.
BufferPool: http://msdn2.microsoft.com/en-us/library/bb517542.aspx
SocketArgsPool: http://msdn2.microsoft.com/en-us/library/bb551675.aspx
TestSocketServer
Now that we've sen to main functionality in the FlawlessCode project we're going to look at a simple socket server implementation using these classes.
TcpSocketListener socketListener = new TcpSocketListener(IPAddress.Any, 12345, 10);
socketListener.SocketConnected += socketListener_SocketConnected;
socketListener.Start();
Fairly straight forward, we fire up out listener on port 12345 and give the listening socket an allowed connection backlog of 10.
static void socketListener_SocketConnected(object sender, SocketEventArgs e)
{
SocketAsyncEventArgs args = socketArgsPool.CheckOut();
bufferManager.CheckOut(args);
ServerConnection connection = new ServerConnection(e.Socket, args,
new DataReceivedCallback(DataReceived),
new DisconnectedCallback(Disconnected));
}
When a client connects we get an SocketAsyncEventArgs and some free buffer space for our client and then we create an instance of ServerConnection. Note that we are passing delegates into the constructor, this is because the ServerConnection begins listening for data immediately and we have to have the callbacks hooked up before hand. If we let the call to the constructor complete and the we hooked to standard events we may have already missed the first batch of data!
static void DataReceived(ServerConnection sender, DataEventArgs e)
{
//Do whatever we want here...
}
static void Disconnected(ServerConnection sender, SocketAsyncEventArgs e)
{
bufferManager.CheckIn(e);
socketArgsPool.CheckIn(e);
}
Here we do whatever processing is necessary when a client sends us data. When a client disconnects we just check our buffer space and SocketAsyncEventArgs back into their respective pools to fight another day.
TestLoadGenerator
I'm not going to go into how the load generation works for now, the code is all very straight forward if you've managed to follow the post this far I would imagine. One thing to note is that if you want to test this code and open thousands of connections you need to tweak a registry setting or windows wont give you enough ports. You will need to add the DWORD MaxUserPort to HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and give it a high enough value that windows won't run out of ports (reboot required, sorry)! Here is a quick examaple of how the load generation classes are used:
{
LoadGenerator generator = new LoadGenerator(15000);
generator.BytesPerDelivery = 2048;
generator.DeliveriesPerSecond = 2;
generator.SocketCount = 15000;
generator.SocketDelay = 5;
generator.SocketsPerDelivery = 3;
generator.Start(IPAddress.Parse("127.0.0.1"), 12345);
Console.ReadLine();
generator.Stop();
}
Pretty easy to use, right? We create an instance of the LoadGenerator class, telling it we'd like 15,000 connections maximum. Then we set some properties saying that we'd like each connected socket to deliver 2K of data twice per second. We'd like 15,000 sockets and we'd like them to connect 5ms apart and that we want on average 3% of sockets to send data in each delivery. Then we just aim and fire! Check it out:
When this was taken, the server executable was using 109MB or RAM and 1% CPU on my desktop machine so I think at 15,000 connections we've got some pretty damned good performance out of this thing! Obviously when we start implementing the server logic and actually processing each packet this will go up, but for a bare socket server, I'm pretty pleased.
