epoll server示例

General way to implement tcp servers is “one thread/process per connection”. But on high loads this approach can be not so efficient and we need to use another patterns of connection handling. In this article I will describe how to implement tcp-server with synchronous connections handling using epoll() system call of Linux 2.6. kernel.

epoll is a new system call introduced in Linux 2.6. It is designed to replace the deprecated select (and also poll). Unlike these earlier system calls, which are O(n), epoll is an O(1) algorithm – this means that it scales well as the number of watched file descriptors increase. select uses a linear search through the list of watched file descriptors, which causes its O(n) behaviour, whereas epoll uses callbacks in the kernel file structure.

Another fundamental difference of epoll is that it can be used in an edge-triggered, as opposed to level-triggered, fashion. This means that you receive “hints” when the kernel believes the file descriptor has become ready for I/O, as opposed to being told “I/O can be carried out on this file descriptor”. This has a couple of minor advantages: kernel space doesn’t need to keep track of the state of the file descriptor, although it might just push that problem into user space, and user space programs can be more flexible (e.g. the readiness change notification can just be ignored).

To use epoll method you need to make following steps in your application:

• Create specific file descriptor for epoll calls:
  

  第一步：

    epfd = epoll_create(EPOLL_QUEUE_LEN);

  
  where EPOLL_QUEUE_LEN is the maximum number of connection descriptors you expect to manage at one time. The return value is a file descriptor that will be used in epoll calls later. This descriptor can be closed with close() when you do not longer need it.

• After first step you can add your descriptors to epoll with following call:
  

  第二步，监听事件

    static struct epoll_event ev;
    int client_sock;
    ...
    ev.events = EPOLLIN | EPOLLPRI | EPOLLERR | EPOLLHUP;
    ev.data.fd = client_sock;
    int res = epoll_ctl(epfd, EPOLL_CTL_ADD, client_sock, &ev);

  
  where ev is epoll event configuration sctucture, EPOLL_CTL_ADD – predefined command constant to add sockets to epoll. Detailed description of epoll_ctl flags can be found in epoll_ctl(2) man page. When client_sock descriptor will be closed, it will be automatically deleted from epoll descriptor.

• When all your descriptors will be added to epoll, your process can idle and wait to something to do with epoll’ed sockets:
  

  第三步，等待，非阻塞

  当返回值nfds大于0表示有在epoll里面的socket有读或写等事件了。这时如果刚才的socket是bind+listen的服务端，那接下来就是accept去获取客户端的socket了。

•  

    while (1) {
      // wait for something to do...
      int nfds = epoll_wait(epfd, events, 
                                  MAX_EPOLL_EVENTS_PER_RUN, 
                                  EPOLL_RUN_TIMEOUT);
      if (nfds < 0) die("Error in epoll_wait!");
    
      // for each ready socket
      for(int i = 0; i < nfds; i++) {
        int fd = events[i].data.fd;
        handle_io_on_socket(fd);
      }
    }

   

Typical architecture of your application (networking part) is described below. This architecture allow almost unlimited scalability of your application on single and multi-processor systems:

• Listener – thread that performs bind() and listen() calls and waits for incoming conncetions. Then new connection arrives, this thread can do accept() on listening socket an send accepted connection socket to one of the I/O-workers.
• I/O-Worker(s) – one or more threads to receive connections from listener and to add them to epoll. Main loop of the generic I/O-worker looks like last step of epoll using pattern described above.
• Data Processing Worker(s) – one or more threads to receive data from and send data to I/O-workers and to perform data processing.

As you can see, epoll() API is very simple but believe me, it is very powerful. Linear scalability allows you to manage huge amounts of parallel connections with small amout of worker processes comparing to classical one-thread per connection.

If you want to read more about epoll or you want to look at some benchmarks, you can visit epoll Scalability Web Page at Sourceforge. Another interesting resources are:

• The C10K problem: a most known page about handling many connections and various I/O paradigms including epoll().
• libevent: high-level event-handling library ontop of the epoll. This page contains some information about performance tests of epoll.

一个简单的客户端、服务端示例：

Here's the simple client code:

#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
 
const char* msg = "friendly ping";
 
int main(int argc, char** argv) {
  if (argc != 2) {
    fprintf(stderr, "Usage: %s <socket>\n", argv[0]);
    exit(-1);
  }
 
  int sock = socket(AF_UNIX, SOCK_STREAM, 0);
  if (sock < 0) {
    perror("socket");
    exit(-1);
  }
 
  struct sockaddr_un addr;
  addr.sun_family = AF_UNIX;
  strncpy(addr.sun_path, argv[1], sizeof(addr.sun_path) - 1);
 
  if (connect(sock, (struct sockaddr*)&addr, sizeof(addr)) == -1) {
    perror("Error connecting");
    exit(-1);
  }
 
  write(sock, msg, strlen(msg));
  close(sock);
 
  return 0;
}

The server code is the following:

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
 
const int kMaxEvents = 100;
const char *kSocketFile = "dummy_socket";
 
//设置成 非阻塞 模式
void MakeNonBlocking(int fd) {
  int flags, s;
 
  flags = fcntl (fd, F_GETFL, 0);
  if (flags == -1) {
    perror ("fcntl");
    exit(-1);
  }
 
  flags |= O_NONBLOCK;
  s = fcntl (fd, F_SETFL, flags);
  if (s == -1) {
    perror ("fcntl");
    exit(-1);
  }
}
 
//当有客户端连接时，服务端获取到客户端的连接socket：insock
void AcceptConnections(int sock, int epoll_fd) {
  struct epoll_event event;
  event.events = EPOLLIN;
 
  int insock = accept(sock, NULL, NULL);
  if (insock < 0) {
    perror("Error accepting connection");
    exit(-1);
  }
 
  // This is the important change.
  event.data.fd = insock;
 
  MakeNonBlocking(insock);
//将insock也加入到epoll的监控队列。此时队列中有 来自客户端的insock和服务端的accept socket两个socket。
  int s = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, insock, &event);
  if (s < 0) {
    perror("Epoll error adding accepted connection");
    exit(-1);
  }
  printf("Connection processed.\n");
}
 
int main(void) {
  int sock, efd, n;
  struct sockaddr_un addr;
  struct epoll_event event;
  struct epoll_event *events;
  char buf[1024];
//创建一个socket，它用来当作服务端的 accept接收来用。
  if ((sock = socket(AF_UNIX, SOCK_STREAM, 0)) < 0) {
    perror("Error creating socket.");
    exit(-1);
  }
 
  addr.sun_family = AF_UNIX;
  strncpy(addr.sun_path, kSocketFile, sizeof(addr.sun_path) - 1);
  if (bind(sock, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
    perror("Error binding name to socket");
    exit(-1);
  }
  //listen将sock设置为被动连接模式，即为服务端的等待客户端连接。
  if (listen(sock, SOMAXCONN) < 0) {
    perror("Error listening on socket");
    exit(-1);
  }
//设置为 非阻塞
  MakeNonBlocking(sock);
 
    //创建 epoll示例，类似一个 fd 文件描述符接口
  if ((efd = epoll_create1(0)) < 0) {
    perror("Epoll initialization error");
    exit(-1);
  }
 
  event.data.fd = sock;
  event.events = EPOLLIN;
    //将服务端accept sock加入epoll监听队列
  if (epoll_ctl(efd, EPOLL_CTL_ADD, sock, &event) < 0) {
    perror("Epoll error adding socket");
    exit(-1);
  }
 
  events = (struct epoll_event*) calloc(kMaxEvents, sizeof(event));
  if (!events) {
    perror("Error allocating event buffers");
    exit(-1);
  }
 
  while(1) {
    printf("Calling epoll_wait\n");
      //非阻塞等待
    if ((n = epoll_wait(efd, events, kMaxEvents, -1)) == -1) {
      perror("epoll_wait failure");
      exit(-1);
    }
 
    for (int i = 0; i < n; ++i) {
      printf("Checking event for fd = %d\n", events[i].data.fd);
        
      if (sock == events[i].data.fd) {
          //检查到这个套接字是刚才的服务端accept的sock
        AcceptConnections(sock, efd);
        continue;
      }
//不是服务端accept的sock，那就是后来打开的和客户端的sock，
      int count = read(events[i].data.fd, buf, 100);
      if (count == 0) {
        close(events[i].data.fd);
      }
      write(1, buf, count);
    }
  }
 
  free(events);
  close(efd);
  close(sock);
  unlink(kSocketFile);
  return 0;
}

 

TCP Echo Server Example in C++ Using Epoll

January 2, 2011

This example is a simple server which accepts connections and echos whatever data sent to the server. This example also demonstrates the use of epoll, which is efficient than poll. In epoll unlike poll all events that need to be monitored are not passed everytime the wait call is made. Epoll uses event registration where events to be watched can be added, modified or removed. This makes it efficient when there are a large number of events to be watched.

IOLoop

In this example the class IOLoop will deal with epoll interface and it will invoke relevant handlers based on events occurred.

class IOLoop {
...
 static IOLoop * getInstance();
 
 IOLoop() {
  this->epfd = epoll_create(this->EPOLL_EVENTS);
 
  if(this->epfd < 0) {
   log_error("Failed to create epoll");
   exit(1);
  }
...
 }
 
 void start() {
  for(;;) {
   int nfds = epoll_wait(this->epfd, this->events, this->MAX_EVENTS, -1 /* Timeout */);
 
   for(int i = 0; i < nfds; ++i) {
    int fd = this->events[i].data.fd;
    Handler *h = handlers[fd];
    h->handle(this->events[i]);
   }
  }
 }
 
 void addHandler(int fd, Handler *handler, unsigned int events) {
  handlers[fd] = handler;
  epoll_event e;
  e.data.fd = fd;
  e.events = events;
 
  if(epoll_ctl(this->epfd, EPOLL_CTL_ADD, fd, &e) < 0) {
   log_error("Failed to insert handler to epoll");
  }
 }
 
 void modifyHandler(int fd, unsigned int events);
 
 void removeHandler(int fd);
};

Handlers used in this example are ServerHandler and EchoHandler which derive from class Handler. Handlers have a member function handle which handles the event occurred.

ServerHandler

ServerHandler will create a server socket and handle in coming connections

class ServerHandler : Handler {
...
 
 ServerHandler(int port) {
  memset(&addr, 0, sizeof(addr));
 
  if ((fd = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0) {
   log_error("Failed to create server socket");
   exit(1);
  }
 
  addr.sin_family = AF_INET;
  addr.sin_addr.s_addr = htonl(INADDR_ANY);
  addr.sin_port = htons(port);
 
  if (bind(fd, (struct sockaddr *) &addr,
                               sizeof(addr)) < 0) {
   log_error("Failed to bind server socket");
   exit(1);
  }
 
  if (listen(fd, MAX_PENDING) < 0) {
   log_error("Failed to listen on server socket");
   exit(1);
  }
  setnonblocking(fd);
 
  IOLoop::getInstance()->addHandler(fd, this, EPOLLIN);
 }
 
 virtual int handle(epoll_event e) {
  sockaddr_in client_addr;
  socklen_t ca_len = sizeof(client_addr);
 
  int client = accept(fd, (struct sockaddr *) &client_addr,
                  &ca_len);
 
  if(client < 0) {
   log_error("Error accepting connection");
   return -1;
  }
 
  cout << "Client connected: " << inet_ntoa(client_addr.sin_addr) << endl;
  new EchoHandler(client, client_addr);
  return 0;
 }
};

Set Non-blocking

Function setnonblocking sets the file descriptor setting to non-clocking.

flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);

EchoHandler:handle

EchoHandler will write whatever it reads from the socket

virtual int handle(epoll_event e) {
 if(e.events & EPOLLHUP) {
  IOLoop::getInstance()->removeHandler(fd);
  return -1;
 }
 
 if(e.events & EPOLLERR) {
  return -1;
 }
 
 if(e.events & EPOLLOUT) {
  if(received > 0) {
   cout << "Writing: " << buffer << endl;
   if (send(fd, buffer, received, 0) != received) {
    log_error("Error writing to socket");
   }
  }
 
  IOLoop::getInstance()->modifyHandler(fd, EPOLLIN);
 }
 
 if(e.events & EPOLLIN) {
  if ((received = recv(fd, buffer, BUFFER_SIZE, 0)) < 0) {
   log_error("Error reading from socket");
  } else if(received > 0) {
   buffer[received] = 0;
   cout << "Reading: " << buffer << endl;
  }
 
  if(received > 0) {
   IOLoop::getInstance()->modifyHandler(fd, EPOLLOUT);
  } else {
   IOLoop::getInstance()->removeHandler(fd);
  }
 }
 
 return 0;
}

Error checking in this code is minimal so it will probably fail unexpectedly in certain scenarios which I have not come across yet. And please leave a comment if you do find any errors or if there are things that could be improved.