Linux socket编程（四）简单聊天室之epoll版

　　这一篇我们用epoll改写之前写的简单聊天室，Epoll是Linux内核为处理大批量句柄而作了改进的poll。

我们要用到epoll的三个函数，分别是：int epoll_create(int size);

int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);

　　　　 int epoll_wait(int epfd, struct epoll_event * events, int maxevents, int timeout);

下面对要用到epoll的操作进行封装

Epoll.h

#ifndef EPOLL_H
#define EPOLL_H

#include "Socket.h"
#include <sys/epoll.h>
#include <sys/resource.h>

const int MAXEPOLLSIZE=MAXCONNECTION+5;

class Epoll
{
    public:
        Epoll();
        bool Add(int fd,int eventsOption);
        //Returns the number of triggered events
        int Wait();
        bool Delete(const int eventIndex);
        int GetEventOccurfd(const int eventIndex) const;
        int GetEvents(const int eventIndex) const;

    private:
        int epollfd;
        int fdNumber;
        struct epoll_event event;
        struct epoll_event events[MAXEPOLLSIZE];
        struct rlimit rt;
};

#endif

Socket类的实现见我的这篇博文Linux socket编程（一）对套接字操作的封装

更好一点的做法是把Socket类做成一个共享函数库

Epoll.cpp

#include "Epoll.h"
#include <stdio.h>
#include <stdlib.h>

Epoll::Epoll()
:fdNumber(0)
{
    //set resource limits respectively
    rt.rlim_max=rt.rlim_cur=MAXEPOLLSIZE;
    if(::setrlimit(RLIMIT_NOFILE, &rt) == -1)
    {
        perror("setrlimit");
        exit(1);
    }
    //create epoll
    epollfd=epoll_create(MAXEPOLLSIZE);
}

bool Epoll::Add(int fd,int eventsOption)
{
    //handle readable event,set Edge Triggered
    event.events=eventsOption;//EPOLLIN | EPOLLET;
    event.data.fd=fd;

    if(epoll_ctl(epollfd,EPOLL_CTL_ADD,fd,&event)<0)
        return false;

    fdNumber++;
    return true;
}

bool Epoll::Delete(const int eventIndex)
{
    if(epoll_ctl(epollfd,EPOLL_CTL_DEL,
                 events[eventIndex].data.fd,&event)<0)
        return false;
    fdNumber--;
    return true;
}

int Epoll::Wait()
{
    int eventNumber;
    eventNumber=epoll_wait(epollfd,events,fdNumber,-1);
    if(eventNumber<0)
    {
        perror("epoll_wait");
        exit(1);
    }
    return eventNumber;
}

int Epoll::GetEventOccurfd(const int eventIndex) const
{
    return events[eventIndex].data.fd;
}

int Epoll::GetEvents(const int eventIndex) const
{
    return events[eventIndex].events;
}

现在考虑如何把epol用到socket的通信中

参考了这篇博文 http://www.cnblogs.com/OnlyXP/archive/2007/08/10/851222.html

epoll有两种触发模式：

LT(level triggered)是缺省的工作方式，并且同时支持block和no-block socket.在这种做法中，内核告诉你一个文件描述符是否就绪了，然后你可以对这个就绪的fd进行IO操作。如果你不作任何操作，内核还是会继续通知你的，所以，这种模式编程出错误可能性要小一点。传统的select/poll都是这种模型的代表．

ET(edge-triggered) 是高速工作方式，只支持no-block socket。在这种模式下，当描述符从未就绪变为就绪时，内核通过epoll告诉你。然后它会假设你知道文件描述符已经就绪，并且不会再为那个文件描述符发送更多的就绪通知，直到你做了某些操作导致那个文件描述符不再为就绪状态了(比如，你在发送，接收或者接收请求，或者发送接收的数据少于一定量时导致了一个EWOULDBLOCK 错误）。但是请注意，如果一直不对这个fd作IO操作(从而导致它再次变成未就绪)，内核不会发送更多的通知(only once)。

接下来我们使用边沿触发这种方式（ET），先看一下手册是怎么说的（man epoll）：

 　　  Q9  Do I need to continuously read/write a file descriptor until EAGAIN when
           using the EPOLLET flag (edge-triggered behavior) ?

       A9  Receiving an event from epoll_wait(2) should suggest to you that such file
           descriptor is ready for the requested I/O operation.  You must consider it
           ready until the next (nonblocking) read/write yields EAGAIN.  When and how
           you will use the file descriptor is entirely up to you.

           For packet/token-oriented files (e.g., datagram socket, terminal in
           canonical mode), the only way to detect the end of the read/write I/O
           space is to continue to read/write until EAGAIN.

           For stream-oriented files (e.g., pipe, FIFO, stream socket), the condition
           that the read/write I/O space is exhausted can also be detected by
           checking the amount of data read from / written to the target file
           descriptor.  For example, if you call read(2) by asking to read a certain
           amount of data and read(2) returns a lower number of bytes, you can be
           sure of having exhausted the read I/O space for the file descriptor.  The
           same is true when writing using write(2).  (Avoid this latter technique if
           you cannot guarantee that the monitored file descriptor always refers to a
           stream-oriented file.)

意思大概是说当使用ET这种方式时，要不断地对文件描诉符进行读/写，直至遇到EAGAIN为止。

为什么要这样呢：

假如发送端流量大于接收端的流量 (意思是epoll所在的程序读比转发的socket要快),由于是非阻塞的socket,那么send()函数虽然返回,但实际缓冲区的数据并未真正发给接收端,这样不断的读和发，当缓冲区满后会产生EAGAIN错误(参考man send),同时,不理会这次请求发送的数据.所以,需要封装socket_send()的函数用来处理这种情况,该函数会尽量将数据写完再返回，同样对于recv函数也要进行相应的封装。

以下是我的封装：（英文注释写的不是很好，大家凑合着看吧）

void EpollServerSocket::SendMessage(Socket& clientSocket,const std::string& message) const
{
    while(true)
    {
        if(Socket::Send(clientSocket,message)==false)
        {
            //
            if(errno == EINTR)
                return;

            //this means the cache queue is full,
            //sleep 1 second and send again
            if(errno==EAGAIN)
            {
                sleep(1);
                continue;
            }
        }

        return;
    }
}

void EpollServerSocket::ReceiveMessage(Socket& clientSocket,std::string& message)
{
    bool done=true;

    while(done)
    {
        int receiveNumber=Socket::Receive(clientSocket,message);
        if(receiveNumber==-1)
        {
            //if errno == EAGAIN, that means we have read all data.
            //so return
            if (errno != EAGAIN)
            {
                perror ("ReceiveMessage error");
                DeleteClient(clientSocket.GetSocketfd());
            }
            return;
        }
        else if(receiveNumber==0)
        {
            // End of file. The remote has closed the connection.
            DeleteClient(clientSocket.GetSocketfd());
        }

        //if receiveNumber is equal to MAXRECEIVE,
        //maybe there is data still in cache,so it has to read again
        if(receiveNumber==MAXRECEIVE)
            done=true;
        else
            done=false;
    }
}

好了接下来是Socket类的派生类，EpollServerSocket类

EpollServerSocket.h

#ifndef EPOLLSERVERSOCKET_H
#define EPOLLSERVERSOCKET_H

#include "Socket.h"
#include "Epoll.h"
#include <map>

class EpollServerSocket:public Socket
{
    public:
        EpollServerSocket(const int port);
        virtual ~EpollServerSocket();

        void Run();

    private:
        //when using the EPOLLET flag,
        //need to continuously read/write a file descriptor until EAGAIN,
        //so we write these two functions for read/write
        void SendMessage(Socket& clientSocket,const std::string& message) const;
        void ReceiveMessage(Socket& clientSocket,std::string& message);

        void ProcessMessage(Socket& clientSocket);
        void SendToAllUsers(const std::string& message) const;
        //add event to epoll
        bool AddNewClient(Socket& clientSocket);
        //delete client from map clientSockets
        void DeleteClient(int sockfd);
        std::map<int,Socket*> clientSockets;
        Epoll epoll;
};

#endif

以下是EpollServerSocket类的实现

View Code

#include "EpollServerSocket.h"
#include "SocketException.h"
#include <iostream>
#include <errno.h>
#include <stdio.h>

#define DEBUG

EpollServerSocket::EpollServerSocket(const int port)
{
      if ( ! Socket::Create() )
        {
          throw SocketException ( "Could not create server socket." );
        }

      if ( ! Socket::Bind ( port ) )
        {
          throw SocketException ( "Could not bind to port." );
        }

      if ( ! Socket::Listen() )
        {
          throw SocketException ( "Could not listen to socket." );
        }

      //set listener socket non-blocking!!
      Socket::SetNonBlocking(true);

}

EpollServerSocket::~EpollServerSocket()
{
    std::map<int,Socket*>::iterator it;
    for(it=clientSockets.begin();it!=clientSockets.end();it++)
        delete it->second;
}

void EpollServerSocket::Run()
{
    //add listener socketfd to epoll
    if(epoll.Add(Socket::GetSocketfd(),EPOLLIN)==false)
        return;

    int i;
    int eventNumber;
    Socket* clientSocket;

    while(true)
    {
        eventNumber=epoll.Wait();

        #ifdef DEBUG
            std::cout<<"eventNumbers: "<<eventNumber<<" ";
        #endif

        for(i=0;i<eventNumber;i++ )
        {
             if ((epoll.GetEvents(i) & EPOLLERR) ||
                 (epoll.GetEvents(i) & EPOLLHUP) ||
                 (!(epoll.GetEvents(i) & EPOLLIN)))
            {
                      /* An error has occured on this fd, or the socket is not
                         ready for reading (why were we notified then?) */
                      perror ("epoll error\n");
                      DeleteClient(epoll.GetEventOccurfd(i));
                      continue;
            }

            //if event is triggered by listener socket
            else if(epoll.GetEventOccurfd(i)==Socket::GetSocketfd())
            {
                clientSocket=new Socket();
                if(AddNewClient(*clientSocket)==false)
                    return;
                clientSockets[clientSocket->GetSocketfd()]=clientSocket;

            }
            //else event is triggered by client sockets
            else
            {
                clientSocket=clientSockets[epoll.GetEventOccurfd(i)];
                ProcessMessage(*clientSocket);
            }
        }
    }
}

void EpollServerSocket::ProcessMessage(Socket& clientSocket)
{
    std::string message;
    ReceiveMessage(clientSocket,message);

    if(message=="exit")
    {
        SendMessage(clientSocket,"user_exit");

        DeleteClient(clientSocket.GetSocketfd());
    }
    else
        SendToAllUsers(message);
}

bool EpollServerSocket::AddNewClient(Socket& clientSocket)
{
    if(Socket::Accept(clientSocket)==false)
        return false;

    //set socket non-blocking!!
    clientSocket.SetNonBlocking(true);

    if(epoll.Add(clientSocket.GetSocketfd(),EPOLLIN | EPOLLET)==false)
        return false;

    #ifdef DEBUG
        std::cout<<"New user...\n";
    #endif

    return true;
}

void EpollServerSocket::DeleteClient(int sockfd)
{
    //epoll doesn't need to handle events from sockfd anymore
    epoll.Delete(sockfd);

    delete clientSockets[sockfd];
    clientSockets.erase(sockfd);
}

void EpollServerSocket::SendToAllUsers(const std::string& message) const
{
    std::map<int,Socket*>::const_iterator it;
    for(it=clientSockets.begin();it!=clientSockets.end();it++)
        SendMessage(*(it->second),message);
}

void EpollServerSocket::SendMessage(Socket& clientSocket,const std::string& message) const
{
    while(true)
    {
        if(Socket::Send(clientSocket,message)==false)
        {
            //this means the socket can be wrote
            if(errno == EINTR)
                return;

            //this means the cache queue is full,
            //sleep 1 second and send again
            if(errno==EAGAIN)
            {
                sleep(1);
                continue;
            }
        }

        return;
    }
}

void EpollServerSocket::ReceiveMessage(Socket& clientSocket,std::string& message)
{
    bool done=true;

    while(done)
    {
        int receiveNumber=Socket::Receive(clientSocket,message);
        if(receiveNumber==-1)
        {
            //if errno == EAGAIN, that means we have read all data.
            if (errno != EAGAIN)
            {
                perror ("ReceiveMessage error");
                DeleteClient(clientSocket.GetSocketfd());
            }
            return;
        }
        else if(receiveNumber==0)
        {
            // End of file. The remote has closed the connection.
            DeleteClient(clientSocket.GetSocketfd());
        }

        //if receiveNumber is equal to MAXRECEIVE,
        //maybe there is data still in cache,so it has to read again
        if(receiveNumber==MAXRECEIVE)
            done=true;
        else
            done=false;
    }
}

（以前写的客户端不用更改，直接可以与这个服务器通信）

对于大数据量的传输，很明显要不断地进行读/写，这样就会出现长时间的阻塞，甚至成为系统的性能瓶颈

但是对于只有较少活跃的socket，同时数据量较小的情况，epoll的效率应该是比select和poll高的（呃，不过没有很好的测试过）

不过好像有一种做法可以避免阻塞，就是利用EPOLLOUT事件

“EPOLLOUT事件的意思就是当前这个socket的发送状态是空闲的，此时处理能力很强，告知用户可以发送数据。
所以在正常情况下，基本上socket在epoll_wait后，都会得到一个socket的EPOLLOUT事件。

【如果你不是一直在写数据或者你不是在传送一个几百M的数据文件，send一半都处于空闲状态】
而这个特性刚好可以处理阻塞问题。
当数据发送不出去的时候，说明网络阻塞或者延迟太厉害了。
那么将要发送的数据放在一个buffer中，当下次你发现了EPOLLOUT事件时，说明现在网络处于空闲状态，OK，此时你可以用另外一个线程来发送上次堆积在buffer中的数据了。这样就不会阻塞了“

本文为原创博文，转载请注明原作者博客地址：http://www.cnblogs.com/-Lei/archive/2012/09/12/2681475.html

posted @ 2012-09-12 14:00 _Lei 阅读(7480) 评论(4) 编辑收藏举报

刷新页面返回顶部

_Lei

Linux socket编程（四） 简单聊天室之epoll版

公告

Linux socket编程（四）简单聊天室之epoll版