Socket网络编程

一、socket网络编程

先来理解TCP,UDP协议

     TCP/IP（Transmission Control Protocol/Internet Protocol）即传输控制协议/网间协议，定义了主机如何连入因特网及数据如何再它们之间传输的标准，

从字面意思来看TCP/IP是TCP和IP协议的合称，但实际上TCP/IP协议是指因特网整个TCP/IP协议族。不同于ISO模型的七个分层，TCP/IP协议参考模型把所有的TCP/IP系列协议归类到四个抽象层中

应用层：TFTP，HTTP，SNMP，FTP，SMTP，DNS，Telnet 等等

传输层：TCP，UDP

网络层：IP，ICMP，OSPF，EIGRP，IGMP

数据链路层：SLIP，CSLIP，PPP，MTU

每一抽象层建立在低一层提供的服务上，并且为高一层提供服务，看起来大概是这样子的

    我们可以利用ip地址＋协议＋端口号唯一标示网络中的一个进程。能够唯一标示网络中的进程后，它们就可以利用socket进行通信了，我们经常把socket翻译为套接字，socket是在应用层和传输层(TCP/IP协议族通信)之间的一个抽象层，是一组接口，它把TCP/IP层复杂的操作抽象为几个简单的接口供应用层调用已实现进程在网络中通信。

    应用程序两端通过“套接字”向网络发出请求或者应答网络请求。可以把socket理解为通信的把手（hand）

    socket起源于UNIX，在Unix一切皆文件哲学的思想下，socket是一种"打开—读/写—关闭"模式的实现，服务器和客户端各自维护一个"文件"，在建立连接打开后，可以向自己文件写入内容供对方读取或者读取对方内容，通讯结束时关闭文件。socket的英文原义是“插槽”或“插座”，就像我们家里座机一样，如果没有网线的那个插口，电话是无法通信的。Socket是实现TCP，UDP协议的接口，便于使用TCP,UDP。

socket通信流程

流程：

1 服务器根据地址类型（ipv4,ipv6）、socket类型、协议创建socket
2 服务器为socket绑定ip地址和端口号
3 服务器socket监听端口号请求，随时准备接收客户端发来的连接，这时候服务器的socket并没有被打开
4 客户端创建socket
5 客户端打开socket，根据服务器ip地址和端口号试图连接服务器socket
6 服务器socket接收到客户端socket请求，被动打开，开始接收客户端请求，直到客户端返回连接信息。这时候socket进入阻塞状态，所谓阻塞即accept()方法一直等到客户端返回连接信息后才返回，开始接收下一个客户端连接请求
7 客户端连接成功，向服务器发送连接状态信息
8 服务器accept方法返回，连接成功
9 客户端向socket写入信息(或服务端向socket写入信息)
10 服务器读取信息(客户端读取信息)
11 客户端关闭
12 服务器端关闭 

相关方法及参数

sk.bind(address)
　　#s.bind(address) 将套接字绑定到地址。address地址的格式取决于地址族。在AF_INET下，以元组（host,port）的形式表示地址。

sk.listen(backlog)
　　#开始监听传入连接。backlog指定在拒绝连接之前，可以挂起的最大连接数量。
      #backlog等于5，表示内核已经接到了连接请求，但服务器还没有调用accept进行处理的连接个数最大为5
      #这个值不能无限大，因为要在内核中维护连接队列

sk.setblocking(bool)
　　#是否阻塞（默认True），如果设置False，那么accept和recv时一旦无数据，则报错。

sk.accept()
　　#接受连接并返回（conn,address）,其中conn是新的套接字对象，可以用来接收和发送数据。address是连接客户端的地址。
　　#接收TCP 客户的连接（阻塞式）等待连接的到来

sk.connect(address)
　　#连接到address处的套接字。一般，address的格式为元组（hostname,port）,如果连接出错，返回socket.error错误。

sk.connect_ex(address)
　　#同上，只不过会有返回值，连接成功时返回 0 ，连接失败时候返回编码，例如：10061

sk.close()
　　#关闭套接字

sk.recv(bufsize[,flag])
　　#接受套接字的数据。数据以字符串形式返回，bufsize指定最多可以接收的数量。flag提供有关消息的其他信息，通常可以忽略。

sk.recvfrom(bufsize[.flag])
　　#与recv()类似，但返回值是（data,address）。其中data是包含接收数据的字符串，address是发送数据的套接字地址。

sk.send(string[,flag])
　　#将string中的数据发送到连接的套接字。返回值是要发送的字节数量，该数量可能小于string的字节大小。即：可能未将指定内容全部发送。

sk.sendall(string[,flag])
　　#将string中的数据发送到连接的套接字，但在返回之前会尝试发送所有数据。成功返回None，失败则抛出异常。
      #内部通过递归调用send，将所有内容发送出去。

sk.sendto(string[,flag],address)
　　#将数据发送到套接字，address是形式为（ipaddr，port）的元组，指定远程地址。返回值是发送的字节数。该函数主要用于UDP协议。

sk.settimeout(timeout)
　　#设置套接字操作的超时期，timeout是一个浮点数，单位是秒。值为None表示没有超时期。一般，超时期应该在刚创建套接字时设置，因为它们可能用于连接的操作
　　（如 client 连接最多等待5s ）

sk.getpeername()
　　#返回连接套接字的远程地址。返回值通常是元组（ipaddr,port）。

sk.getsockname()
　　#返回套接字自己的地址。通常是一个元组(ipaddr,port)

sk.fileno()
　　#套接字的文件描述符

实例：第一段 建立连接

# 第一段建立连接
###############################服务端#################################
# import socket
# ip_port = ('127.0.0.1',8766)  #建立服务端的Ip和端口号
# sk = socket.socket()  #创建socket
# sk.bind(ip_port)    #绑定IP和端口号
# sk.listen(6)   #监听端口端的等待链接数量
# print('服务端已开启。。。')
#
# # 建立连接
# conn,addr = sk.accept()   #accept阻塞，等待连接
# client_data =conn.recv(1024)   #接收字段，以字符串形式接收
# print(str(client_data,'UTF8'))   #打印读取数据，接收的时Byte要转化为字符串读取
# conn.sendall(input(),encoding='utf8')   #发送数据
#
# sk.close()

##############################客户端######################################

# 第一段建立连接
# import  socket
#
# ip_port = ('127.0.0.1',8866)  #连接端IP和端口号
# sk = socket.socket()  #创建socket
# sk.connect(ip_port)  #连接socket
# print('客户端开启。。。')
#
# sk.sendall(input,'utf8')   #发出数据
# sever_reply = sk.recv(1024)  #读取数据
# print(sever_reply,'utf8')  #打印数据

第二段  正确关闭

# 第二段 正确关闭连接
###############服务端####################
# import socket
# ip_port = ('127.0.0.1',8766)  #建立服务端的Ip和端口号
# sk = socket.socket()  #创建socket
# sk.bind(ip_port)    #绑定IP和端口号
# sk.listen(6)   #监听端口端的等待链接数量
# print('服务端已开启。。。')
#
# # 建立连接
# conn,addr = sk.accept()   #accept阻塞，等待连接
# while True:
#         client_data = conn.recv(1024)   #接收字段，以字符串形式接收
#         if str(client_data,'utf8') == 'exit':
#             break
#         print(str(client_data,'utf8'))   #打印读取数据，接收的时Byte要转化为字符串读取
#         inp = input('请输入：')
#         conn.sendall(bytes(inp,'utf8'))   #发送数据
# conn.close()

#################客户端##################
# import  socket
#
# ip_port = ('127.0.0.1',8766)  #连接端IP和端口号
# sk = socket.socket()  #创建socket
# sk.connect(ip_port)  #连接socket
# print('客户端开启。。。')
# while True:
#         inp = input()
#         sk.sendall(bytes(inp,'utf8'))   #发出数据
#         if inp == 'exit':
#             break
#         sever_reply = sk.recv(1024)  #读取数据
#         print(str(sever_reply,'utf8'))  #打印数据
# sk.close()

第三段 关闭一个连接排队连接接入(并发)

###################服务端################## import socket
ip_port = ('127.0.0.1',8766)  #建立服务端的Ip和端口号
sk = socket.socket()  #创建socket
sk.bind(ip_port)    #绑定IP和端口号
sk.listen(6)   #监听端口端的等待链接数量
print('服务端已开启。。。')

# 建立连接
while True:
    conn,addr = sk.accept()   #accept阻塞，等待连接
    while True:
            client_data = conn.recv(1024)   #接收字段，以字符串形式接收
            if str(client_data,'utf8') == 'exit':
                break
            print(str(client_data,'utf8'))   #打印读取数据，接收的时Byte要转化为字符串读取
            inp = input('请输入：')
            conn.sendall(bytes(inp,'utf8'))   #发送数据
    conn.close()

################客户端###############
import  socket
ip_port = ('127.0.0.1',8766)  #连接端IP和端口号
sk = socket.socket()  #创建socket
sk.connect(ip_port)  #连接socket
print('客户端开启。。。')
while True:
        inp = input()
        sk.sendall(bytes(inp,'utf8'))   #发出数据
        if inp == 'exit':
            break
        sever_reply = sk.recv(1024)  #读取数据
        print(str(sever_reply,'utf8'))  #打印数据
sk.close()

实例二：命令传送

###################服务端#################
import  socket
import  subprocess

Ip_port =('127.0.0.1',8989)
sk = socket.socket()
sk.bind(Ip_port)
sk.listen(5)
print('开启服务端。。。')
while True:
    conn,addr = sk.accept()
    while True:
        try:
            client_data = conn.recv(1024)
        except Exception as e:
            print(e)
            break
        print(str(client_data,'utf8'))
        # server_data = input("请输入:")
        # conn.sendall(bytes(server_data,'utf8'))
        cmd = str(client_data,'utf8').strip()
        cmd_call = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
        cmd_result = cmd_call.stdout.read()
        # if len(cmd_result) == 0:
        #     cmd_result = '读取完毕'
        conn.sendall(cmd_result)
    conn.close()

######################客户端###############
import socket
ip_port = ('127.0.0.1',8989)
sk = socket.socket()
sk.connect(ip_port)
print('客户端启动')

while True:
    inp = input('cmd:').strip()
    if len(inp) == 0:
        continue
    if inp == 'exit':
        break
    sk.sendall(bytes(inp,'utf8'))
    client_data = sk.recv(1024)
    print(str(client_data,'gbk'))
    # if inp =='quit':
    #    break
sk.close()

import socketserver
import subprocess

class Myserver(socketserver.BaseRequestHandler):
    def handle(self):
        while True:
            conn=self.request
            conn.sendall(bytes("欢迎登录","utf8"))
            while True:
                client_bytes=conn.recv(1024)
                if not client_bytes:break
                client_str=str(client_bytes,"utf8")
                print(client_str)
                command=client_str

                result_str=subprocess.getoutput(command)
                result_bytes = bytes(result_str,encoding='utf8')
                info_str="info|%d"%len(result_bytes)
                conn.sendall(bytes(info_str,"utf8"))
                # conn.recv(1024)
                conn.sendall(result_bytes)
            conn.close()

if __name__=="__main__":
    server=socketserver.ThreadingTCPServer(("127.0.0.1",9998),Myserver)
    server.serve_forever()

#####################################client


import socket
ip_port=("127.0.0.1",9998)

sk=socket.socket()
sk.connect(ip_port)
print("客户端启动...")

print(str(sk.recv(1024),"utf8"))

while True:
    inp=input("please input:").strip()


    sk.sendall(bytes(inp,"utf8"))
    basic_info_bytes=sk.recv(1024)
    print(str(basic_info_bytes,"utf8"))
    # sk.send(bytes('ok','utf8'))
    result_length=int(str(basic_info_bytes,"utf8").split("|")[1])

    print(result_length)
    has_received=0
    content_bytes=bytes()
    while has_received<result_length:
        fetch_bytes=sk.recv(1024)
        has_received+=len(fetch_bytes)
        content_bytes+=fetch_bytes
    cmd_result=str(content_bytes,"utf8")
    print(cmd_result)

sk.close()

**********************服务端************************
import socket
import subprocess
ip_port = ('127.0.0.1', 8899)
sk = socket.socket()
sk.bind(ip_port)
sk.listen(5)
print('服务端开启。。。。')

# 建立链接
while True:
    conn, addr = sk.accept()
    while True:
        try:
            Recv_server = str(conn.recv(1024), 'utf8')
        except Exception as e:
            print('错误')
            break
        cmd = Recv_server.strip()
        if Recv_server == 'exit':
            break
        cmd_all = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
        cmd_result = cmd_all.stdout.read()  # 读取cmd结果,打印是以bytes输出
        len_cmd_result = len(cmd_result)  # 打印cmd的bytes长度，
        conn.send(bytes(str(len_cmd_result),'utf8'))  # 发送长度

        conn.recv(1024)  # 解决粘包

        conn.send(cmd_result)  # 打印输出
    conn.close()

*****************客户端***************

import  socket

ip_port = ('127.0.0.1',8899)
sk = socket.socket()
sk.connect(ip_port)
print('客户端已经开启。。。')

while True:
    inp = input('cmd:').strip()
    print(inp)
    if len(inp) == 0:
        continue
    if inp == 'exit':
        sk.sendall(bytes(inp, 'utf8'))
        break
    sk.sendall(bytes(inp, 'utf8'))
    data_client_len = int(str(sk.recv(1024),'utf8'))

    sk.send(bytes('解决粘包添加', 'utf8'))  # 解决粘包，添加一个收发

    data_len = 0
    data = bytes()  # 接收文本的初始化
    while data_client_len != data_len:
            data_client = sk.recv(50)
            data_len += len(data_client) # 接收bytes长度增加
            data += data_client # 输入文件，源文件增加
            print(data_len)   # 打印字节的长度
    print(str(data, 'GBK'))  # 打印接收文件
    print(len(str(data, 'GBK')))  # 打印str格式下的文件

sk.close()

实例三：上传文件

import socket,os
ip_port=("127.0.0.1",8898)
sk=socket.socket()
sk.bind(ip_port)
sk.listen(5)
BASE_DIR=os.path.dirname(os.path.abspath(__file__))

while True:
    print("waiting connect")
    conn,addr=sk.accept()
    flag = True
    while flag:

            client_bytes=conn.recv(1024)
            client_str=str(client_bytes,"utf8")
            func,file_byte_size,filename=client_str.split("|",2)  # 接收文件信息获取，文件方式|文件大小|文件名

            path=os.path.join(BASE_DIR,'yuan',filename)  # 获取接收文件存放位置
            has_received=0
            file_byte_size=int(file_byte_size)
            # 获取文件，写入文件
            f=open(path,"wb")
            while has_received<file_byte_size:
                data=conn.recv(1024)
                f.write(data)
                has_received+=len(data)
            print("ending")
            f.close()

#----------------------------------------------client
#----------------------------------------------
import socket
import re,os,sys
ip_port=("127.0.0.1",8898)
sk=socket.socket()
sk.connect(ip_port)
BASE_DIR=os.path.dirname(os.path.abspath(__file__))
print("客户端启动....")

while True:
    inp=input("please input:")    # post|文件名

    if inp.startswith("post"):    # 判断是否以post开头
        method,local_path=inp.split("|",1)  # 切片出 method=post  local_path=文件名
        local_path=os.path.join(BASE_DIR,local_path)  # 拼接目录路径和文件名，获取绝对路径
        file_byte_size=os.stat(local_path).st_size  # 获取文件大小
        file_name=os.path.basename(local_path)  # 获取名
        post_info="post|%s|%s"%(file_byte_size,file_name)  # 文件信息 post|文件大小|文件名
        sk.sendall(bytes(post_info,"utf8"))  # 发送文件信息（注意接收到文件信息，服务端也做了切片）
        #  循环发送
        has_sent=0
        file_obj=open(local_path,"rb")
        while has_sent<file_byte_size:
            data=file_obj.read(1024)
            sk.sendall(data)
            has_sent+=len(data)
        file_obj.close()
        print("上传成功")

二、 Socketserver

socketserver模块可以简化网络服务器的编写，Python把网络服务抽象成两个主要的类，一个是Server类，用于处理连接相关的网络操作，另外一个则是RequestHandler类，用于处理数据相关
的操作。并且提供两个MixIn 类，用于扩展 Server，实现多进程或多线程。

Server类

它包含了种五种server类，BaseServer(不直接对外服务)。TCPServer使用TCP协议，UDPServer使用UDP协议，还有两个不常使用的，即UnixStreamServer和UnixDatagramServer,这两个类仅仅在unix环境下有用(AF_unix)。

class BaseServer
This uses the Internet TCP protocol, which provides for continuous streams of data between the client and server. 

class socketserver.TCPServer(server_address, RequestHandlerClass, bind_and_activate=True)
This uses datagrams, which are discrete packets of information that may arrive out of order or be lost while in transit. The parameters are 
the same as for TCPServer

class socketserver.UDPServer(server_address, RequestHandlerClass, bind_and_activate=True)
These more infrequently used classes are similar to the TCP and UDP classes, but use Unix domain sockets; they’re not available on non-Unix 
platforms. The parameters are the same as for TCPServer.　

class socketserver.UnixStreamServer(server_address, RequestHandlerClass, bind_and_activate=True)
class socketserver.UnixDatagramServer(server_address, RequestHandlerClass,bind_and_activate=True)

class UnixStreamServer(TCPServer):
    address_family = socket.AF_UNIX

class UnixDatagramServer(UDPServer):
    address_family = socket.AF_UNIX

BaseServer的源码：

class BaseServer:

    """Base class for server classes.

    Methods for the caller:

    - __init__(server_address, RequestHandlerClass)
    - serve_forever(poll_interval=0.5)
    - shutdown()
    - handle_request()  # if you do not use serve_forever()
    - fileno() -> int   # for select()

    Methods that may be overridden:

    - server_bind()
    - server_activate()
    - get_request() -> request, client_address
    - handle_timeout()
    - verify_request(request, client_address)
    - server_close()
    - process_request(request, client_address)
    - shutdown_request(request)
    - close_request(request)
    - service_actions()
    - handle_error()

    Methods for derived classes:

    - finish_request(request, client_address)

    Class variables that may be overridden by derived classes or
    instances:

    - timeout
    - address_family
    - socket_type
    - allow_reuse_address

    Instance variables:

    - RequestHandlerClass
    - socket

    """

    timeout = None

    def __init__(self, server_address, RequestHandlerClass):
        """Constructor.  May be extended, do not override."""
        self.server_address = server_address
        self.RequestHandlerClass = RequestHandlerClass
        self.__is_shut_down = threading.Event()
        self.__shutdown_request = False

    def server_activate(self):
        """Called by constructor to activate the server.

        May be overridden.

        """
        pass

    def serve_forever(self, poll_interval=0.5):
        """Handle one request at a time until shutdown.

        Polls for shutdown every poll_interval seconds. Ignores
        self.timeout. If you need to do periodic tasks, do them in
        another thread.
        """
        self.__is_shut_down.clear()
        try:
            while not self.__shutdown_request:
                # XXX: Consider using another file descriptor or
                # connecting to the socket to wake this up instead of
                # polling. Polling reduces our responsiveness to a
                # shutdown request and wastes cpu at all other times.
                r, w, e = _eintr_retry(select.select, [self], [], [],
                                       poll_interval)
                if self in r:
                    self._handle_request_noblock()

                self.service_actions()
        finally:
            self.__shutdown_request = False
            self.__is_shut_down.set()

    def shutdown(self):
        """Stops the serve_forever loop.

        Blocks until the loop has finished. This must be called while
        serve_forever() is running in another thread, or it will
        deadlock.
        """
        self.__shutdown_request = True
        self.__is_shut_down.wait()

    def service_actions(self):
        """Called by the serve_forever() loop.

        May be overridden by a subclass / Mixin to implement any code that
        needs to be run during the loop.
        """
        pass

    # The distinction between handling, getting, processing and
    # finishing a request is fairly arbitrary.  Remember:
    #
    # - handle_request() is the top-level call.  It calls
    #   select, get_request(), verify_request() and process_request()
    # - get_request() is different for stream or datagram sockets
    # - process_request() is the place that may fork a new process
    #   or create a new thread to finish the request
    # - finish_request() instantiates the request handler class;
    #   this constructor will handle the request all by itself

    def handle_request(self):
        """Handle one request, possibly blocking.

        Respects self.timeout.
        """
        # Support people who used socket.settimeout() to escape
        # handle_request before self.timeout was available.
        timeout = self.socket.gettimeout()
        if timeout is None:
            timeout = self.timeout
        elif self.timeout is not None:
            timeout = min(timeout, self.timeout)
        fd_sets = _eintr_retry(select.select, [self], [], [], timeout)
        if not fd_sets[0]:
            self.handle_timeout()
            return
        self._handle_request_noblock()

    def _handle_request_noblock(self):
        """Handle one request, without blocking.

        I assume that select.select has returned that the socket is
        readable before this function was called, so there should be
        no risk of blocking in get_request().
        """
        try:
            request, client_address = self.get_request()
        except OSError:
            return
        if self.verify_request(request, client_address):
            try:
                self.process_request(request, client_address)
            except:
                self.handle_error(request, client_address)
                self.shutdown_request(request)

    def handle_timeout(self):
        """Called if no new request arrives within self.timeout.

        Overridden by ForkingMixIn.
        """
        pass

    def verify_request(self, request, client_address):
        """Verify the request.  May be overridden.

        Return True if we should proceed with this request.

        """
        return True

    def process_request(self, request, client_address):
        """Call finish_request.

        Overridden by ForkingMixIn and ThreadingMixIn.

        """
        self.finish_request(request, client_address)
        self.shutdown_request(request)

    def server_close(self):
        """Called to clean-up the server.

        May be overridden.

        """
        pass

    def finish_request(self, request, client_address):
        """Finish one request by instantiating RequestHandlerClass."""
        self.RequestHandlerClass(request, client_address, self)

    def shutdown_request(self, request):
        """Called to shutdown and close an individual request."""
        self.close_request(request)

    def close_request(self, request):
        """Called to clean up an individual request."""
        pass

    def handle_error(self, request, client_address):
        """Handle an error gracefully.  May be overridden.

        The default is to print a traceback and continue.

        """
        print('-'*40)
        print('Exception happened during processing of request from', end=' ')
        print(client_address)
        import traceback
        traceback.print_exc() # XXX But this goes to stderr!
        print('-'*40)

Base Server类下的继承关系

 RequestHandler类

所有requestHandler都继承BaseRequestHandler基类。

class BaseRequestHandler:

    """Base class for request handler classes.

    This class is instantiated for each request to be handled.  The
    constructor sets the instance variables request, client_address
    and server, and then calls the handle() method.  To implement a
    specific service, all you need to do is to derive a class which
    defines a handle() method.

    The handle() method can find the request as self.request, the
    client address as self.client_address, and the server (in case it
    needs access to per-server information) as self.server.  Since a
    separate instance is created for each request, the handle() method
    can define arbitrary other instance variariables.

    """

    def __init__(self, request, client_address, server):
        self.request = request
        self.client_address = client_address
        self.server = server
        self.setup()
        try:
            self.handle()
        finally:
            self.finish()

    def setup(self):
        pass

    def handle(self):
        pass

    def finish(self):
        pass

创建一个socketserver 至少分以下几步

1. First, you must create a request handler class by subclassing the BaseRequestHandlerclass and overriding its handle() method; this method will process incoming requests. 　　
2. Second, you must instantiate one of the server classes, passing it the server’s address and the request handler class.
3. Then call the handle_request() orserve_forever() method of the server object to process one or many requests.
4. Finally, call server_close() to close the socket.

import socketserver

class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The request handler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print("{} wrote:".format(self.client_address[0]))
        print(self.data)
        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    HOST, PORT = "localhost", 9999

    # Create the server, binding to localhost on port 9999
    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

    # Activate the server; this will keep running until you
    # interrupt the program with Ctrl-C
    server.serve_forever()

Socketserver并发下的类

class socketserver.ForkingTCPServer

class socketserver.ForkingUDPServer

class socketserver.ThreadingTCPServer

class socketserver.ThreadingUDPServer

所以：

 

server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
#替换为
server = socketserver.ThreadingTCPServer((HOST, PORT), MyTCPHandler)