Python基础及语法（十二）

目录

• 模块化
  • • 自定义模块
    • 导入语句
    • if __name__ == '__main__':的用途
• 包管理
  • • 主要工具
    • 使用setuptools模块打包
• logging模块
  • • 日志级别
• 打印树和堆排序
• 并发
  • • 并发和并行的区别
    • 并发的解决
• 进程和线程
  • • 线程的状态
• threading模块
  • • threading.local类
• queue模块
  • • Queque先进先出队列
    • LifoQueue后进先出队列
    • PriorityQueue小顶堆排序队列
• 线程同步
  • • Event标记
    • Lock锁
    • RLock可重入锁
    • Condition状况
    • Semaphore信号量
• 多进程multiprocessing模块
  • • Process进程
    • Pool进程池
• concurrent.futures模块
  • • ThreadPoolExecutor对象
    • ProcessPoolExecutor对象
• 网络编程
  • • TCP编程
    • UDP编程
    • socketserver模块
      • • 总结
• IO多路复用
  • • selectors模块

模块化

一般来说，编程语言中，库，包，模块是同一种概念，是代码的组织形式。
Python中只有一种模块对象类型，但是为了模块化组织便利，提供了“包”的概念。
模块module：指Python的源代码文件。
包package：指模块组织在一起的和包同名目录下的相关文件和文件夹。

自定义模块

在路径下新建以下文件
t
|-- __init__.py
|-- t2.py
|-- t1
  |-- __init__.py

导入语句

import sys as s  # import完全导入， as别名
import t.t2
from t import t1  # from部分导入
print(s.path)  # Python会在sys.path下的目录按顺序查找模块
print(t.t2)  # <module 't.t2' from '.\\t\\t2.py'>
print(t1)  # <module 't.t1' from '.\\t\\t1\\__init__.py'>

if __name__ == '__main__':的用途

1. 本模块的功能测试
2. 避免主模块的变更的副作用

包管理

Python的模块或者源文件直接可以复制到目标项目中，就可以使用了。
但是为了更多项目调用使用，或者共享给别人，就需要打包或者发布到网络。
Pypi，公共的模块存储中心，https://pypi.org/

主要工具

主要工具有distutils,setuptools,pip,wheel

使用setuptools模块打包

详情可在官网查询:https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files

from setuptools import setup, find_packages
setup(
    name="HelloWorld",
    version="0.1",
    packages=find_packages(),
    scripts=["say_hello.py"],

    # Project uses reStructuredText, so ensure that the docutils get
    # installed or upgraded on the target machine
    install_requires=["docutils>=0.3"],

    package_data={
        # If any package contains *.txt or *.rst files, include them:
        "": ["*.txt", "*.rst"],
        # And include any *.msg files found in the "hello" package, too:
        "hello": ["*.msg"],
    },

    # metadata to display on PyPI
    author="Me",
    author_email="me@example.com",
    description="This is an Example Package",
    keywords="hello world example examples",
    url="http://example.com/HelloWorld/",   # project home page, if any
    project_urls={
        "Bug Tracker": "https://bugs.example.com/HelloWorld/",
        "Documentation": "https://docs.example.com/HelloWorld/",
        "Source Code": "https://code.example.com/HelloWorld/",
    },
    classifiers=[
        "License :: OSI Approved :: Python Software Foundation License"
    ]

    # could also include long_description, download_url, etc.
)

最低限度设置

from setuptools import setup, find_packages
setup(
    name="HelloWorld",  # 设置包名
    version="0.1",  # 设置版本号
    packages=find_packages(),  # 自动搜索包文件，可用列表自定义
)

把代码另存为，通常命名为setup.py
然后可以在命令行界面操作打包
setup.py --help # 帮助查询
setup.py build # 创建一个build目录把需打包的模块复制一份进去
setup.py install # build后安装包
setup.py sdist # 创建一个dist目录，创建一个压缩包，可供pip或setup.py安装

# 二进制分发包
# 制作windows系统安装包
setup.py bdist_wininst
setup.py bdist_msi
setup.py bdist --format=msi
# rpm包
setup.py bdist_rpm
setup.py bdist --format=rpm
# 压缩文件
setup.py bdist --format=zip
setup.py bdist --format=gztar
# wheel包，依赖wheel模块
setup.py bdist --format=egg
setup.py bdist --format=wheel

logging模块

详情可在官网查询:https://docs.python.org/3/library/logging.html

import logging
FORMAT = '%(add1)s\t%(asctime)s\t%(funcName)s\t%(levelno)s\t%(lineno)s\t%(module)s\t%(process)s\t%(thread)s\t' \
         '%(processName)s\t%(threadName)s\t%(name)s\t%(message)s'  # 设置文本格式
d = {'add1': 'HI'}  # 自定义文本
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=0)  # 设置基础配置，时间格式
log1 = logging.getLogger('s1')  # 初始化名称
log2 = logging.getLogger('s1.s2')
log3 = logging.getLogger('s1.s2.s3')
hander = logging.FileHandler('logging.txt', 'w')  # 初始化一个hander类，设置日志保存路径和读取方式
hander.setFormatter(logging.Formatter(FORMAT))  # 设置格式
hander.setLevel(50)  # 设置等级
log2.setLevel(40)
log1.addHandler(hander)  # 增加hander方法，默认hander标准输出
log2.addHandler(hander)
log3.addHandler(hander)
filter = logging.Filter('s1.s2')  # 设置过滤名称
log2.addFilter(filter)  # 设置过滤器
log3.debug('log3.debug', extra=d)  # 日志
log1.info('log1.info', extra=d)
log2.warning('log2.warning', extra=d)
log3.warning('log3.warning', extra=d)
log1.error('log1.error', extra=d)
log2.critical('log2.critical', extra=d)
log1.debug('log1.debug', extra=d)
log3.info('log1.info', extra=d)
log1.warning('log3.warning', extra=d)
log3.error('log3.error', extra=d)
print(log1.getEffectiveLevel(), log2.getEffectiveLevel(), log3.getEffectiveLevel())  # 0 40 40

日志级别

日志级别	数值
critical	50
error	40
warning	30,默认级别
info	20
debug	10
notset	0

打印树和堆排序

import math
import random


def print_tree(ls):
    origin_len = len(ls)
    size = len(str(max(ls)))
    line = math.ceil(math.log(origin_len, 2))
    index = 0
    for i in range(line):
        for j in range(2 ** i):
            space = ' ' * size * (2 ** (line-i-1) - 1)
            print('{}{:{}}{}'.format(space, ls[index], size * 2, space), end='')
            index += 1
            if index == origin_len:
                break
        print('')
    print('-' * 60)


def big_endian(arr, start, end):
    root = start
    child = root * 2 + 1  # 左孩子
    while child <= end:
        # 孩子比最后一个节点还大，也就意味着最后一个叶子节点了，就得跳出去一次循环，已经调整完毕
        if child + 1 <= end and arr[child] < arr[child + 1]:
            # 为了始终让其跟子元素的较大值比较，如果右边大就左换右，左边大的话就默认
            child += 1
        if arr[root] < arr[child]:
            # 父节点小于子节点直接交换位置，同时坐标也得换，这样下次循环可以准确判断：是否为最底层，
            # 是不是调整完毕
            arr[root], arr[child] = arr[child], arr[root]
            root = child
            child = root * 2 + 1
        else:
            break


def heap_sort(arr):  # 无序区大根堆排序
    first = len(arr) // 2 - 1
    for start in range(first, -1, -1):
        # 从下到上，从左到右对每个节点进行调整，循环得到非叶子节点
        big_endian(arr, start, len(arr) - 1)  # 去调整所有的节点
    arr[0], arr[-1] = arr[-1], arr[0]
    for end in range(len(arr) - 1, 0, -1):
        big_endian(arr, 0, end - 1)  # 重新调整子节点的顺序，从顶开始调整
        arr[0], arr[end] = arr[end], arr[0]  # 顶部尾部互换位置
    print_tree(arr)
    return arr


if __name__ == "__main__":
    data = [random.randint(1, 100) for i in range(25)]
    print(data)
    print_tree(data)
    print(heap_sort(data))

并发

并发和并行的区别

并行：同时做某些事情，可以互不干扰的同时做多件事情
并发：也是同时做某些事情，但是强调一个时段内有事情要处理

并发的解决

1. 队列、缓冲区：有顺序的排成队列，这个队列就是缓冲区，如queue模块的Queue、LifoQueue、PriorityQueue
2. 争抢:设置锁，抢到了就上锁，用完后解锁让下一个抢得再上锁
3. 预处理：使用多的数据先预处理后保存下来，当有需要调用时可直接得到结果，使用少的数据有需要的时候再作处理
4. 并行：开多进程、多线程，同时做多件事情
5. 提速：提升硬件性能
6. 消息中间件：云服务，如阿里云

进程和线程

进程：是计算机中的程序关于某数据集合上的一次运行活动，是系统进行资源分配和调度的基本单位，是操作系统结构的基础
线程：是程序执行流的最小单元，
进程如同一个独立的王国，进程之间不可以随便共享数据，线程如同王国里的一个个城市，同一个进程内的线程可以共享数据

线程的状态

就绪：线程能够运行，在等待被调用，可能刚创建启动，或刚刚从阻塞中恢复，或被其他线程抢占
运行：线程正在运行
阻塞：线程因某些设定被阻塞，无法运行
终止：线程完成，或退出，或被取消

threading模块

import threading
import datetime
import logging


def test2():
    logging.info('{}'.format(threading.current_thread()))


def test(n):
    logging.info('{} {} {}'.format(threading.get_ident(), threading.main_thread(),
                                   threading.current_thread()))  # 返回线程ID号，主线程对象，当前线程对象
    threading.Timer(n, test2).run()


t = datetime.datetime.now()
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
for i in range(5):
    thr = threading.Thread(target=test, args=(i,), name='T{}'.format(i),
                           daemon=True)  # 创建线程, 守护线程daemon为True时主线程结束，子线程消亡，默认设置为当前线程的值，主线程为non-daemon,即daemon=False
    thr.start()  # 启动线程
    print(thr.ident, thr.name, thr.is_alive())  # 返回线程id，名字，是否存活
    if i < 2:
        thr.join()  # 线程等待
    print(thr.is_alive())
# thr.join() 写在循环外或不写实现线程并行
threading.Timer(1, test, args=(0,)).start()  # Timer延时器， run是普通调用不能实现线程并行， start可以
print(threading.enumerate(), threading.active_count())  # 返回活着的线程列表，处于alive状态的线程数
print((datetime.datetime.now()-t).total_seconds())

threading.local类

使用threading.local类设置局部变量，让不同线程不能调用，否则报AttributeError

import threading


class Test:
    def __init__(self):
        self.x = 0
        self.data = threading.local()
        self.data.x = 1000

    def run(self, n):
        print(threading.current_thread(),  self.data, n)  # <Thread(Thread, started)> <_thread._local object> 1000
        try:
            print(self.data.x)
        except AttributeError:
            print('_thread._local\' object has no attribute \'x\'')

    def start(self):
        for self.i in range(3):
            threading.Thread(target=self.run, args=(self.data.x,)).start()


test = Test()
test.start()
print(test.data)  # <_thread._local object>
print(test.data.x)  # 1000

queue模块

Queque先进先出队列

import queue
import threading
import time


class Test:
    def __init__(self):
        self.q = queue.Queue(3)  # 实例化，确定队列容量

    def get(self):
        while True:
            time.sleep(1)
            print(self.q.qsize(), self.q.full, self.q.empty())  # 返回队列元素数量，队列对象，bool队列没有元素返回True，
            print(self.q.get(block=False, timeout=2))  # 弹出元素，默认block=True, timeout=None一直阻塞，

    def put(self):
        for i in range(5):
            self.q.put(i)  # 传入元素

    def start(self):
        threading.Thread(target=self.put).start()
        threading.Thread(target=self.get).start()


test = Test()
test.start()

LifoQueue后进先出队列

import queue
import threading
import time


class Test:
    def __init__(self):
        self.q = queue.LifoQueue(3)  # 实例化，确定队列容量

    def get(self):
        while True:
            time.sleep(1)
            if self.q.empty():  # 队列没有参数退出循环
                break
            print(self.q.get())  # 弹出元素，默认block=True, timeout=None一直阻塞，

    def put(self):
        for i in range(5):
            self.q.put(i)  # 传入元素

    def start(self):
        threading.Thread(target=self.put).start()
        threading.Thread(target=self.get).start()


test = Test()
test.start()

PriorityQueue小顶堆排序队列

import queue
import threading
import time
import random


class Test:
    def __init__(self):
        self.q = queue.PriorityQueue(5)  # 实例化，确定队列容量

    def get(self):
        while True:
            time.sleep(1)
            if self.q.empty():  # 队列没有参数退出循环
                break
            print(self.q.get())  # 弹出元素，默认block=True, timeout=None一直阻塞，

    def put(self):
        ls = list(zip([chr(i) for i in range(97, 102)] * 2, [i for i in range(10)]))
        print(ls)
        random.shuffle(ls)
        print(ls)
        for i in ls:
            self.q.put(i)  # 传入元素

    def start(self):
        threading.Thread(target=self.put).start()
        threading.Thread(target=self.get).start()


test = Test()
test.start()

线程同步

Event标记

import threading
import datetime
import time
import logging


def test2(e):
    while e.wait(3):  # 设置等待时长，默认一直等待
        if e.is_set():  # 判断标记是否为True
            logging.info('{}'.format(threading.current_thread()))
            time.sleep(1)
    print(threading.current_thread(), (datetime.datetime.now() - t).total_seconds())


def test(e):
    n = 0
    while n < 10:
        time.sleep(0.5)
        n += 1
        logging.info(n)
        if n % 3 == 0:
            e.set()  # 标记设置为True
        else:
            e.clear()  # False
    print(threading.current_thread(), (datetime.datetime.now() - t).total_seconds())


t = datetime.datetime.now()
event = threading.Event()  # 标记实例化
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
threading.Thread(target=test, args=(event,)).start()
threading.Thread(target=test2, args=(event,)).start()
print(threading.current_thread(), (datetime.datetime.now()-t).total_seconds())

Lock锁

对比是否枷锁

import threading
import datetime
import time
import logging


def test():
    global n
    for _ in range(10000):
        for j in range(100):
            if j < 50:
                # lock.acquire()  # 锁
                n += 1
                # lock.release()  # 解锁
            else:
                # lock.acquire()  # 锁
                n -= 1
                # lock.release()  # 解锁


t = datetime.datetime.now()
n = 0
lock = threading.Lock()
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)

for i in range(10):
    threading.Thread(target=test).start()

print(threading.current_thread(), (datetime.datetime.now()-t).total_seconds())
while True:
    time.sleep(1)
    if threading.active_count() == 1:
        logging.info('{} {} {}'.format(threading.current_thread(), (datetime.datetime.now() - t).total_seconds(), n))
        break

RLock可重入锁

import threading


def test(l):
    l.acquire(timeout=3)
    l.acquire(True)
    l.acquire(False)
    l.release()
    l.release()
    l.release()
    l.release()
    l.release()


lock = threading.RLock()  # RLock可实现多重入锁
ret = lock.acquire()
lock.acquire(timeout=3)
lock.acquire(True)
lock.acquire(False)
lock.release()
lock.release()
lock.release()
lock.release()
threading.Thread(target=test, args=(lock,))
# lock.release()  # 重入锁，解锁对应加锁次数，不然会报RuntimeError: cannot release un-acquired lock

Condition状况

import threading
import datetime
import time
import logging


def test2(c):
    while True:
        with c:
            c.wait()  # 等待
            logging.info('{}'.format(threading.current_thread()))


def test(c):
    n = 0
    while n < 10:
        time.sleep(0.5)
        cond.acquire()  # 加锁
        n += 1
        cond.release()  # 解锁
        logging.info(n)
        if n % 5 == 0:
            with c:
                c.notify_all()  # 通知全部
        elif n % 9 == 0:
            with c:
                c.notify(2)  # 按顺序通知指定数量个线程
        else:
            with c:
                c.notify(1)
    print(threading.current_thread(), (datetime.datetime.now() - t).total_seconds())


t = datetime.datetime.now()
cond = threading.Condition()  # 条件实例化
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
threading.Thread(target=test2, args=(cond,), daemon=True).start()
threading.Thread(target=test2, args=(cond,), daemon=True).start()
threading.Thread(target=test2, args=(cond,), daemon=True).start()
threading.Thread(target=test, args=(cond,)).start()
print(threading.current_thread(), (datetime.datetime.now()-t).total_seconds())

Semaphore信号量

import threading
import datetime
import time
import logging


def test():
    for _ in range(3):
        s.acquire()  # 信号量-1，当信号量为0时阻塞
        logging.info('msg={} {}'.format((datetime.datetime.now() - t).total_seconds(), s._value))
        time.sleep(1)
        s.release()  # 信号量+1


t = datetime.datetime.now()
s = threading.Semaphore(2)  # 信号量实例化，默认_value = 1
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
for _ in range(3):
    threading.Thread(target=test).start()
logging.info('msg={} {}'.format((datetime.datetime.now() - t).total_seconds(), s._value))

多进程multiprocessing模块

Process进程

import datetime
import threading
import logging
import multiprocessing


def test(ti):
    n = 0
    mat = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
    logging.basicConfig(format=mat, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
    logging.info('msg=start {}'.format((datetime.datetime.now() - ti).total_seconds()))
    ti = datetime.datetime.now()
    while n < 100000000:
        n += 1
    logging.info('msg={} {}'.format((datetime.datetime.now() - ti).total_seconds(), n))


if __name__ == '__main__':
    t = datetime.datetime.now()
    ps = []
    ts = []
    FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
    logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
    for _ in range(3):
        p = multiprocessing.Process(target=test, args=(t,))  # 实例化
        ps.append(p)
        p.start()  # 启动进程
        print(p, p.name, p.exitcode, p.pid)  # 进程，进程名称，进程状态None为启动中，0为进程已退出，进程id
    for _ in range(3):
        thr = threading.Thread(target=test, args=(t,))
        thr.start()
        ts.append(thr)
    for i in ps:
        i.join()  # 主进程阻塞等待
        print(i, i.name, i.exitcode, i.pid)
    for i in ts:
        i.join()
        print(i, i.name)

    logging.info('msg={}'.format((datetime.datetime.now() - t).total_seconds()))
# 测试结果可看出进程的启动速度比线程慢，运算的速度比线程快

Pool进程池

import datetime
import logging
import multiprocessing


def test(ti):
    n = 0
    mat = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
    logging.basicConfig(format=mat, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
    logging.info('msg=start {}'.format((datetime.datetime.now() - ti).total_seconds()))
    ti = datetime.datetime.now()
    while n < 100000000:
        n += 1
    logging.info('msg={} {}'.format((datetime.datetime.now() - ti).total_seconds(), n))
    return n  # callback取得的数据


if __name__ == '__main__':
    t = datetime.datetime.now()
    ps = []
    FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
    logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)
    p = multiprocessing.Pool(2)  # 实例化，设置进程池大小
    for _ in range(3):
        ret = p.apply_async(test, args=(t,), callback=lambda x: logging.info('msg={}'.format(x)))  
        # 异步调用，不阻塞执行，执行后立刻回调，可使用callback，error_callbac参数，执行需要的函数
        # p.apply(test, args=(t,))  # 同步调用。阻塞执行，一个个进程执行
        print(ret.__dict__)
    p.close()  # 关闭池
    p.join()  # 主进程阻塞
    logging.info('msg={}'.format((datetime.datetime.now() - t).total_seconds()))

concurrent.futures模块

池模块

ThreadPoolExecutor对象

线程池

import concurrent.futures
import logging
import datetime
import time
import threading
t = datetime.datetime.now()
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)


def test(n):
    logging.info('msg={}start{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    time.sleep(5)
    logging.info('msg={}finished{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    return n


def test2(n):
    logging.info('msg={}start{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    time.sleep(5)
    logging.info('msg={}finished{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    raise ZeroDivisionError('HI,ZeroDivision')


fs = []
executor = concurrent.futures.ThreadPoolExecutor(2)  # 实例化，定义线程程池大小
for i in range(5):
    future = executor.submit(test, n=i)  # 提交执行函数与参数，返回future类
    print(type(future), future)
    fs.append(future)
ret = concurrent.futures.wait(fs, timeout=None, return_when='FIRST_COMPLETED')
# 生成进程池等待实例，可设置等待时间，等待方式默认等待所有进程完成或被取消,FIRST_COMPLETED任意一个完成或被取消时结束等待，FIRST_EXCEPTION任意一个抛出异常时结束等待
for f in concurrent.futures.as_completed(fs):  # 等待进程池完成一个返回一个
    states = [f.done() for f in fs]  # 返回bool值，代表进程是否执行完成
    print(states)
    print(threading.enumerate())
    print(f, f.done(), f.result())
print(type(ret.done), ret.done)  # 返回done集合
print(type(ret.not_done), ret.not_done)  # 返回not_done集合
results = [f.result() for f in fs]  # 取得返回值，拿不到返回值会抛出异常，会阻塞等待，可设置timeout等待时间
print(results)
future = executor.submit(test2, 4)
print(future.exception())  # 返回异常，没有异常为None，会阻塞等待，可设置timeout等待时间
e = future.exception()
print(type(e), e)
logging.info('msg={}'.format((datetime.datetime.now() - t).total_seconds()))
executor.shutdown()  # 清理进程池，支持上下文管理可以用"with executor:"在前面代替

ProcessPoolExecutor对象

进程池，跟线程池功能几乎一样

import concurrent.futures
import logging
import datetime
import time
import threading
t = datetime.datetime.now()
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)


def test(n):
    logging.info('msg={}start{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    time.sleep(5)
    logging.info('msg={}finished{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    return n


def test2(n):
    logging.info('msg={}start{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    time.sleep(5)
    logging.info('msg={}finished{}'.format(n, (datetime.datetime.now() - t).total_seconds()))
    raise ZeroDivisionError('HI,ZeroDivision')


if __name__ == '__main__':  # 进程操作需要加上主模块识别
    fs = []
    executor = concurrent.futures.ProcessPoolExecutor(2)  # 实例化，定义进程池大小，功能跟线程池基本一样
    with executor:  # 支持上下文管理，省略executor.shutdown()
        for i in range(5):
            future = executor.submit(test, n=i)  # 提交执行函数与参数，返回future类
            print(type(future), future)
            fs.append(future)
        ret = concurrent.futures.wait(fs, timeout=None, return_when='FIRST_COMPLETED')
        # 生成进程池等待实例，可设置等待时间，等待方式默认等待所有进程完成或被取消,FIRST_COMPLETED任意一个完成或被取消时结束等待，FIRST_EXCEPTION任意一个抛出异常时结束等待
        for f in concurrent.futures.as_completed(fs):  # 等待进程池完成一个返回一个
            states = [f.done() for f in fs]  # 返回bool值，代表进程是否执行完成
            print(states)
            print(threading.enumerate())
            print(f, f.done(), f.result())
        print(type(ret.done), ret.done)  # 返回done集合
        print(type(ret.not_done), ret.not_done)  # 返回not_done集合
        results = [f.result() for f in fs]  # 取得返回值，拿不到返回值会抛出异常，会阻塞等待，可设置timeout等待时间
        print(results)
        future = executor.submit(test2, 4)
        print(future.exception())  # 返回异常，没有异常为None，会阻塞等待，可设置timeout等待时间
        e = future.exception()
        print(type(e), e)
        logging.info('msg={}'.format((datetime.datetime.now() - t).total_seconds()))
    # executor.shutdown()  # 清理进程池

网络编程

TCP编程

server端

import socket
import threading
import logging
import time
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)


class Server:

    def __init__(self):
        self.sock = socket.socket()  # 创建socket对象
        # 默认family=AF_INET，IPV4，AF_INET6为IPV6，AF_UNIX为Unix Domain Socket；type=SOCK_STREAM，TCP协议，SOCK_DGRAM为UDP协议
        self.addr = ('127.0.0.1', 9999)
        self.clients = {}
        self.event = threading.Event()
        self.lock = threading.Lock()

    def recv(self, new_sock):
        logging.info('start')
        while not self.event.is_set():
            try:
                data = new_sock.recv(1024)  # 阻塞的，直到接收数据，或异常断开
            except Exception:
                data = 'quit'
            print(data)
            if data == 'quit':
                with self.lock:
                    self.clients.pop(new_sock)
                print('*'*30)
                print(*self.clients, sep='\n')
                print('*' * 30)
                new_sock.close()
                logging.info('finished')
                break
            logging.info('recv msg={}'.format(data))
            for _, sock in self.clients.values():
                sock.send(data)  # 发送数据

    def recv_file(self, new_sock, f):
        logging.info('start')
        while not self.event.is_set():
            try:
                data = f.readline()  # 会阻塞，等待数据读入
            except Exception:
                data = 'quit'
            print(data)
            if data == 'quit':
                with self.lock:
                    self.clients.pop(new_sock)
                print('*'*30)
                print(*self.clients, sep='\n')
                print('*' * 30)
                new_sock.close()
                logging.info('finished')
                break
            logging.info('recv_file msg={}'.format(data))
            c = []
            for f, sock in self.clients.values():
                f.write(data)  # 写入数据
                try:
                    f.flush()  # 发送数据
                except Exception:
                    c.append(sock)
                    sock.close()
            with self.lock:
                for sock in c:
                    self.clients.pop(sock)
                    print('*' * 30)
                    print(*self.clients, sep='\n')
                    print('*' * 30)

    def start(self):
        print('启动服务端')
        self.sock.bind(self.addr)  # 绑定地址
        self.sock.listen()  # 监听
        threading.Thread(target=self.accept, daemon=True).start()

    def accept(self):
        while not self.event.is_set():
            try:
                new_sock, addr = self.sock.accept()  # 会阻塞，返回一个新的socket对象和客户端信息
                f = new_sock.makefile('rwb')  # 创建一个makefile对象,使用二进制读写模式
            except Exception:
                self.stop()
                break
            print(type(new_sock), new_sock)
            print(type(addr), addr)
            print(new_sock.getsockname(), new_sock.getpeername())  # 本地， 对端
            with self.lock:
                self.clients[new_sock] = f, new_sock
            print('*'*30)
            print(*self.clients, sep='\n')
            print('*' * 30)
            threading.Thread(target=self.recv_file, args=(new_sock, f)).start()  # makefile对象方式的读写
            # threading.Thread(target=self.recv, args=(new_sock,)).start()  # 一般使用的读写
            print('=' * 100)

    def stop(self):
        self.event.set()
        for _, sock in self.clients:
            sock.close()  # 关闭
        self.sock.close()  # 结束关闭


s = Server()
s.start()
while True:
    i = input()
    if i == 'quit':
        s.stop()
        threading.Event().wait(3)
        break

client端

import socket
import threading
from datetime import datetime as dt


class Client:

    def __init__(self):
        self.sock = socket.socket()
        self.addr = ('127.0.0.1', 9999)
        self.event = threading.Event()

    def send(self):
        while not self.event.is_set():
            try:
                data = self.sock.recv(1024).decode()
            except Exception:
                self.stop()
                break
            print(data)

    def start(self, name):
        print('启动客户端')
        self.sock.connect(self.addr)  # 客户端链接服务端
        threading.Thread(target=self.send).start()
        while not self.event.is_set():
            i = input()
            if i == 'quit':
                self.stop()
                break
            data = '{}\t{}\t{}\n'.format(name, dt.strftime(dt.now(), '%Y-%m-%d %H:%M:%S'), i)
            self.sock.send(data.encode())

    def stop(self):
        self.sock.close()


c = Client()
c.start('Tom')

UDP编程

server端

import socket
import threading
import time
from datetime import datetime as dt


class Client:
    def __init__(self):
        self.sock = socket.socket(type=socket.SOCK_DGRAM)
        self.addr = ('127.0.0.1', 9999)
        self.event = threading.Event()

    def send(self):
        while not self.event.is_set():
            try:
                data = self.sock.recv(1024).decode()
            except Exception:
                self.stop()
                break
            print(data)

    def heart(self):
        while True:
            time.sleep(3)
            try:
                self.sock.sendto(b'\x01', self.addr)
            except Exception:
                continue

    def start(self, name):
        print('启动客户端')
        self.sock.connect(self.addr)
        threading.Thread(target=self.send).start()
        threading.Thread(target=self.heart, daemon=True).start()
        while not self.event.is_set():
            i = input()
            if i == 'quit':
                self.stop()
                break
            data = '{}\t{}\t{}\n'.format(name, dt.strftime(dt.now(), '%Y-%m-%d %H:%M:%S'), i)
            self.sock.sendto(data.encode(), self.addr)

    def stop(self):
        self.sock.close()


c = Client()
c.start('A君')

client端

import socket
import threading
import time
from datetime import datetime as dt


class Client:
    def __init__(self):
        self.sock = socket.socket(type=socket.SOCK_DGRAM)  # 创建UDP协议得socket对象
        self.addr = ('127.0.0.1', 9999)
        self.event = threading.Event()

    def send(self):
        while not self.event.is_set():
            try:
                data = self.sock.recv(1024).decode()  #  接收数据
            except Exception:
                self.stop()
                break
            print(data)

    def heart(self):
        while True:
            time.sleep(3)
            try:
                self.sock.sendto(b'\x01', self.addr)
            except Exception:
                continue

    def start(self, name):
        print('启动客户端')
        self.sock.connect(self.addr)
        threading.Thread(target=self.send).start()
        threading.Thread(target=self.heart, daemon=True).start()
        while not self.event.is_set():
            i = input()
            if i == 'quit':
                self.stop()
                break
            data = '{}\t{}\t{}\n'.format(name, dt.strftime(dt.now(), '%Y-%m-%d %H:%M:%S'), i)
            self.sock.sendto(data.encode(), self.addr)  # 发送数据与地址

    def stop(self):
        self.sock.close()


c = Client()
c.start('Tom')

socketserver模块

服务端高级模块，在socket模块的基础上对服务端编程进行了一定的封装

import socketserver
import threading
import logging
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)


class Server(socketserver.BaseRequestHandler):  # 基类StreamRequestHandler文件类型操作，相对来说还是BaseRequestHandler封装得好点
    print('启动服务端')
    clients = {}

    def setup(self):  # 启动项
        super().setup()
        self.event = threading.Event()
        self.lock = threading.Lock()
        self.clients[self.client_address] = self.request
        print('*' * 30)
        print(*self.clients, sep='\n')
        print('*' * 30)

    def handle(self):  # 线程项
        super().handle()
        for k, v in self.__dict__.items():
            print(k, v)
        print('-' * 30)
        while not self.event.is_set():
            try:
                data = self.request.recv(1024)
            except Exception:
                data = 'quit'
            print(data)
            if data != 'quit':
                logging.info('recv msg={}'.format(data))
                for sock in self.clients.values():
                    try:
                        sock.send(data)
                    except Exception:
                        print('k', sock)
            else:
                break

    def finish(self):  # 结束项
        self.clients.pop(self.client_address)
        print('*' * 30)
        print(*self.clients, sep='\n')
        print('*' * 30)
        super().finish()
        self.event.set()


addr = ('127.0.0.1', 9999)
server = socketserver.ThreadingTCPServer(addr, Server)
# ThreadingTCPServer为TCP协议,ThreadingUDPServer为UDP协议线程为异步,TCPServer，UDPServer为线程同步
server.daemon_threads = True  # 线程均设为daemon=True
threading.Thread(target=server.serve_forever, daemon=True).start()  # 持续运行
while True:
    i = input()
    if i == 'quit':
        server.server_close()
        break

总结

UDP协议适合大型文件传输，不合适设计群聊，特点传输快，不可靠，无连接
TCP协议为面向链接，几乎可以实现所有事情，特点传输慢，可靠，需要连接

IO多路复用

阻塞，不立即返回
非阻塞，立即返回
异步，不直接得到结果
同步，直接得到结果
区别
同步异步强调是否得到最终结果
阻塞非阻塞强调是否需要等待

selectors模块

Python提供的高级IO复用库

import socket
import threading
import logging
import selectors
FORMAT = '%(asctime)s %(threadName)s  %(thread)s  %(threadName)s %(message)s'
logging.basicConfig(format=FORMAT, datefmt='%Y/%m/%d %I:%M:%S', level=logging.INFO)


class Server:

    def __init__(self):
        self.sock = socket.socket()
        self.addr = ('127.0.0.1', 9999)
        self.clients = {}
        self.event = threading.Event()
        self.lock = threading.Lock()
        self.selector = selectors.DefaultSelector()  # 创建selectors对象

    def recv(self, new_sock):
        logging.info('start')
        try:
            data = new_sock.recv(1024)
        except Exception:
            data = 'quit'
        logging.info('recv msg={}'.format(data))
        if data == 'quit':
            self.selector.unregister(new_sock)
            new_sock.close()
            return  # selectors多路复用使用return结束
        else:
            for key in self.selector.get_map().values():
                if key.data == self.recv:
                    key.fileobj.send(data)

    def start(self):
        print('启动服务端')
        self.sock.bind(self.addr)
        self.sock.listen()
        self.sock.setblocking(False)
        key = self.selector.register(self.sock, selectors.EVENT_READ, self.accept)  # 记录对象，模式，函数
        while not self.event.is_set():
            events = self.selector.select()  # 监听
            for key, mask in events:
                callback = key.data
                callback(key.fileobj)  # 调用

    def accept(self, sock):
        new_sock, addr = sock.accept()
        with self.lock:
            self.clients[new_sock] = new_sock
        new_sock.setblocking(False)
        key = self.selector.register(new_sock, selectors.EVENT_READ, self.recv)


s = Server()
s.start()