multiprocessing

 

由于Python的GIL,多进程并非CPU密集型程序的better choice

多进程可以完全独立的进程环境中运行程序,可以充分的利用多核心

但进程本身隔离带来数据共享问题,而且线程比进程轻量级

 

multiprocessing

Process类:

import multiprocessing,datetime

def b():
    s=0
    for _ in range(1000000000):
        s+=1

if __name__ == '__main__':
    commence=datetime.datetime.now()
    lst=[]
    for bb in range(5):
        p=multiprocessing.Process(target=b,args=tuple(),name='p-{}'.format(bb))
        p.start()
        lst.append(p)
    for p in lst:
        p.join()

    delta=(datetime.datetime.now()-commence).total_seconds()
    print(delta)

 

 

 

 

 

 

 

multiprocessing.Pool() 进程池

import multiprocessing,logging,datetime
FORMAT='%(asctime)-10s[%(processName)s:%(threadName)s, %(process)s:%(thread)s] %(message)s'
logging.basicConfig(level=logging.WARNING,format=FORMAT)

def c(b):
    s=0
    for _ in range(1000000000):
        s+=1
    logging.error('s: {}, b: {}'.format(s,b))

if __name__ == "__main__":
    commence=datetime.datetime.now()
    pool=multiprocessing.Pool(5)
    for b in range(5):
        pool.apply_async(c,args=(b,))
    pool.close()
    pool.join()
    delta=(datetime.datetime.now()-commence).total_seconds()
    logging.error(delta)

2020-10-29 16:52:44,141[SpawnPoolWorker-3:MainThread, 18536:22440] s: 1000000000, b: 2
2020-10-29 16:52:44,181[SpawnPoolWorker-5:MainThread, 22708:25700] s: 1000000000, b: 4
2020-10-29 16:52:44,365[SpawnPoolWorker-2:MainThread, 24636:21052] s: 1000000000, b: 0
2020-10-29 16:52:44,559[SpawnPoolWorker-4:MainThread, 15624:24500] s: 1000000000, b: 3
2020-10-29 16:52:44,561[SpawnPoolWorker-1:MainThread, 21932:18940] s: 1000000000, b: 1
2020-10-29 16:52:44,596[MainProcess:MainThread, 15032:18020] 0:01:22.439126

 

 

 

进程间同步

进程间同步提供了和线程同步一样的类,使用的方法一样

进程间代价要高于线程,而且底层实现是不同的,只不过Python屏蔽了底层细节

multiprocessing还提供了共享内存,服务器进程来共享数据,还提供了Queue队列,Pipe管道用于进程间通信

通信方式不同

• 多进程就是启动多个特解释器进程,进程间通信必须序列化,反序列化
• 数据的线程安全性问题
  由于每个进程中没有实现多线程,GIL可以忽略