Python学习-day10
一、多进程multiprocessing
multiprocessing包是Python中的多进程管理包。与threading.Thread类似，它可以利用multiprocessing.Process对象来创建一个进程。该Process对象与Thread对象的用法相同，也有start(), run(), join()的方法。
1 import multiprocessing,threading 2 import time 3 4 def thread_run(): 5 print(threading.get_ident()) 6 def run(name): 7 time.sleep(2) 8 print('Hello ',name) 9 t = threading.Thread(target=thread_run) 10 t.start() 11 12 13 if __name__== '__main__': 14 for i in range(10): 15 p = multiprocessing.Process(target=run, args=('bob_%s'%i,)) 16 p.start()
运行结果：
1 Hello bob_0 2 1144 3 Hello bob_8 4 1268 5 Hello bob_4 6 4360 7 Hello bob_2 8 768 9 Hello bob_6 10 5308 11 Hello bob_5 12 Hello bob_1 13 6076 14 5088 15 Hello bob_9 16 6104 17 Hello bob_3 18 5196 19 Hello bob_7 20 748
二、进程池
如果要创建多个进程，可以使用进程池，启用进程池需要使用Pool库，使用指令pool=Pool()可自动调用所有CPU，
map()函数相当于一个循环，将参数2中的列表元素逐次灌入参数1的函数中。
1 from multiprocessing import Pool 2 3 def squre(num): 4 return num ** 2 5 6 7 if __name__ == '__main__': 8 numbers = [0,1,2,3,4,5] 9 pool = Pool(processes=5)#进程池中最多能放入5个进程 10 print(pool.map(squre,numbers))
运行结果：
[0, 1, 4, 9, 16, 25]
Pool还有以下常用的方法:
- apply_async(func, args)从进程池中取出一个进程执行func，args为func的参数。它将返回一个AsyncResult的对象，我们可以调用get()方法以获得结果。
- close() 关闭进程池，不再创建新的进程
- join() wait()进程池中的全部进程。但是必须对Pool先调用close()方法才能join.
1 from multiprocessing import Process, Pool,freeze_support 2 import time 3 import os 4 5 def Foo(i): 6 time.sleep(2) 7 print("in process",os.getpid()) 8 return i + 100 9 10 def Bar(arg): 11 print('-->exec done:', arg,os.getpid()) 12 13 if __name__ == '__main__': 14 #freeze_support() 15 pool = Pool(processes=3) 16 print("主进程",os.getpid()) 17 for i in range(10): 18 pool.apply_async(func=Foo, args=(i,), callback=Bar) #callback=回调 19 20 print('end') 21 pool.close() 22 pool.join()
运行结果：
1 主进程 5660 2 end 3 in process 7048 4 -->exec done: 100 5660 5 in process 3396 6 -->exec done: 101 5660 7 in process 6728 8 -->exec done: 102 5660 9 in process 7048 10 -->exec done: 103 5660 11 in process 3396 12 -->exec done: 104 5660 13 in process 6728 14 -->exec done: 105 5660 15 in process 7048 16 -->exec done: 106 5660 17 in process 3396 18 -->exec done: 107 5660 19 in process 6728 20 -->exec done: 108 5660 21 in process 7048 22 -->exec done: 109 5660
除了主进程，其它结果是三个一组执行的，因为进程池中每次最多有三个进程。
三、进程通信
进程间通信常用两种方法：Queue和pipe,Queue可以用在多个进程间实现通信，pipe用在两个进程间通信。
1 import os 2 import multiprocessing 3 import time 4 #================== 5 # input worker 6 def inputQ(queue): 7 info = str(os.getpid()) + '(put):' + str(time.time()) 8 queue.put(info) 9 10 # output worker 11 def outputQ(queue,lock): 12 info = queue.get() 13 lock.acquire() 14 print (str(os.getpid()) + '(get):' + info) 15 lock.release() 16 17 if __name__ == '__main__': 18 record1 = [] # store input processes 19 record2 = [] # store output processes 20 lock = multiprocessing.Lock() # To prevent messy print 21 queue = multiprocessing.Queue(3) 22 23 24 for i in range(10): 25 process = multiprocessing.Process(target=inputQ, args=(queue,)) 26 process.start() 27 record1.append(process) 28 29 30 for i in range(10): 31 process = multiprocessing.Process(target=outputQ, args=(queue, lock)) 32 process.start() 33 record2.append(process) 34 35 for p in record1: 36 p.join() 37 38 queue.close() 39 40 for p in record2: 41 p.join()
运行结果：
1 4004(get):4556(put):1476337412.4875286 2 512(get):5088(put):1476337412.6345284 3 8828(get):7828(put):1476337412.7965286 4 8372(get):1032(put):1476337412.8185284 5 7740(get):1496(put):1476337412.9205284 6 4176(get):632(put):1476337412.9855285 7 5828(get):8508(put):1476337412.9595284 8 4236(get):9204(put):1476337412.9925284 9 7632(get):8956(put):1476337413.2055285 10 6376(get):4160(put):1476337413.0705285
Pipe可以是单向(half-duplex)，也可以是双向(duplex)。我们通过mutiprocessing.Pipe(duplex=False)创建单向管道 (默认为双向)。一个进程从PIPE一端输入对象，然后被PIPE另一端的进程接收，单向管道只允许管道一端的进程输入，而双向管道则允许从两端输入。
1 from multiprocessing import Process, Pipe 2 3 4 def f(conn): 5 conn.send([42, None, 'hello from child']) 6 conn.send([42, None, 'hello from child2']) 7 print("from parent:",conn.recv()) 8 conn.close() 9 10 if __name__ == '__main__': 11 parent_conn, child_conn = Pipe() 12 p = Process(target=f, args=(child_conn,)) 13 p.start() 14 print(parent_conn.recv()) # prints "[42, None, 'hello']" 15 print(parent_conn.recv()) # prints "[42, None, 'hello']" 16 parent_conn.send("chupi可好") # prints "[42, None, 'hello']" 17 p.join()
运行结果：
1 [42, None, 'hello from child'] 2 [42, None, 'hello from child2'] 3 from parent: chupi可好
