python并行多个线程和进程

python并行多个线程和进程

工作站配置了20核CPU，平时运行程序只让一个CPU进行运转，浪费了很多时间。下面介绍同时启动多个CPU运行一个程序的方法：

一个进程（CPU）包含多个线程，线程并行的python库为threading，进程并行的库为multiprocessing。

父进程（主函数）运行结束后，如果子进程（子函数）还没有运行结束，需要使用join方法让父进程等待全部结束后再结束。

下面程序中，父进程每隔30s检查子进程的存留情况，等全部子进程运行结束后通过join结束整个程序。

并行多个线程的例子

from threading import Thread #并行线程
from threading import active_count
from time import time, sleep

def process(times,data):
    print("now is",times,"th process")
    sleep(15)
    f   = open(times,"w")
    f.write(data)
    f.close()
    print("OK!",times,"is finished")

if __name__ == '__main__':
    tstart    = time()
    threads   = []
    for i in range(5): #一个进程里并行5个线程
        name='testdata{:02d}.txt'.format(i)
        na  ='{:6.4f}'.format(i)
        print(name)
        t = Thread(target=process,args=(name,na))
        threads.append(t)
        t.setDaemon(True)
        t.start()
        sleep(2)
    th_no = active_count()
    while (th_no>1): #主线程监视间隔30s
        sleep(30)
        th_no = active_count()
        trun = time()
        print("now there are",th_no,"threads, time=",trun-tstart,"sec")
    for t in threads:
        t.join()
    tend      = time()
    print("sum up cost",tend-tstart,"sec")

 

并行多个进程的例子

CPU占用较低的小程序，并行多线程效率更高，CPU占用100%的大程序，并行多个进程效率更高，并行多个线程反而会使效率降低。

from multiprocessing import Process #不同点1
from multiprocessing import active_children #不同点2
from time import time, sleep

def process(times,data):
    print("now is",times,"th process")
    sleep(15)
    f   = open(times,"w")
    f.write(data)
    f.close()
    print("OK!",times,"is finished")

if __name__ == '__main__':
    tstart    = time()
    threads   = []
    for i in range(10): #数量要小于CPU核数
        name='testdata{:02d}.txt'.format(i)
        na  ='{:6.4f}'.format(i)
        print(name)
        t = Process(target=process,args=(name,na)) #与threading.Thread使用方法相似
        threads.append(t)
        t.start()
        sleep(1.0)
    pro_no = len(active_children()) #每隔30s父进程监视子进程留存数量
    while (pro_no>1):
        sleep(30)
        pro_no = len(active_children())
        trun     = time()
        print("Running CPU NO=",pro_no," Running time=",trun-tstart,"sec")
    for t in threads: #等待子进程全部结束开始join
        t.join()
    tend      = time()
    print("sum up cost",tend-tstart,"sec")

参考官方文档：

并行多个进程 https://docs.python.org/zh-cn/3/library/multiprocessing.html

并行多个线程 https://docs.python.org/zh-cn/3/library/threading.html