进程池中传递实例方法问题
1、多个进程,参数给实例方法,逐次运行时没有问题的,代码如下:
import multiprocessing class MyClass(object): def calc(self,num): print 'the number is ',num if __name__ == '__main__': mc = MyClass() ths = [multiprocessing.Process(target = mc.calc,args = (i,)) for i in range(3)] for th in ths: th.start()
运行结果:
the number is 0 the number is 1 the number is 2
2、放到进程池中一起并发
import multiprocessing class MyClass(object): def calc(self,num): print 'the number is ',num if __name__ == '__main__': mc = MyClass() pool = multiprocessing.Pool(processes = 3) pool.apply_async(mc.calc,(3,)) pool.apply_async(mc.calc,(4,)) pool.apply_async(mc.calc,(5,)) pool.close() pool.join() print 'end!'
运行结果:
end!
只打印了最终结果,传递进去的实例方法压根没有运行!!!
3、解决方法如下:
import multiprocessing import copy_reg import types def _reduce_method(m): if m.im_self is None: return getattr, (m.im_class, m.im_func.func_name) else: return getattr, (m.im_self, m.im_func.func_name) copy_reg.pickle(types.MethodType, _reduce_method) class MyClass(object): def calc(self,num): print 'the number is ',num if __name__ == '__main__': mc = MyClass() pool = multiprocessing.Pool(processes = 3) pool.apply_async(mc.calc,(3,)) pool.apply_async(mc.calc,(4,)) pool.apply_async(mc.calc,(5,)) pool.close() pool.join() print 'end!'
运行结果:
the number is 4 the number is 3 the number is 5 end!
当在子进程的进程池中使用实例方法,程序实际调用multiprocessing.Pipe进行序列化数据。python2.7中,multiprocessing.Pipe使用C语言实现的,会立即调用
pickle_dumps,而不是调用ForkingPickler,所以传递进去的实例方法无法工作。
然而,如果使用copy_reg注册实例方法,如上不可能将变为可能。
python3.0之后的版本,可正常序列化实例方法,就不需要使用copy_reg了。
详细说明还可参考:
http://stackoverflow.com/questions/27318290/why-can-i-pass-an-instance-method-to-multiprocessing-process-but-not-a-multipro