Python使用multiprocessing实现一个最简单的分布式作业调度系统过程分析
介绍
Python的multiprocessing模块不但支持多进程,其中managers子模块还支持把多进程分布到多台机器上。一个服务进程可以作为调度者,将任务分布到其他多个机器的多个进程中,依靠网络通信。
想到这,就在想是不是可以使用此模块来实现一个简单的作业调度系统。
实现
以下导入库可能有个错,需改成:
from queue import Queue
Job
首先创建一个Job类,为了测试简单,只包含一个job id属性
job.py
1 2 3 4 5 | #!/usr/bin/env python # -*- coding: utf-8 -*- class Job: def __init__( self , job_id): self .job_id = job_id |
Master
Master用来派发作业和显示运行完成的作业信息
master.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | from Queue import Queue from multiprocessing.managers import BaseManager from job import Job class Master: def __init__( self ): # 派发出去的作业队列 self .dispatched_job_queue = Queue() # 完成的作业队列 self .finished_job_queue = Queue() def get_dispatched_job_queue( self ): return self .dispatched_job_queue def get_finished_job_queue( self ): return self .finished_job_queue def start( self ): # 把派发作业队列和完成作业队列注册到网络上 BaseManager.register( 'get_dispatched_job_queue' , callable = self .get_dispatched_job_queue) BaseManager.register( 'get_finished_job_queue' , callable = self .get_finished_job_queue) # 监听端口和启动服务 manager = BaseManager(address = ( '0.0.0.0' , 8888 ), authkey = 'jobs' ) manager.start() # 使用上面注册的方法获取队列 dispatched_jobs = manager.get_dispatched_job_queue() finished_jobs = manager.get_finished_job_queue() # 这里一次派发10个作业,等到10个作业都运行完后,继续再派发10个作业 job_id = 0 while True : for i in range ( 0 , 10 ): job_id = job_id + 1 job = Job(job_id) print ( 'Dispatch job: %s' % job.job_id) dispatched_jobs.put(job) while not dispatched_jobs.empty(): job = finished_jobs.get( 60 ) print ( 'Finished Job: %s' % job.job_id) manager.shutdown() if __name__ = = "__main__" : master = Master() master.start() |
Slave
Slave用来运行master派发的作业并将结果返回
slave.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | import time from Queue import Queue from multiprocessing.managers import BaseManager from job import Job class Slave: def __init__( self ): # 派发出去的作业队列 self .dispatched_job_queue = Queue() # 完成的作业队列 self .finished_job_queue = Queue() def start( self ): # 把派发作业队列和完成作业队列注册到网络上 BaseManager.register( 'get_dispatched_job_queue' ) BaseManager.register( 'get_finished_job_queue' ) # 连接master server = '127.0.0.1' print ( 'Connect to server %s...' % server) manager = BaseManager(address = (server, 8888 ), authkey = 'jobs' ) manager.connect() # 使用上面注册的方法获取队列 dispatched_jobs = manager.get_dispatched_job_queue() finished_jobs = manager.get_finished_job_queue() # 运行作业并返回结果,这里只是模拟作业运行,所以返回的是接收到的作业 while True : job = dispatched_jobs.get(timeout = 1 ) print ( 'Run job: %s ' % job.job_id) time.sleep( 1 ) finished_jobs.put(job) if __name__ = = "__main__" : slave = Slave() slave.start() |
测试
分别打开三个linux终端,第一个终端运行master,第二个和第三个终端用了运行slave,运行结果如下
master
>$ python master.py Dispatch job: 1 Dispatch job: 2 Dispatch job: 3 Dispatch job: 4 Dispatch job: 5 Dispatch job: 6 Dispatch job: 7 Dispatch job: 8 Dispatch job: 9 Dispatch job: 10 Finished Job: 1 Finished Job: 2 Finished Job: 3 Finished Job: 4 Finished Job: 5 Finished Job: 6 Finished Job: 7 Finished Job: 8 Finished Job: 9 Dispatch job: 11 Dispatch job: 12 Dispatch job: 13 Dispatch job: 14 Dispatch job: 15 Dispatch job: 16 Dispatch job: 17 Dispatch job: 18 Dispatch job: 19 Dispatch job: 20 Finished Job: 10 Finished Job: 11 Finished Job: 12 Finished Job: 13 Finished Job: 14 Finished Job: 15 Finished Job: 16 Finished Job: 17 Finished Job: 18 Dispatch job: 21 Dispatch job: 22 Dispatch job: 23 Dispatch job: 24 Dispatch job: 25 Dispatch job: 26 Dispatch job: 27 Dispatch job: 28 Dispatch job: 29 Dispatch job: 30
slave1
$ python slave.py Connect to server 127.0.0.1... Run job: 1 Run job: 2 Run job: 3 Run job: 5 Run job: 7 Run job: 9 Run job: 11 Run job: 13 Run job: 15 Run job: 17 Run job: 19 Run job: 21 Run job: 23
slave2
>$ python slave.py Connect to server 127.0.0.1... Run job: 4 Run job: 6 Run job: 8 Run job: 10 Run job: 12 Run job: 14 Run job: 16 Run job: 18 Run job: 20 Run job: 22 Run job: 24
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】凌霞软件回馈社区,博客园 & 1Panel & Halo 联合会员上线
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】博客园社区专享云产品让利特惠,阿里云新客6.5折上折
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步