python BitTornado P2P分发大文件
P2P分发大文件思路
1、将软件包生成种子文件
2、通过saltstack将种子文件分发至每台服务器
3、每台服务器进行种子下载
推荐使用Twitter开源的murder。Twitter用它来分发大文件完成代码更新。
下载地址: https://github.com/lg/murder
本文介绍murder的dist，它是基于python开发的。
murder dist
- murder_tracker.py 可以理解它为调度中心
- murder_make_torrent.py 生成种子，并且注册在调度中心
- murder_client.py 播种和获取文件包（上传、下载）
举例部署结构：
192.168.1.11 调度中心
192.168.1.12 种子生成、第一个播种文件服务器
192.168.1.100-200 下载文件
1、启动调度中心
python murder_tracker.py --port 8998 --dfile data --logfile urder_tracker.log
- --port 监听的端口,默认是8998
- --dfile 存储近期下载信息的文件
- --logfile 日志文件，默认是标准输出
shell启动脚本
1 #!/bin/bash 2 # 3 # Start/Stop murder-tracker 4 # 5 # description: murder-tracker 6 # processname: murder-tracker 7 8 if [ -f /etc/rc.d/init.d/functions ]; then 9 . /etc/rc.d/init.d/functions 10 fi 11 12 name="murder-tracker" 13 murder_tracker_bin="/usr/local/dist/murder_tracker.py" 14 murder_tracker_log="/usr/local/dist/logs/murder_tracker.log" 15 murder_tracker_data="/usr/local/dist/var/tracker_data" 16 murder_user=murder 17 18 find_tracker_process () { 19 PID=`ps -ef | grep murder_tracker | grep python |grep -v $0|grep -v grep|grep -v sh|grep -v root| awk '{ print $2 }'` 20 echo $PID 21 } 22 23 start () { 24 getent passwd $murder_user >/dev/null || useradd -r -s /sbin/nologin $murder_user 25 LOG_DIR=`dirname ${murder_tracker_log}` 26 DATA_DIR=`dirname ${murder_tracker_data}` 27 if [ ! -d $LOG_DIR ]; then 28 echo -e "\e[35mLog dir ${LOG_DIR} doesn't exist. Creating\e[0m" 29 mkdir -p $LOG_DIR 30 fi 31 if [ ! -d $DATA_DIR ]; then 32 echo -e "\e[35mLog dir ${DATA_DIR} doesn't exist. Creating\e[0m" 33 mkdir -p $DATA_DIR 34 fi 35 chown -R $murder_user:$murder_user $DATA_DIR $LOG_DIR 36 37 find_tracker_process 38 if [ "$PID" != "" ]; then 39 echo -e "\e[35m$name is already running!\e[0m" 40 else 41 daemon --user $murder_user nohup python $murder_tracker_bin --dfile $murder_tracker_data --logfile $murder_tracker_log > /dev/null 2>&1 & 42 echo -e "\e[35mStarting $name Done\e[0m" 43 fi 44 } 45 46 stop () { 47 find_tracker_process 48 if [ "$PID" != "" ]; then 49 echo -e "\e[35mStopping $name\e[0m" 50 kill $PID 51 else 52 echo -e "\e[35m$name is not running yet\e[0m" 53 fi 54 } 55 56 case $1 in 57 start) 58 start 59 ;; 60 stop) 61 stop 62 exit 0 63 ;; 64 restart) 65 stop 66 sleep 2 67 start 68 ;; 69 status) 70 find_tracker_process 71 if [ "$PID" != "" ]; then 72 echo -e "\e[35m$name is running: $PID\e[0m" 73 exit 0 74 else 75 echo -e "\e[35m$name is not running\e[0m" 76 exit 1 77 fi 78 ;; 79 *) 80 echo -e "\e[35mUsage: $0 {start|stop|restart|status}\e[0m" 81 RETVAL=1 82 esac 83 exit 0
2、生成种子文件
python murder_make_torrent.py deploy.tar.gz 192.168.1.11:8998 deploy.torrent
3、创建第一个播种文件服务节点
python murder_client.py seed deploy.torrent deploy.tar.gz 192.168.1.12
- 最后一个参数是本机的IP地址
4、通过saltstack将种子文件deploy.torrent发送到每台服务器上
5、所有节点192.168.1.100-200获取文件包，这些服务器下载文件，同时也承担播种节点
python murder_client.py peer deploy.torrent deploy.tar.gz 192.168.1.xxx
- 最后一个参数是本机的IP地址
备注：这些主机分发时防火墙需要放开，程序中默认使用10000-60000之间端口，BitTornado/download_bt1.py中定义
('minport', 10000, 'minimum port to listen on, counts up if unavailable'),
('maxport', 60000, 'maximum port to listen on'),
('random_port', 1, 'whether to choose randomly inside the port range ' +
'instead of counting up linearly'),
