python网络爬虫-提升爬虫的速度（八）

提升爬虫的速度

从前面几篇已经可以从获取网页、解析网页、存储数据来实现一些基本的爬虫。现在记录一些进阶部分：提升爬虫速度，主要有3中方法：多线程爬虫、多进程爬虫、多协程爬虫。对比普通单线程爬虫，使用这3种方法爬虫的速度能成倍的提升。

并发和并行

并发是指在一段时间内发生的若干时间的情况
并行是值在同一时刻发生若干事件的情况

同步和异步

同步就是并发并行的各个任务不是独自运行的，任务之间有一定交替顺序，像接力赛一样。
异步就是并发和并行的各个任务独立运行互不干扰。每个任务都不在同一个赛道上面跑步的速度不受其他选手影响

多线程爬虫

多线程爬虫是以并发的方式执行的。也就是说，多线程并不能真正的同时执行，而是通过进程的快速切换加快网络爬虫的速度的。
在操作IO的时候使用多线程可以提升程序执行效率

简单单线程爬虫

点击查看代码

	http://www.baidu.com
	http://www.qq.com
	http://www.naver.com
	http://www.taobao.com
	http://www.reddit.com
	http://www.sohu.com
	http://www.tmall.com
	http://www.sina.com.cn
	http://www.daum.net
	http://www.jd.com
	http://www.360.cn
	http://www.weibo.com
	http://www.aliexpress.com
	http://www.linkedin.com
	http://www.alipay.com
	http://www.hao123.com
	http://www.csdn.net
	http://www.youth.cn
	http://www.live.com
	http://www.tianya.cn
	http://www.microsoftonline.com
	http://www.office.com
	http://www.soso.com
	http://www.so.com
	http://www.gmw.cn
	http://www.china.com
	http://www.nate.com
	http://www.huaban.com
	http://www.bing.com
	http://www.xinhuanet.com
	http://www.youku.com
	http://www.zhihu.com
	http://www.cctv.com
	http://www.airasia.com
	http://www.douyu.com
	http://www.babytree.com
	http://www.apple.com
	http://www.sogou.com
	http://www.china.com.cn
	http://www.yelp.com
	http://www.ocbc.com
	http://www.microsoft.com
	http://www.mama.cn
	http://www.bitauto.com
	http://www.bankofamerica.com
	http://www.1688.com
	http://www.stackoverflow.com
	http://www.163.com
	http://www.39.net
	http://www.cnblogs.com
	http://www.bilibili.com
	http://www.interpark.com
	http://www.huanqiu.com
	http://www.cnzz.com
	http://www.chinadaily.com.cn
	http://www.openrice.com
	http://www.msn.com
	http://www.k618.cn
	http://www.yesky.com
	http://www.caijing.com.cn
	http://www.emirates.com
	http://www.amazon.cn
	http://www.aliyun.com
	http://www.eastday.com
	http://www.youdao.com
	http://www.oeeee.com
	http://www.ci123.com
	http://www.baike.com
	http://www.adobe.com
	http://www.rednet.cn
	http://www.iqiyi.com
	http://www.wemakeprice.com
	http://www.douban.com
	http://www.familydoctor.com.cn
	http://www.agoda.com
	http://www.jrj.com.cn
	http://www.read01.com
	http://www.17ok.com
	http://www.chinaz.com
	http://www.youboy.com
	http://www.tesco.com
	http://www.alibaba.com
	http://www.gearbest.com
	http://www.51sole.com
	http://www.dbs.com
	http://www.suning.com
	http://www.oschina.net
	http://www.voc.com.cn
	http://www.zol.com.cn
	http://www.asos.com
	http://www.chinaso.com
	http://www.jianshu.com
	http://www.ifeng.com
	http://www.stockstar.com
	http://www.zhanqi.tv
	http://www.52pk.com
	http://www.whatsbuying.com
	http://www.cqnews.net
	http://www.gongchang.com
	http://www.godaddy.com
	http://www.godaddy.com
	http://www.wtoip.com
	http://www.segmentfault.com
	http://www.evernote.com
	http://www.dianping.com
	http://www.qingdaonews.com
	http://www.guancha.cn
	http://www.standardchartered.com
	http://www.singaporeair.com
	http://www.toutiao.com
	http://www.jiameng.com
	http://www.dm5.com
	http://www.w3school.com.cn
	http://www.zhaopin.com
	http://www.99.com
	http://www.mi.com
	http://www.b2b.cn
	http://www.cathaypacific.com
	http://www.southcn.com
	http://www.battle.net
	http://www.ups.com
	http://www.jb51.net
	http://www.comcast.net
	http://www.alicdn.com
	http://www.v2ex.com
	http://www.firefoxchina.cn
	http://www.360doc.com
	http://www.xunlei.com
	http://www.sharepoint.com
	http://www.scol.com.cn
	http://www.admaimai.com
	http://www.v1.cn
	http://www.51cto.com
	http://www.jqw.com
	http://www.bzw315.com
	http://www.126.com
	http://www.beanfun.com
	http://www.chooseauto.com.cn
	http://www.renren.com
	http://www.taleo.net
	http://www.51.la
	http://www.zcool.com.cn
	http://www.4399.com
	http://www.duba.com
	http://www.globaltimes.cn
	http://www.ycwb.com
	http://www.sfacg.com
	http://www.hotelscombined.com
	http://www.mydrivers.com
	http://www.taoche.com
	http://www.runoob.com
	http://www.tlscontact.com
	http://www.nba.com
	http://www.gamebase.com.tw
	http://www.zhibo8.cc
	http://www.hexun.com
	http://www.xiami.com
	http://www.finnair.com
	http://www.feng.com
	http://www.cdstm.cn
	http://www.uniqlo.com
	http://www.iciba.com
	http://www.qudong.com
	http://www.panda.tv
	http://www.cnbeta.com
	http://www.nipic.com
	http://www.sznews.com
	http://www.huawei.com
	http://www.tuicool.com
	http://www.baimao.com
	http://www.umeng.com
	http://www.ccidnet.com
	http://www.klm.com
	http://www.qcloud.com
	http://www.hupu.com
	http://www.ikanman.com
	http://www.3dmgame.com
	http://www.icolor.com.cn
	http://www.360.com
	http://www.36kr.com
	http://www.miui.com
	http://www.boc.cn
	http://www.gamersky.com
	http://www.joyme.com
	http://www.17173.com
	http://www.uc.cn
	http://www.alimama.com
	http://www.oasgames.com
	http://www.focus.cn
	http://www.cnr.cn
	http://www.miomio.tv
	http://www.jjwxc.net
	http://www.5dcar.com
	http://www.hjenglish.com
	http://www.dangdang.com
	http://www.springer.com
	http://www.to8to.com
	http://www.xiaomi.com
	http://www.ctrip.com
	http://www.delta.com
	http://www.anjuke.com
	http://www.cnki.net
	http://www.surveymonkey.com
	http://www.tower.im
	http://www.baiducontent.com
	http://www.acfun.cn
	http://www.people.com.cn
	http://www.jmw.com.cn
	http://www.worktile.com
	http://www.newsmth.net
	http://www.vmall.com
	http://www.07073.com
	http://www.qyer.com
	http://www.hujiang.com
	http://www.cnnic.cn
	http://www.meituan.com
	http://www.yinxiang.com
	http://www.ngacn.cc
	http://www.smzdm.com
	http://www.ccb.com
	http://www.ali213.net
	http://www.alibaba-inc.com
	http://www.3158.cn
	http://www.vmall.com
	http://www.nike.com
	http://www.eqxiu.com
	http://www.jandan.net
	http://www.office365.com
	http://www.imooc.com
	http://www.ikea.com
	http://www.united.com
	http://www.ly.com
	http://www.epwk.com
	http://www.tudou.com
	http://www.leagueoflegends.com
	http://www.aa.com
	http://www.garena.com
	http://www.mafengwo.cn
	http://www.ifensi.com
	http://www.pptv.com
	http://www.fobshanghai.com
	http://www.asiamiles.com
	http://www.znds.com
	http://www.hc360.com
	http://www.job853.com
	http://www.sf-express.com
	http://www.lianjia.com
	http://www.guokr.com
	http://www.cmbchina.com
	http://www.modernweekly.com
	http://www.ynet.com
	http://www.dell.com
	http://www.dict.cn
	http://www.yinyuetai.com
	http://www.aizhan.com
	http://www.gome.com.cn
	http://www.meishichina.com
	http://www.51hejia.com
	http://www.ule.com
	http://www.ea3w.com
	http://www.saraba1st.com
	http://www.chsi.com.cn
	http://www.vlive.tv
	http://www.sonhoo.com
	http://www.hongkongairlines.com
	http://www.jxnews.com.cn
	http://www.free.com.tw
	http://www.docin.com
	http://www.liepin.com
	http://www.chinaunix.net
	http://www.weibo.cn
	http://www.ifanr.com
	http://www.51auto.com
	http://www.ebrun.com
	http://www.10010.com
	http://www.hebei.com.cn
	http://www.tgbus.com
	http://www.mtime.com
	http://www.vip.com
	http://www.kdslife.com
	http://www.www.gov.cn
	http://www.cncn.org.cn
	http://www.techcrunch.com
	http://www.zbj.com
	http://www.ip138.com
	http://www.cyol.com
	http://www.pc6.com
	http://www.joox.com
	http://www.178.com
	http://www.lagou.com
	http://www.18183.com
	http://www.365jia.cn
	http://www.autohome.com.cn
	http://www.battlenet.com.cn
	http://www.oracle.com
	http://www.miaopai.com
	http://www.sina.cn
	http://www.ch.com
	http://www.yxdown.com
	http://www.etao.com
	http://www.vietnamairlines.com
	http://www.iyiou.com
	http://www.shop.com
	http://www.588ku.com
	http://www.le.com
	http://www.sina.com
	http://www.jstv.com
	http://www.ceconline.com
	http://www.koreanair.com
	http://www.skype.com
	http://www.ih5.cn
	http://www.ems.com.cn
	http://www.efu.com.cn
	http://www.pcbaby.com.cn
	http://www.shimo.im
	http://www.macaolife.com
	http://www.xiu.com
	http://www.eastmoney.com
	http://www.xiumi.us
	http://www.yhd.com
	http://www.jiemian.com
	http://www.daikuan.com
	http://www.ximalaya.com
	http://www.marriott.com
	http://www.d1ev.com
	http://www.xitek.com
	http://www.chuansong.me
	http://www.alitrip.com
	http://www.xiaomi.cn
	http://www.51job.com
	http://www.91jm.com
	http://www.2cto.com
	http://www.qoo10.com
	http://www.centadata.com
	http://www.lufthansa.com
	http://www.techweb.com.cn
	http://www.kugou.com
	http://www.80018.cn
	http://www.tmtpost.com
	http://www.house365.com
	http://www.hp.com
	http://www.unity3d.com
	http://www.zoom.us
	http://www.kafan.cn
	http://www.liansuo.com
	http://www.netease.com
	http://www.10jqka.com.cn
	http://www.xiazaiba.com
	http://www.fang.com
	http://www.smartisan.com
	http://www.photofans.cn
	http://www.ooopic.com
	http://www.zybang.com
	http://www.gw-ec.com
	http://www.wed114.cn
	http://www.huomao.com
	http://www.ithome.com
	http://www.ccb.com.cn
	http://www.chinanews.com
	http://www.doc88.com
	http://www.sanguosha.com
	http://www.evaair.com
	http://www.icbc.com.cn
	http://www.youxidudu.com
	http://www.verycd.com
	http://www.netcoc.com
	http://www.pepper.com
	http://www.dygang.com
	http://www.liaoxuefeng.com
	http://www.flyasiana.com
	http://www.sciencenet.cn
	http://www.feiyang.com
	http://www.800hr.com
	http://www.iconfont.cn
	http://www.youzan.com
	http://www.360kan.com
	http://www.chinabyte.com
	http://www.samsung.com
	http://www.zxart.cn
	http://www.gucheng.com
	http://www.bootcss.com
	http://www.cankaoxiaoxi.com
	http://www.58pic.com
	http://www.81.cn
	http://www.csair.com
	http://www.chiphell.com
	http://www.antpedia.com
	http://www.xiachufang.com
	http://www.winshang.com
	http://www.fzg360.com
	http://www.chaduo.com
	http://www.12306.cn
	http://www.morningpost.com.cn
	http://www.soku.com
	http://www.sspai.com
	http://www.yoox.com
	http://www.huxiu.com
	http://www.nyu.edu
	http://www.jiwu.com
	http://www.u17.com
	http://www.jiayuan.com
	http://www.yy.com
	http://www.duowan.com
	http://www.mbalib.com
	http://www.wanfangdata.com.cn
	http://www.ibuying.com
	http://www.chouti.com
	http://www.71.net
	http://www.hrloo.com
	http://www.meizu.com
	http://www.miercn.com
	http://www.fengniao.com
	http://www.fangdd.com
	http://www.htc.com
	http://www.jdzj.com
	http://www.pcauto.com.cn
	http://www.kaola.com
	http://www.kuaidi100.com
	http://www.yougov.com
	http://www.ku6.com
	http://www.sanwen8.cn
	http://www.yiwugou.com
	http://www.lottedfs.com
	http://www.cisco.com
	http://www.wallstreetcn.com
	http://www.gamedog.cn
	http://www.tencent.com
	http://www.tvhome.com
	http://www.xbox.com
	http://www.cr173.com
	http://www.onlinedown.net
	http://www.ebay.com.hk
	http://www.searchs.cn
	http://www.17track.net
	http://www.hyundai.com
	http://www.baixing.com
	http://www.258.com
	http://www.cn2che.com
	http://www.pudn.com
	http://www.dv37.com
	http://www.dv37.com
	http://www.uisdc.com
	http://www.sojump.com
	http://www.d1net.com
	http://www.ganji.com
	http://www.jobbole.com
	http://www.pearsoncmg.com
	http://www.kongfz.com
	http://www.365jilin.com
	http://www.strawberrynet.com
	http://www.11467.com
	http://www.jobui.com
	http://www.hh010.com
	http://www.teambition.com
	http://www.woshipm.com
	http://www.lge.com
	http://www.kanxi.cc
	http://www.leiphone.com
	http://www.d1com.com
	http://www.114so.cn
	http://www.d1com.com
	http://www.114so.cn
	http://www.duomai.com
	http://www.win007.com
	http://www.weidian.com
	http://www.qiku.com
	http://www.cli.im
	http://www.flyertea.com
	http://www.lenovo.com.cn
	http://www.aso100.com
	http://www.xueqiu.com
	http://www.bp.com
	http://www.dingtalk.com
	http://www.processon.com
	http://www.flyme.cn
	http://www.a9vg.com
	http://www.sinaimg.cn
	http://www.saic.gov.cn
	http://www.mgtv.com
	http://www.nuomi.com
	http://www.tiexue.net
	http://www.vvvdj.com
	http://www.tvmao.com
	http://www.panduoduo.net
	http://www.wechat.com
	http://www.52pojie.cn
	http://www.miwifi.com
	http://www.iteye.com
	http://www.kanzhun.com
	http://www.mango.com
	http://www.cheaa.com
	http://www.13322.com
	http://www.jikexueyuan.com
	http://www.taisha.org
	http://www.mydigit.cn
	http://www.gusuwang.com
	http://www.pinggu.org
	http://www.lbldy.com
	http://www.sgcn.com
	http://www.misumi-ec.com
	http://www.lofter.com
	http://www.unrealengine.com
	http://www.gao7.com
	http://www.leju.com
	http://www.home77.com
	http://www.qunar.com
	http://www.xdowns.com
	http://www.oa.com
	http://www.sgcn.com
	http://www.szjy188.com
	http://www.tuniu.com
	http://www.135editor.com
	http://www.f.com
	http://www.zhibo.tv
	http://www.jiyoujia.com
	http://www.95516.com
	http://www.yiqifa.com
	http://www.cocoachina.com
	http://www.babyschool.com.cn
	http://www.iweihai.cn
	http://www.haowu.com
	http://www.hm.com
	http://www.wish.com
	http://www.fitbit.com
	http://www.taojindi.com
	http://www.koolearn.com
	http://www.xabbs.com
	http://www.020.com
	http://www.qiniu.com
	http://www.25pp.com
	http://www.nga.cn
	http://www.educity.cn
	http://www.zealer.com
	http://www.xdowns.com
	http://www.liqu.com
	http://www.qichacha.com
	http://www.51credit.com
	http://www.duomai.com
	http://www.juooo.com
	http://www.shanbay.com
	http://www.juooo.com
	http://www.shanbay.com
	http://www.meishij.net
	http://www.th7.cn
	http://www.jia400.com
	http://www.cas.cn
	http://www.wenwuchina.com
	http://www.189.cn
	http://www.liuxue86.com
	http://www.klook.com
	http://www.shfft.com
	http://www.8264.com
	http://www.china.cn
	http://www.zhifang.com
	http://www.made-in-china.com
	http://www.rabbitpre.com
	http://www.sap.com
	http://www.macx.cn
	http://www.everychina.com
	http://www.9game.cn
	http://www.ca800.com
	http://www.dgtle.com
	http://www.cloudscar.com
	http://www.bdhome.cn
	http://www.news18a.com
	http://www.shilladfs.com
	http://www.net-a-porter.com
	http://www.zealer.com
	http://www.discoverhongkong.com
	http://www.80s.tw
	http://www.9ku.com
	http://www.33lc.com
	http://www.thepaper.cn
	http://www.scswl.cn
	http://www.officedepot.com
	http://www.fx678.com
	http://www.banma.com
	http://www.eee114.com
	http://www.9384.com
	http://www.xuexila.com
	http://www.9384.com
	http://www.xuexila.com
	http://www.cheshen.cn
	http://www.mr-world.com
	http://www.fx112.com
	http://www.97665.com
	http://www.chinahr.com
	http://www.acs.org
	http://www.mikecrm.com
	http://www.checheng.com
	http://www.appgame.com
	http://www.linkhaitao.com
	http://www.meipai.com
	http://www.linuxidc.com
	http://www.fliggy.com
	http://www.amap.com
	http://www.4px.com
	http://www.qpic.cn
	http://www.modao.cc
	http://www.dianxiaomi.com
	http://www.56.com
	http://www.java.com
	http://www.hdpfans.com
	http://www.thinkphp.cn
	http://www.2345.com
	http://www.baoku.com
	http://www.tiancity.com
	http://www.bcsh.com
	http://www.bozhong.com
	http://www.zhiding.cn
	http://www.longzhu.com
	http://www.xjtour.com
	http://www.kancloud.cn
	http://www.open-open.com
	http://www.itpub.net
	http://www.elong.com
	http://www.pchome.net
	http://www.pps.tv
	http://www.qinqinbaby.com
	http://www.chuandong.com
	http://www.coding.net
	http://www.yidianzixun.com
	http://www.51nb.com
	http://www.dhgate.com
	http://www.10086.cn
	http://www.6vhao.com
	http://www.5acbd.com
	http://www.atobo.com.cn
	http://www.kubo365.com
	http://www.111cn.net
	http://www.zhongmin.cn
	http://www.weiyangx.com
	http://www.juesheng.com
	http://www.uuu9.com
	http://www.siilu.com
	http://www.pconline.com.cn
	http://www.dji.com
	http://www.west.cn
	http://www.ctfile.com
	http://www.idianfa.com
	http://www.smm.cn
	http://www.shejis.com
	http://www.zhangyu.tv
	http://www.17zwd.com
	http://www.dhl.com
	http://www.shfft.com
	http://www.wanmei.com
	http://www.122.gov.cn
	http://www.51nb.com
	http://www.xici.net
	http://www.cnki.com.cn
	http://www.redocn.com
	http://www.qvc.com
	http://www.aipai.com
	http://www.dapenti.com
	http://www.3lian.com
	http://www.guidechem.com
	http://www.jiankang.com
	http://www.tgfcer.com
	http://www.freebuf.com
	http://www.sodao.com
	http://www.zhcw.com
	http://www.sh.com
	http://www.ablesky.com
	http://www.microsoftstore.com.cn
	http://www.7k7k.com
	http://www.southmoney.com
	http://www.btc123.com
	http://www.digitaling.com
	http://www.meitu.com
	http://www.chinaaet.com
	http://www.kaoyan.com
	http://www.aipai.com
	http://www.tripadvisor.cn
	http://www.colg.cn
	http://www.admin5.com
	http://www.ncar.cc
	http://www.intel.com
	http://www.wanyx.com
	http://www.chmotor.cn
	http://www.mxhichina.com
	http://www.jzb.com
	http://www.it168.com
	http://www.1kkk.com
	http://www.cnodejs.org
	http://www.hudong.com
	http://www.ucweb.com
	http://www.xyw.gov.cn
	http://www.airasiago.com
	http://www.damai.cn
	http://www.farnell.com
	http://www.hi-pda.com
	http://www.wenku1.com
	http://www.haosou.com
	http://www.ishuhui.com
	http://www.paopaoche.net
	http://www.csai.cn
	http://www.zhaoshangbao.com
	http://www.eol.cn
	http://www.excelhome.net
	http://www.missevan.com
	http://www.cncv.org.cn
	http://www.365yg.com
	http://www.huim.com
	http://www.zxxk.com
	http://www.51yes.com
	http://www.cainiao.com
	http://www.nh87.cn
	http://www.b0yp.com
	http://www.qdaily.com
	http://www.kongzhong.com
	http://www.shangc.net
	http://www.dongqiudi.com
	http://www.jiankang.com
	http://www.dzsc.com
	http://www.chinaacc.com
	http://www.vcg.com
	http://www.oneplusbbs.com
	http://www.xuetangx.com
	http://www.fz222.com
	http://www.cnwnews.com
	http://www.chinadmd.com
	http://www.b2b168.com
	http://www.pingan.com
	http://www.pushauction.com
	http://www.sdo.com
	http://www.9978.cn
	http://www.ltaaa.com
	http://www.gxyj.com
	http://www.kuaizhan.com
	http://www.airchina.com.cn
	http://www.gcl-power.com
	http://www.medsci.cn
	http://www.lbxcn.com
	http://www.lzgd.com.cn
	http://www.oray.com
	http://www.taobao.org
	http://www.btbtdy.com
	http://www.i2ya.com
	http://www.istar.cn
	http://www.xgo.com.cn
	http://www.66law.cn
	http://www.heiguang.com
	http://www.ao.com
	http://www.jq22.com
	http://www.qidian.com
	http://www.goldcarpet.cn
	http://www.zxbtz.cn
	http://www.jiushang.cn
	http://www.cicpa.org.cn
	http://www.wowenda.com
	http://www.coursera.org
	http://www.fangdr.com
	http://www.cps.com.cn
	http://www.kmf.com
	http://www.cri.cn
	http://www.lmjx.net
	http://www.lonshinetech.cn
	http://www.infoq.com
	http://www.gushiwen.org
	http://www.ecp888.com
	http://www.tongtool.com
	http://www.dajie.com
	http://www.co188.com
	http://www.fumanhua.net
	http://www.maiche168.com
	http://www.sankuai.com
	http://www.ucas.ac.cn
	http://www.lamabang.com
	http://www.huajiao.com
	http://www.accorhotels.com
	http://www.wendangku.net
	http://www.dragonparking.com
	http://www.6789.com
	http://www.xdf.cn
	http://www.tucao.tv
	http://www.91yunxiao.com
	http://www.liebiao.com
	http://www.9lianmeng.com
	http://www.51240.com
	http://www.zhiyoo.com
	http://www.silkair.com
	http://www.313.cn
	http://www.ssl-images-amazon.com
	http://www.eepw.com.cn
	http://www.gs307.com
	http://www.yindou.com
	http://www.i1515.com
	http://www.imiker.com
	http://www.lvmama.com
	http://www.louisvuitton.com
	http://www.nowgoal.com
	http://www.makeding.com
	http://www.xz7.com
	http://www.guitarchina.com
	http://www.wto168.net
	http://www.abchina.com
	http://www.fzdm.com
	http://www.ichacha.net
	http://www.1024sj.com
	http://www.ef43.com.cn
	http://www.newrank.cn
	http://www.ceair.com
	http://www.zimuku.net
	http://www.ppkoo.com
	http://www.jc35.com
	http://www.dnspod.cn
	http://www.hsw.cn
	http://www.caixin.com
	http://www.manmanbuy.com
	http://www.23us.com
	http://www.asus.com
	http://www.zoosnet.net
	http://www.xp510.com
	http://www.vgtime.com
	http://www.qiushibaike.com
	http://www.jinshuju.net
	http://www.115.com
	http://www.3367.com
	http://www.fanli.com
	http://www.newcger.com
	http://www.kepu.net.cn
	http://www.findlaw.cn
	http://www.jiumei.com
	http://www.gkstk.com
	http://www.ihg.com
	http://www.blizzard.com
	http://www.lenovo.com
	http://www.longau.com
	http://www.seedit.com
	http://www.ofweek.com
	http://www.61baobao.com
	http://www.400.cn
	http://www.wines-info.com
	http://www.innisfree.com
	http://www.weather.com.cn
	http://www.che168.com
	http://www.dilidili.wang
	http://www.7po.com
	http://www.qiushibaike.com
	http://www.9r.cn
	http://www.weather.com.cn
	http://www.107cine.com
	http://www.coolapk.com
	http://www.ixueshu.com
	http://www.iplaysoft.com
	http://www.blizzard.cn
	http://www.dangbei.com
	http://www.hellorf.com
	http://www.21food.cn
	http://www.libaclub.com
	http://www.outofmemory.cn
	http://www.ele.me
	http://www.shihuo.cn
	http://www.zmz2017.com
	http://www.zybuluo.com
	http://www.66ys.tv
	http://www.sczw.com
	http://www.xtx6.com
	http://www.tutorabc.com
	http://www.zhipin.com
	http://www.cgdc.com.cn
	http://www.61learn.com
	http://www.sm.cn
	http://www.571xz.com
	http://www.sobt5.org
	http://www.starwoodhotels.com
	http://www.qqtn.com
	http://www.sgamer.com
	http://www.120ask.com
	http://www.appinn.com
	http://www.qianzhan.com
	http://www.888pic.com
	http://www.tianyancha.com
	http://www.k73.com
	http://www.yiibai.com
	http://www.downxia.com
	http://www.managershare.com
	http://www.downcc.com
	http://www.biquge.tw
	http://www.fgowiki.com
	http://www.p2peye.com
	http://www.haosou.com
	http://www.yimu100.com
	http://www.fox.com
	http://www.mrporter.com
	http://www.genshuixue.com
	http://www.jisutiyu.com
	http://www.topfo.com
	http://www.right.com.cn
	http://www.5ewin.com
	http://www.dongnanshan.com
	http://www.jizhangla.com
	http://www.laawoo.com
	http://www.3618med.com
	http://www.ahgame.com
	http://www.mamicode.com
	http://www.wugu.com.cn
	http://www.115.com
	http://www.genshuixue.com
	http://www.57mh.com
	http://www.oiegg.com
	http://www.21csp.com.cn
	http://www.kekenet.com
	http://www.c5game.com
	http://www.juejin.im
	http://www.baofeng.com
	http://www.kuwo.cn
	http://www.6.cn
	http://www.chayu.com
	http://www.sanwen.net
	http://www.962.net
	http://www.etest.net.cn
	http://www.innisfree.com
	http://www.dragonair.com
	http://www.vjshi.com
	http://www.lawtime.cn
	http://www.sccnn.com
	http://www.qqbaobao.com
	http://www.dragonair.com
	http://www.vjshi.com
	http://www.lawtime.cn
	http://www.sccnn.com
	http://www.qqbaobao.com
	http://www.chinaswitch.com
	http://www.5118.com
	http://www.cntv.cn
	http://www.knowsky.com
	http://www.skyscanner.com
	http://www.wrz.com
	http://www.wasu.cn
	http://www.mojifen.com
	http://www.nvidia.com
	http://www.oceanpark.com.hk
	http://www.pcbeta.com
	http://www.psnine.com
	http://www.228.com.cn
	http://www.zhuixinfan.com
	http://www.okcoin.cn
	http://www.huya.com
	http://www.1ppt.com
	http://www.fyber.com
	http://www.72byte.com
	http://www.cpic.com.cn
	http://www.wlmq.com
	http://www.lusongsong.com
	http://www.fanjian.net
	http://www.hopetrip.com.hk
	http://www.hnjy.com.cn
	http://www.8kana.com
	http://www.8d.cc
	http://www.linux.cn
	http://www.enterprise.com
	http://www.iqing.in
	http://www.sg560.com
	http://www.mnw.cn
	http://www.trendmicro.com
	http://www.sipo.gov.cn
	http://www.a.com.cn
	http://www.hangame.com
	http://www.cngold.org
	http://www.95095.com
	http://www.ishuo.cn
	http://www.tecenet.com
	http://www.jinti.com
	http://www.sobaidupan.com
	http://www.ichunqiu.com
	http://www.xilu.com
	http://www.3987.com
	http://www.rr-sc.com
	http://www.99114.com
	http://www.haodou.com
	http://www.wolfram.com
	http://www.expreview.com
	http://www.myexception.cn
	http://www.shixiseng.com
	http://www.bjjs.gov.cn
	http://www.xxbiquge.com
	http://www.lesports.com
	http://www.hea.cn
	http://www.24home.com
	http://www.yeah.net
	http://www.qcw.com
	http://www.shoes.net.cn
	http://www.9c9v.com
	http://www.bjhjyd.gov.cn
	http://www.ecvv.com
	http://www.fanlibang.com
	http://www.jxmall.com
	http://www.xcar.com.cn
	http://www.go108.com.cn
	http://www.divcss5.com
	http://www.sc.com
	http://www.watchstore.com.cn
	http://www.mexgroup.com
	http://www.xunyingwang.com
	http://www.chinagate.cn
	http://www.zdic.net
	http://www.bdimg.com

link_list = []
with open(r'C:\Users\K1567\Desktop\alexa.txt', 'r') as file:
    file_list = file.readlines()
    for e in file_list:
        link = e.split('\t')[1]
        link = link.replace('\n', '')
        link_list.append(link)
stat = time.time()
for e in link_list:
    try:
        r = requests.get(e)
        print(r.status_code, e)
    except Exception as erro:
        print('Error:', erro)
end = time.time()
print('串行的总时长为：', end - stat)

学习python多线程

python两种使用多线程的方法。
函数式：调用_thread模块中的start_new_thread()
类包装式：调用Threading库创建线程，从threading.thread继承。
1.

# 为线程定义一个函数
def print_time(threadName, delay):
count = 0
while count < 3:
    time.sleep(delay)
    count += 1
    print(threadName, time.ctime())

# _thread.start_new_thread(print_time, ("Thread-1", 1))
# _thread.start_new_thread(print_time, ("Thread-2", 2))
# print("Main Finished")
 
class myThread(threading.Thread):
def __init__(self, name, delay):
    threading.Thread.__init__(self)
    self.name = name
    self.delay = delay
 
def run(self):
    print("Starting" + self.name)
    print_time(self.name, self.delay)
    print("Exiting" + self.name)
 
def print_time(threadName, delay):
    counter = 0
    while counter < 3:
        time.sleep(delay)
        print(threadName, time.ctime())
    counter += 1
threads = []
 
# 创建新线程
thread1 = myThread("Thread-1", 1)
thread2 = myThread("Thread-2", 2)
 
# 开启新线程
thread1.start()
thread2.start()
 
# 添加线程到线程列表
threads.append(thread1)
threads.append(thread2)
 
# 等待所有线程完成
for t in threads:
	t.join()
print("Exiting Main Thread")

run():以表示线程活动的方法
start():启动线程活动
join([time]):组设调用线程直至线程的join()方法被调用为止
isAlive():返回线程是否是活动的
getNmae():返回线程名称
setName():设置线程名
上面代码中，thread1 = myThread("Thread-1", 1)，然后在myThread这个类中对线程进行设置，使用run()表示线程运行方法当counter小于3时打印线程名称和时间。然后使用thread1.start()开启线程，使用threads.append(thread1)添加线程到线程列表中，用t.join()等待所有线程完成才会继续执行主线程。

简单的多线程爬虫实例

import threading
link_list = []
with open(r'C:\Users\K1567\Desktop\alexa.txt', 'r') as file:
	file_list = file.readlines()
	for e in file_list:
		link = e.split('\t')[1]
		link = link.replace('\n', '')
		link_list.append(link)
stat = time.time()
 
 
class myThread(threading.Thread):
	def __init__(self, name, link_range):
		threading.Thread.__init__(self)
		self.name = name
		self.link_range = link_range
 
	def run(self):
		print("Starting" + self.name)
		crawler(self.name, self.link_range)
		print("Exiting" + self.name)
 
 
def crawler(threaName, link_range):
	for i in range(link_range[0], link_range[1] + 1):
		try:
			r = requests.get(link_list[i], timeout=20)
			print(threaName, r.status_code, link_list[i])
		except Exception as e:
			print(threaName, 'Error:', e)
 
 
thread_list = []
 
link_range_list = [(0, 200), (201, 400), (401, 600), (601, 800), (801, 1000)]
 
# 创建
for i in range(1, 6):
	thread = myThread("Thread-" + str(i), link_range_list[i - 1])
	thread.start()
	thread_list.append(thread)
# 等待所有线程完成
for i in thread_list:
	i.join()
end = time.time()
 
print('简单多线程爬虫的总时长为：', end - stat)

上面代码中，将1000个网页分成5份，然后利用for循环创建了5个线程，将这些网页分别指派到5个线程中运行

使用Queue的多线程爬虫

python的Queue模块提供了同步的、线程安全的队列类，包括FIFO（先进先出）队列、LIFO（后入先出）队列和优先级队列PriorityQueue。
例子：
开启五个线程然后通过队列的方式，把一千个网页平均分配给这五个线程

link_list = []  # 网页连接
with open(r'C:\Users\K1567\Desktop\alexa.txt', 'r') as file:
	file_list = file.readlines()
	for e in file_list:
		link = e.split('\t')[1]
		link = link.replace('\n', '')
		link_list.append(link)
# 开始时间
stat = time.time()
# 继承Thread类
class myThread(threading.Thread):
	def __init__(self, name, q):
		threading.Thread.__init__(self)
		self.name = name
		self.q = q
 
	def run(self):
		print("Starting" + self.name)
		while True:
			try:
				crawler(self.name, self.q)
			except Exception as e:
				break
		print("Exiting" + self.name)
 
 
def crawler(threaName, q):
	# 获取队列中的链接
	url = q.get(timeout=2)
	try:
		r = requests.get(url, timeout=20)
		print(q.qsize(), threaName, r.status_code, url)
	except Exception as e:
		print(q.qsize(), threaName, url, 'Error', e)
 
 
threadlist = ['Thread-1', 'Thread-2', 'Thread-3', 'Thread-4', 'Thread-5']
# 建立一个队列对象
workQueue = Queue.Queue(1000)
threads = []
 
# 创建新线程
for tName in threadlist:
	thread = myThread(tName, workQueue)
	thread.start()
	threads.append(thread)
 
# 填充队列
for url in link_list:
	workQueue.put(url)  # 填充队列
 
# 等待所有线程完成
for t in threads:
	t.join()
 
end = time.time()
print('简单多线程爬虫的总时长为：', end - stat)

多进程爬虫

python的多线程爬虫只能运行在单核上，各个线程以并发的方式异步运行。由于GIL的存在，多线程并不能发挥多核CPU的资源。
作为提升python网络爬虫的速度的另外一种方法，多进程爬虫则可以利用CPU的多核，多进程就需要用到multiprocessing这个库。
使用multiprocess这个库有两种方法，一种是使用Process+queue的方法，另外一种是pool+queue的方法。

使用multiprocessing的多进程爬虫

当进程数大于cpu的内核数量时，等待运行的进程会等其他进程运行完让出内核。所以我们需要了解计算机的cpu核心数量。

查看当前电脑spu核
from multiprocessing import cpu_count
print(cpu_count())

多线程爬虫实例：
1.Process+queue的方法，在多进程中，每个进程都可以单独设置它的属性，如果将daemon设置为true，当父进程结束后，子进程就会自动终止。

from multiprocessing import Queue, Process
link_list = []
with open(r'C:\Users\K1567\Desktop\alexa.txt', 'r') as file:
	file_list = file.readlines()
	for e in file_list:
		link = e.split('\t')[1]
		link = link.replace('\n', '')
		link_list.append(link)
stat = time.time()
 
# Process子进程
class MyProcess(Process):
	def __init__(self, q):
		Process.__init__(self)
		self.q = q
 
	def run(self):
		print("Starting", self.pid)
		while not self.q.empty():
			crawler(self.q)
		print("Exiting" + str(self.pid))
 
 
def crawler(q):
	url = q.get(timeout=2)
	try:
		r = requests.get(url, timeout=20)
		print(q.qsize(), r.status_code, url)
	except Exception as e:
		print(q.qsize(), url, 'Error', e)
 
 
if __name__ == '__main__':
	workQueue = Queue(1000)
 
	#     填充队列
	for url in link_list:
		workQueue.put(url)
 
	for i in range(0, 5):
		p = MyProcess(workQueue)
		p.daemon = True
		p.start()
		p.join()
	end = time.time()
	print('简单多进程爬虫的总时长为：', end - stat)

2.使用pool+queue的多进程爬虫：当被操作数目不大时，可以直接利用multiprocessing中的process动态生成多个进程，十几个还好，如果成百上千个目标，手动的限制进程数量就太繁琐，此时可以使用pool发挥进程池的功效。