python并发编程实战（四）：使用多线程，python爬虫被加速10倍

1|0python创建多线程的方法

2|0单线程、多线程爬取博客园速度对比

tmp/blog_spider.py

import requests

urls = [
    f"https://www.cnblogs.com/#p{page}"
    for page in range(1, 50+1)
]

def craw(url):
    r = requests.get(url)
    print(url, len(r.text))


craw(urls[0])

01.multi_thread_craw.py

import blog_spider
import threading, time


def single_thread():
    print("single_thread begin")
    for url in blog_spider.urls:
        blog_spider.craw(url)
    print("single_thread end")


def multi_thread():
    print("single_thread begin")
    threads = []
    for url in blog_spider.urls:
        threads.append(
            threading.Thread(target=blog_spider.craw, args=(url, ))
        )


    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

    print("single_thread end")



if __name__ == '__main__':
    start = time.time()
    single_thread()
    end = time.time()
    print("single_thread cost: ", end - start, "seconds")

    start = time.time()
    multi_thread()
    end = time.time()
    print("multi_thread cost: ", end - start, "seconds")

爬取速度对比
可以看到，速度足足提高了8倍，当然和电脑的性能也有关系
单线程

多线程

__EOF__

本文作者：cnhkzyy
本文链接：https://www.cnblogs.com/my_captain/p/16437989.html
关于博主：评论和私信会在第一时间回复。或者直接私信我。
版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
声援博主：如果您觉得文章对您有帮助，可以点击文章右下角【推荐】一下。您的鼓励是博主的最大动力！