python并发编程实战（四）：使用多线程，python爬虫被加速10倍

python创建多线程的方法

单线程、多线程爬取博客园速度对比

tmp/blog_spider.py

import requests

urls = [
    f"https://www.cnblogs.com/#p{page}"
    for page in range(1, 50+1)
]

def craw(url):
    r = requests.get(url)
    print(url, len(r.text))


craw(urls[0])

01.multi_thread_craw.py

import blog_spider
import threading, time


def single_thread():
    print("single_thread begin")
    for url in blog_spider.urls:
        blog_spider.craw(url)
    print("single_thread end")


def multi_thread():
    print("single_thread begin")
    threads = []
    for url in blog_spider.urls:
        threads.append(
            threading.Thread(target=blog_spider.craw, args=(url, ))
        )


    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

    print("single_thread end")



if __name__ == '__main__':
    start = time.time()
    single_thread()
    end = time.time()
    print("single_thread cost: ", end - start, "seconds")

    start = time.time()
    multi_thread()
    end = time.time()
    print("multi_thread cost: ", end - start, "seconds")

爬取速度对比
可以看到，速度足足提高了8倍，当然和电脑的性能也有关系
单线程

多线程