1、asyncio aiohttp aiofile 异步爬取图片

前后折腾了好多天，不废话，先直接上代码，再分析：

import aiohttp
import asyncio
import aiofiles

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1',
                  'Referer': 'https://www.mzitu.com/',
                   'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
                   'Accept-Encoding': 'gzip',
                   }

async def fetch(session, url):
    async with session.get(url, proxy='http://59.62.164.252:9999') as response:
        return await response.read()

async def main():
    async with aiohttp.ClientSession(headers=header) as session:
        content = await fetch(session, 'https://i.meizitu.net/thumbs/2019/03/174061_01e35_236.jpg')
        print(content)
        async with aiofiles.open('D:/a.jpg', 'wb') as f:
            f.write(content)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

 

开始心路历程：

1、看了廖雪峰老师python教程中协程一章节、《流畅的python》中协程一章节，以及前前后后网上查询的资料，不管怎么改均报错，人接近暴走状态。

最后Google查询ClientSession：Client Reference，复制源码做尝试：

import aiohttp
import asyncio

async def fetch(client):
    async with client.get('http://python.org') as resp:
        assert resp.status == 200
        return await resp.text()

async def main():
    async with aiohttp.ClientSession() as client:
        html = await fetch(client)
        print(html)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

运行成功

2、改为下载图片，并想fetch函数能不能直接返回response？

import aiohttp
import asyncio
import aiofiles

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1',
                  'Referer': 'https://www.mzitu.com/',
                   'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
                   'Accept-Encoding': 'gzip',
                   }

async def fetch(session, url):
    async with session.get(url) as response:
        return response

async def main():
    async with aiohttp.ClientSession() as session:
        response = await fetch(session, 'https://i.meizitu.net/thumbs/2019/03/174061_01e35_236.jpg')
        print(response.read())
        with open('D:/a.jpg', 'wb') as f:
            f.write(response.read())

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

运行直接报错：

貌似fetch函数中不能返回response？百思不得姐，问题先放这，以后再解决吧

3、根据上面ClientSession文档中介绍：

请求头header应放在ClientSession实例化中

4、aiohttp supports HTTP/HTTPS proxies

但是，它根本就不支持 https 代理。

可参考 Python3 异步代理爬虫池

 

头疼，先写这么多吧

最后尝试貌似代理ip又有问题，晕