使用asyncio模块处理文件报错
错误:ValueError: too many file descriptors in select()
解决方案:
- 加并发限制semaphore = asyncio.Semaphore(200),但是只能针对爬虫调接口
semaphore = asyncio.Semaphore(100) async with semaphore: async with aiohttp.ClientSession() as session: async with session.post(url, data=data, headers=header) as res: if int(res.status) == 200: text = await res.json(encoding="utf-8") print(text) return text
- 出现这个问题是因为千级文件操作,使用ProactorEventLoop,适用于windows io操作过多的情况,问题解决
from asyncio import ProactorEventLoop asyncio.set_event_loop(ProactorEventLoop()) loop = asyncio.get_event_loop()
# 源码 if sys.platform == 'win32': ProactorEventLoop: Type[AbstractEventLoop]