使用asyncio模块处理文件报错

错误:ValueError: too many file descriptors in select()

解决方案:

  1. 加并发限制semaphore = asyncio.Semaphore(200),但是只能针对爬虫调接口
    semaphore = asyncio.Semaphore(100)
    async with semaphore:
        async with aiohttp.ClientSession() as session:
            async with session.post(url, data=data, headers=header) as res:
                if int(res.status) == 200:
                    text = await res.json(encoding="utf-8")
                    print(text)
                    return text
    
  2. 出现这个问题是因为千级文件操作,使用ProactorEventLoop,适用于windows io操作过多的情况,问题解决
    from asyncio import ProactorEventLoop
    asyncio.set_event_loop(ProactorEventLoop())
    loop = asyncio.get_event_loop()
    
    # 源码
    if sys.platform == 'win32':
        ProactorEventLoop: Type[AbstractEventLoop]
    
posted @ 2020-06-03 16:26  今日店休  阅读(419)  评论(0编辑  收藏  举报