scrapy采集gb2312网页中文乱码笔记

由于目标采集资源为gb2312发生乱码，采用中间件的解决方式，中间件为DownloaderMiddleware

    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.
        # Must either;
        # - return a Response object
        # - return a Request object
        response = HtmlResponse(url=response.url, body=response.body, encoding='utf-8')
        # - or raise IgnoreRequest
        return response

即在下载网页阶段是将网页转换为utf-8格式，另外需要将中间激活，在配置文件settings.py文件中插入代码，以激活

DOWNLOADER_MIDDLEWARES = {'news.middlewares.NewsDownloaderMiddleware': 1000}

至此，爬虫文件中不需要进行额外的转码，即可正常显示中文了