scrapy爬取后中文乱码,解决word转为html 时cp1252编码问题
解决思路1、
循环暴力寻找编码,但是不如思路3
def parse(self, response): print(response.text[:100]) body = response.body#直接是bytes,response.text是str encodings = ['utf-8', 'gbk', 'gb2312', 'iso-8859-1', 'latin1']#实际可用response.encoding获得 for encoding in encodings: try: print(body.decode(encoding)[:100])#decode必须是bytes except Exception as e: print('decode {0}, error: {1}\n'.format(encoding, e)) pass
解决思路2、
download minddlewares中有个process_response方法,修改它的encoding即可需要自己组装一个修改了charset的页面response,利用HtmlResponse可以完美解决,实现中文乱码的解决,当然你要在setting.py中启用该download middleware
from scrapy.http import HtmlResponse def process_response(self,request, response, spider): # 修改页面编码 if response.encoding == 'cp1252': response = HtmlResponse(url=response.url, body=response.body, encoding='utf-8') return response
解决思路3、
scrapy爬取编码为gb2312的网页时出现中文乱码
python3用库chardet查看编码方式;先用encode编码成bytes,再用decode编码成str
import chardet txt_b=response.xpath('//title').extract()[0].encode(response.encoding)#对找到的具体内容用encode编码成bytes print(chardet.detect(txt_b)) txt_str= txt_b.decode(response.encoding,errors='ignore')#对bytes内容用decode解码成str print(txt_str)