在使用ImagesPipeline对妹子图网站图片进行下载时,遇到302错误,页面被强制跳转。
解决办法如下:
# -*- coding: utf-8 -*- # Define your item pipelines here # # Don't forget to add your pipeline to the ITEM_PIPELINES setting # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html from scrapy.pipelines.images import ImagesPipeline import scrapy class MeizituPipeline(object): def process_item(self, item, spider): return item class download_pic_pipeline(ImagesPipeline): header = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWeb\ Kit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36', 'Referer': 'http://img.mmjpg.com/' } def get_media_requests(self, item, info): for i in item['image_urls']: yield scrapy.Request(url= i,headers=self.header)
down_load_pic_pipeline类继承了ImagesPipeline类,我们重写get_meidia_requests方法,在请求图片时,加上一个hearders,多次尝试发现参数Referer是该网站用来反爬的关键点,我们添加这个参数,并设置为http://img.mmjpg.com/ ,运行后状态码为200,问题得到解决!