配置settings ------ (四)
配置和普通的spider是差不多的,具体添加如下:
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" SCHEDULER = "scrapy_redis.scheduler.Scheduler" SCHEDULER_PERSIST = True REDIS_HOST = '*********' REDIS_PORT = 6379 ITEM_PIPELINES = { 'scrapy_redis.pipelines.RedisPipeline': 100 # 'sina.pipelines.SinaPipeline': 300, }
清澈的爱,只为中国