scrapy settings

1 scrapy settings  

复制代码
# -*- coding: utf-8 -*-

# Scrapy settings for phone project

BOT_NAME = 'phone'

SPIDER_MODULES = ['phone.spiders']
NEWSPIDER_MODULE = 'phone.spiders'


ROBOTSTXT_OBEY = False

DOWNLOAD_DELAY = 3

COOKIES_ENABLED = False

SPIDER_MIDDLEWARES = {
    # 'phone.middlewares.PhoneSpiderMiddleware': 543,
}

DOWNLOADER_MIDDLEWARES = {
    # 'phone.middlewares.MyCustomDownloaderMiddleware': 543,
}

ITEM_PIPELINES = {
    # 'phone.pipelines.PhonePipeline': 300,
}


SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL = 'redis://root:123456@58.221.49.25:6379'
复制代码

 

posted @   yugengde  阅读(139)  评论(0编辑  收藏  举报
努力加载评论中...
点击右上角即可分享
微信分享提示