scrapy 修改URL爬取起始位置

import scrapy
from Autopjt.items import myItem
from scrapy.http import Request

class AutospdSpider(scrapy.Spider):
    name = "fulong_spider"
    start_urls = ['http://category.dangdang.com/pg1-cid4007379.html']
    url2 = ('http://dangdang.com','http://jd.com','http://tianmao.com',)

    def start_requests(self):
        for url in self.url2:
            yield self.make_requests_from_url(url)

    def parse(self, response):
        item = myItem()
        item['name'] =response.xpath('/html/head/title/text()').extract()
        print(item['name'])
需要重写start_requests方法
posted @ 2017-05-10 13:15  Erick-LONG  阅读(1678)  评论(0编辑  收藏  举报