一个简单的scrapy爬虫抓取豆瓣刘亦菲的图片地址
一.第一步是创建一个scrapy项目
sh-3.2# scrapy startproject liuyifeiImage
sh-3.2# chmod -R 777 liuyifeiImage/
二.分析图片特征
1.解决分页url部分：
我们爬虫的start_url是"http://movie.douban.com/celebrity/1049732/photos/?type=C&start=0&sortby=vote&size=a&subtype=a",
第二页地址是"http://movie.douban.com/celebrity/1049732/photos/?type=C&start=40&sortby=vote&size=a&subtype=a"，
第三页是"http://movie.douban.com/celebrity/1049732/photos/?type=C&start=80&sortby=vote&size=a&subtype=a",能显而易见得到豆瓣图片的分页规则，因此我们的start_urls可以用一个for循环把所有的页面的url放进来。
start_urls = []; for i in range(0,1120,40): start_urls.append('http://movie.douban.com/celebrity/1049732/photos/ type=C&start=%d&sortby=vote&size=a&subtype=a'%i)
2.解决每一页的图片url部分：
我们在"http://movie.douban.com/celebrity/1049732/photos/?type=C&start=0&sortby=vote&size=a&subtype=a"这一页来分析，审查第一张图片的页面元素
href对应的是每张图的大图地址，而<img src对应的是缩略图地址，我们来看看原图地址链接，
而根据href地址进入的页面图片地址为：
<img src="http://img3.douban.com/view/photo/photo/public/p752034364.jpg">
因此，显而易见，如果想要得到原图地址，只要吧".../view/photo/thumb/public/..."中的"thumb"替换成"photo"或者"raw"即可。
所以spider中的parse部分对应为：
def parse(self,response):
hxs=HtmlXPathSelector(response) sites=hxs.select('//ul/li/div/a/img/@src').extract()
for site in sites:
＃site=site.replace('thumb','photo')
site=site.replace('thumb','raw')
三.保存生成的url列表
在这里用了两种保存方式json和txt
1.先来看看txt保存方式：
f=open('liuyifei_pic_address.txt','wb') def parse(self,response): hxs=HtmlXPathSelector(response) sites=hxs.select('//ul/li/div/a/img/@src').extract() items=[] for site in sites: site=site.replace('thumb','raw') self.f.write(site) self.f.write('\r\n')
2.json保存：
直接在命令行里用参数执行即可：
scrapy crawl liuyifei -o image.json -t json
这样就能把url列表放置在本地文件image.json中，当然，运行scrapy时也是这条命令。
四.接下来，看看这个scrapy的全貌吧，主要修改的文件就是item.py和liuyifei.py(自己创建的spider文件)。
以下是items.py文件
#items.py from scrapy.item import Item,Field class LiuyifeiimageItem(Item): # define the fields for your item here like: # name = scrapy.Field() ImageAddress = Field() pass
以下是liuyifei.py文件：
＃liuyifei.py from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from liuyifeiImage.items import LiuyifeiimageItem '''http://movie.douban.com/celebrity/1049732/photos/''' class liuyifeiImage(BaseSpider): name='liuyifei' allowed_domain=["douban.com"] start_urls=[] f=open('liuyifei_pic_address.txt','wb') for i in range(0,1120,40): start_urls.append('http://movie.douban.com/celebrity/1049732/photos/?type=C&start=%d&sortby=vote&size=a&subtype=a'%i) def parse(self,response): hxs=HtmlXPathSelector(response) sites=hxs.select('//ul/li/div/a/img/@src').extract() items=[] for site in sites: site=site.replace('thumb','raw') self.f.write(site) self.f.write('\r\n') item=LiuyifeiimageItem() item['ImageAddress']=site items.append(item) return items
最后，运行scrapy,以下是部分打印结果。
sh-3.2# scrapy crawl liuyifei -o image.json -t json /Users/lsf/PycharmProjects/liuyifeiImage/liuyifeiImage/spiders/liuyifei.py:8: ScrapyDeprecationWarning: liuyifeiImage.spiders.liuyifei.liuyifeiImage inherits from deprecated class scrapy.spider.BaseSpider, please inherit from scrapy.spider.Spider. (warning only on first subclass, there may be others) class liuyifeiImage(BaseSpider): 2014-10-04 12:57:37+0800 [scrapy] INFO: Scrapy 0.24.4 started (bot: liuyifeiImage) 2014-10-04 12:57:37+0800 [scrapy] INFO: Optional features available: ssl, http11 2014-10-04 12:57:37+0800 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'liuyifeiImage.spiders', 'FEED_FORMAT': 'json', 'SPIDER_MODULES': ['liuyifeiImage.spiders'], 'FEED_URI': 'image.json', 'BOT_NAME': 'liuyifeiImage'} 2014-10-04 12:57:37+0800 [scrapy] INFO: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState 2014-10-04 12:57:37+0800 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 2014-10-04 12:57:37+0800 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 2014-10-04 12:57:37+0800 [scrapy] INFO: Enabled item pipelines: 2014-10-04 12:57:37+0800 [liuyifei] INFO: Spider opened 2014-10-04 12:57:37+0800 [liuyifei] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2014-10-04 12:57:37+0800 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6026 2014-10-04 12:57:37+0800 [scrapy] DEBUG: Web service listening on 127.0.0.1:6083 2014-10-04 12:57:38+0800 [liuyifei] DEBUG: Crawled (200) <GET http://movie.douban.com/celebrity/1049732/photos/?type=C&start=240&sortby=vote&size=a&subtype=a> (referer: None) /Users/lsf/PycharmProjects/liuyifeiImage/liuyifeiImage/spiders/liuyifei.py:17: ScrapyDeprecationWarning: scrapy.selector.HtmlXPathSelector is deprecated, instantiate scrapy.Selector instead. hxs=HtmlXPathSelector(response) /Users/lsf/PycharmProjects/liuyifeiImage/liuyifeiImage/spiders/liuyifei.py:18: ScrapyDeprecationWarning: Call to deprecated function select. Use .xpath() instead. sites=hxs.select('//ul/li/div/a/img/@src').extract() /Library/Python/2.7/site-packages/Scrapy-0.24.4-py2.7.egg/scrapy/selector/unified.py:106: ScrapyDeprecationWarning: scrapy.selector.HtmlXPathSelector is deprecated, instantiate scrapy.Selector instead. for x in result] 2014-10-04 12:57:38+0800 [liuyifei] DEBUG: Scraped from <200 http://movie.douban.com/celebrity/1049732/photos/?type=C&start=240&sortby=vote&size=a&subtype=a> {'ImageAddress': u'http://img3.douban.com/view/photo/raw/public/p2179423125.jpg'} 2014-10-04 12:57:38+0800 [liuyifei] DEBUG: Scraped from <200 http://movie.douban.com/celebrity/1049732/photos/?type=C&start=240&sortby=vote&size=a&subtype=a> {'ImageAddress': u'http://img3.douban.com/view/photo/raw/public/p2179423105.jpg'} 2014-10-04 12:57:38+0800 [liuyifei] DEBUG: Scraped from <200 http://movie.douban.com/celebrity/1049732/photos/?type=C&start=240&sortby=vote&size=a&subtype=a> {'ImageAddress': u'http://img3.douban.com/view/photo/raw/public/p2179423084.jpg'} ... 2014-10-04 13:34:17+0800 [liuyifei] DEBUG: Scraped from <200 http://movie.douban.com/celebrity/1049732/photos/?type=C&start=1040&sortby=vote&size=a&subtype=a> {'ImageAddress': u'http://img3.douban.com/view/photo/raw/public/p958573512.jpg'} 2014-10-04 13:34:17+0800 [liuyifei] DEBUG: Scraped from <200 http://movie.douban.com/celebrity/1049732/photos/?type=C&start=1040&sortby=vote&size=a&subtype=a> {'ImageAddress': u'http://img5.douban.com/view/photo/raw/public/p958572938.jpg'} 2014-10-04 13:34:17+0800 [liuyifei] INFO: Closing spider (finished) 2014-10-04 13:34:17+0800 [liuyifei] INFO: Stored json feed (1120 items) in: image.json 2014-10-04 13:34:17+0800 [liuyifei] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 8331, 'downloader/request_count': 28, 'downloader/request_method_count/GET': 28, 'downloader/response_bytes': 221405, 'downloader/response_count': 28, 'downloader/response_status_count/200': 28, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2014, 10, 4, 5, 34, 17, 736723), 'item_scraped_count': 1120, 'log_count/DEBUG': 1150, 'log_count/INFO': 8, 'response_received_count': 28, 'scheduler/dequeued': 28, 'scheduler/dequeued/memory': 28, 'scheduler/enqueued': 28, 'scheduler/enqueued/memory': 28, 'start_time': datetime.datetime(2014, 10, 4, 5, 34, 14, 681268)} 2014-10-04 13:34:17+0800 [liuyifei] INFO: Spider closed (finished)
以下是json文件和txt文件：
image.json:
liuyifei_pic_address.txt
