python 爬虫学习篇1
python爬取diameizi网页，然后下载图片
python 环境是2.7.3
代码地址：https://gist.github.com/zjjott/5270366
作者讨论地址：http://tieba.baidu.com/p/2239765168?fr=itb_feed_jing#30880553662l
需要抓的美女图片地址：http://diameizi.diandian.com/
1 #coding=utf-8 2 import os 3 os.system("wget -r --spider http://diameizi.diandian.com 2>|log.txt")#非常简单的抓取整个网页树结构的语句————实质上是一种偷懒 4 filein=open('log.txt','r') 5 fileout=open('dst','w+')#一个装最后的结果的没用的文件 6 filelist=list(filein) 7 import urllib2,time 8 from bs4 import BeautifulSoup 9 header={ 10 'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0.1) Gecko/20100101 Firefox/8.0.1'} 11 def getsite(url): 12 req=urllib2.Request(url,None,header) 13 site=urllib2.urlopen(req) 14 return site.read()##上面这六句基本万金油了。。 15 try: 16 dst=set() 17 for p in filelist: 18 if p.find('http://diameizi.diandian.com/post')>-1: 19 p=p[p.find('http'):] 20 dst.add(p) 21 i=0 22 for p in dst: 23 #if i<191: 24 # i+=1 25 # continue##断点续传部分 26 pagesoup=BeautifulSoup(getsite(p)) 27 pageimg=pagesoup.find_all('img') 28 for href in pageimg: 29 print i,href['src'] 30 picpath="pic/"+href['src'][-55:-13]+href['src'][-4:]##名字的起法有问题。。。不过效果还行。。 31 pic=getsite(href['src']) 32 picfile=open(picpath,'wb') 33 picfile.write(pic) 34 i+=1 35 picfile.close() 36 finally: 37 for p in dst: 38 fileout.write(p) 39 fileout.close()
上面的log.txt
文件大体就是下面的内容。
Spider mode enabled. Check if remote file exists. --2013-03-29 23:00:10-- http://diameizi.diandian.com/ Resolving diameizi.diandian.com (diameizi.diandian.com)... 113.31.29.120, 113.31.29.121 Connecting to diameizi.diandian.com (diameizi.diandian.com)|113.31.29.120|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 30502 (30K) [text/html] Remote file exists and could contain links to other resources -- retrieving. --2013-03-29 23:00:11-- http://diameizi.diandian.com/ Reusing existing connection to diameizi.diandian.com:80. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `diameizi.diandian.com/index.html' 0K .......... .......... ......... 94.6K=0.3s 2013-03-29 23:00:12 (94.6 KB/s) - `diameizi.diandian.com/index.html' saved [30502] Loading robots.txt; please ignore errors. --2013-03-29 23:00:12-- http://diameizi.diandian.com/robots.txt Reusing existing connection to diameizi.diandian.com:80. HTTP request sent, awaiting response... 200 OK Length: 209 [text/plain] Saving to: `diameizi.diandian.com/robots.txt' 0K 100% 20.8M=0s 2013-03-29 23:00:12 (20.8 MB/s) - `diameizi.diandian.com/robots.txt' saved [209/209] Removing diameizi.diandian.com/robots.txt. Removing diameizi.diandian.com/index.html. Spider mode enabled. Check if remote file exists. --2013-03-29 23:00:12-- http://diameizi.diandian.com/rss Reusing existing connection to diameizi.diandian.com:80. HTTP request sent, awaiting response... 200 OK Length: 0 [text/xml] Remote file exists but does not contain any link -- not retrieving. Removing diameizi.diandian.com/rss. unlink: No such file or directory Spider mode enabled. Check if remote file exists. --2013-03-29 23:00:12-- http://diameizi.diandian.com/archive Reusing existing connection to diameizi.diandian.com:80. HTTP request sent, awaiting response... 200 OK Length: 82303 (80K) [text/html] Remote file exists and could contain links to other resources -- retrieving. --2013-03-29 23:00:12-- http://diameizi.diandian.com/archive Reusing existing connection to diameizi.diandian.com:80.
从上面的文本文件中寻找需要的相关资料。
上面的代码还没有测试成功，因为是2.7.3平台的缘故吧。
例子上给的应该是python3.x版本。有些出入
