Selenimu做爬虫

前面有介绍过Selenium作为开源的自动化测试框架，其实Selenium还可以作为爬虫工具。

举个例子，我想爬取中国概念股的一些信息，例如这样的：http://app.finance.ifeng.com/list/usstock_cn.php

先建个SeleniumUtil.py的文件

#!/usr/bin/python2.5.2
#-*- coding: utf8 -*-

from selenium import selenium

class SeleniumUtil:
  seleniums = {}
  
  def StartSeleniumForUrl(self, url):
    sel = selenium("localhost", 4444,"*chrome", url)
    sel.start()
    sel.set_timeout("90000")
    return sel

  def GetSelenium(self, websiteName, url):
    #if websiteName in self.seleniums:
     # return self.seleniums[websiteName]
    self.seleniums[websiteName] = self.StartSeleniumForUrl(url)
    return self.seleniums[websiteName]
  
  def StopSelenium(self, websiteName):
    if websiteName in self.seleniums:
      self.seleniums[websiteName].stop()
    

 

接着写个IfengFinanceSite.py文件来爬取需要的信息，例子如下，

#!/usr/bin/python2.4
# -*- coding: utf8 -*-

#Ifeng Site
#

import codecs,time
from SeleniumUtil import SeleniumUtil

ExchangeUrlMap = {
  "CCS": "http://app.finance.ifeng.com/list/usstock_cn.php",
  #"SHA_B": "http://stock.finance.sina.com.cn/stock/quote/shb%s.html",
}
ExchangeXPath = "//html/body/div[4]/div/div[2]/div/table/tbody/tr[%s]/td[1]"


class IfengFinanceSite:

  # result file for diff exchanges
  def GetAllTickers(self, exchange, resultFiles):
    sln = SeleniumUtil().GetSelenium("Ifeng", "http://app.finance.ifeng.com/")
    myfile=codecs.open(resultFiles % exchange, 'w', 'utf-8')

    count = 0

    sln.open(ExchangeUrlMap[exchange])
    time.sleep(5)
      
    for j in range(2, 200):
        if sln.is_element_present(ExchangeXPath % j):
          context = sln.get_text(ExchangeXPath % j).strip()
          print >> myfile, context, '\r'
          count = count + 1
        else: break
    print "%s companies for exchange %s recorded." % (count, exchange)
    sln.stop()

exchangelist=["CCS"] #"SHA_A","SHA_B","SHE_A","SHE_B","SHA_Q","SHA_CEF","SHE_CEF","SHA_Bond","SHE_Bond"
for exchange in exchangelist:    
    print exchange
    resultFiles="Ifeng_company_list_%s.txt"    
    IfengFinanceSite().GetAllTickers(exchange, resultFiles)

 

之后启动Selenium服务，调动浏览器就能获取所有中国概念股股票代码，如果还需要其他的信息如名称、价格，只需要取得Xpath就行了。

再扩展开就是可以用Selenium做diff工具了，前后版本的对比，自有产品与竞争对手信息的对比。

当然，Selenium RC因为要调出浏览器，所以效率还是很一般，可以考虑用Selenium其他产品。