Web::Scraper 页面提取分析

一组用来提取HTML文档中元素内容的工具集，它能够理解HTML和CSS选择器以及XPath表达式。

语法

use URI;
  use Web::Scraper;

  # First, create your scraper block
  my $tweets = scraper {
      # Parse all LIs with the class "status", store them into a resulting
      # array 'tweets'.  We embed another scraper for each tweet.
      process "li.status", "tweets[]" => scraper {
          # And, in that array, pull in the elementy with the class
          # "entry-content", "entry-date" and the link
          process ".entry-content", body => 'TEXT';
          process ".entry-date", when => 'TEXT';
          process 'a[rel="bookmark"]', link => '@href';
      };
  };

  my $res = $tweets->scrape( URI->new("http://twitter.com/miyagawa") );

  # The result has the populated tweets array
  for my $tweet (@{$res->{tweets}}) {
      print "$tweet->{body} $tweet->{when} (link: $tweet->{link})\n";
  }

 

process

scraper {
      process "tag.class", key => 'TEXT';
      process '//tag[contains(@foo, "bar")]', key2 => '@attr';
      process '//comment()', 'comments[]' => 'TEXT';
  };

如果你传入的参数是URI或HTTP response，那Web::Scaper自动去寻找Content-Type header和META标签以判断文件编码。否则你压根先把HTML内容解码为Unicode后再传给scape函数。

  $res = $scraper->scrape(URI->new($uri));
  $res = $scraper->scrape($html_content);
  $res = $scraper->scrape(\$html_content);
  $res = $scraper->scrape($http_response);
  $res = $scraper->scrape($html_element);

当你把HTML内容作为参数传给scrape函数时，你还要考虑一个问题：HTML文档中出现 相对路径怎么办？所以这个时候你可以把base url一并作为参数传进去。

res= scraper->scrape($html_content, "http://example.com/foo");

 

它有两个参数，当第一个参数以"//"或"id("开头时作为XPath对待；否则作为HTML或CSS选择器对待。

# <span class="date">2008/12/21</span>
  # date => "2008/12/21"
  process ".date", date => 'TEXT';

  # <div class="body"><a href="http://example.com/">foo</a></div>
  # link => URI->new("http://example.com/")
  process ".body > a", link => '@href';

  # <div class="body"><!-- HTML Comment here --><a href="http://example.com/">foo</a></div>
  # comment => " HTML Comment here "
  #
  # NOTES: A comment nodes are accessed when installed
  # the HTML::TreeBuilder::XPath (version >= 0.14) and/or
  # the HTML::TreeBuilder::LibXML (version >= 0.13)
  process "//div[contains(@class, 'body')]/comment()", comment => 'TEXT';

  # <div class="body"><a href="http://example.com/">foo</a></div>
  # link => URI->new("http://example.com/"), text => "foo"
  process ".body > a", link => '@href', text => 'TEXT';

  # <ul><li>foo</li><li>bar</li></ul>
  # list => [ "foo", "bar" ]
  process "li", "list[]" => "TEXT";

  # <ul><li id="1">foo</li><li id="2">bar</li></ul>
  # list => [ { id => "1", text => "foo" }, { id => "2", text => "bar" } ];
  process "li", "list[]" => { id => '@id', text => "TEXT" };