beautifulsoup学习过程（一）

Beautiful Soup 是用Python写的一个HTML/XML的解析器，它可以很好的处理不规范标记并生成剖析树(parse tree)。 它提供简单又常用的导航（navigating），搜索以及修改剖析树的操作。它可以大大节省你的编程时间。 对于Ruby，使用Rubyful Soup。

官网摘取内容：

Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. Three features make it powerful:

  1. Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting(解剖) a document and extracting what you need. It doesn't take much code to write an application
  2. Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don't have to think about encodings, unless the document doesn't specify an encoding and Beautiful Soup can't detect one. Then you just have to specify the original encoding.
  3. Beautiful Soup sits on top of popular Python parsers like lxml and html5lib, allowing you to try out different parsing strategies or trade speed for flexibility.

以下内容摘自：http://rsj217.diandian.com/post/2012-11-01/40041235132

使用 BeautifulSoup

1. 导入BeautifulSoup ，创建BeautifulSoup 对象

1
2
3
4
5
6
7
8
9
10
11
12
from BeautifulSoup import BeautifulSoup           # HTML
from BeautifulSoup import BeautifulStoneSoup      # XML
import BeautifulSoup                              # ALL
                                                                                               
doc = [
    '<html><head><title>Page title</title></head>',
    '<body><p id="firstpara" align="center">This is paragraph <b>one</b>.',
    '<p id="secondpara" align="blah">This is paragraph <b>two</b>.',
    '</html>'
]
# BeautifulSoup 接受一个字符串参数
soup = BeautifulSoup(''.join(doc))

2. BeautifulSoup对象简介

用BeautifulSoup 解析 html文档时，BeautifulSoup将 html文档类似 dom文档树一样处理。BeautifulSoup文档树有三种基本对象。

2.1. soup BeautifulSoup.BeautifulSoup

1
2
type(soup)
<class 'BeautifulSoup.BeautifulSoup'>

2.2. 标记 BeautifulSoup.Tag

1
2
type(soup.html)
<class 'BeautifulSoup.Tag'>

2.3 文本 BeautifulSoup.NavigableString

1
2
type(soup.title.string)
<class 'BeautifulSoup.NavigableString'>

3. BeautifulSoup 剖析树

3.1 BeautifulSoup.Tag对象方法

获取 标记对象（Tag）

标记名获取法 ，直接用 soup对象加标记名，返回 tag对象.这种方式，选取唯一标签的时候比较有用。或者根据树的结构去选取，一层层的选择

1
2
3
4
5
6
7
>>> html = soup.html
>>> html
<html><head><title>Page title</title></head><body><p id="firstpara" align="center">This is paragraph<b>one</b>.</p><p id="secondpara" align="blah">This is paragraph<b>two</b>.</p></body></html>
>>> type(html)
<class 'BeautifulSoup.Tag'>
>>> title = soup.title
<title>Page title</title>

content方法

content方法 根据文档树进行搜索，返回标记对象（tag）的列表

1
2
>>> soup.contents
[<html><head><title>Page title</title></head><body><p id="firstpara" align="center">This is paragraph<b>one</b>.</p><p id="secondpara" align="blah">This is paragraph<b>two</b>.</p></body></html>]
1
2
3
4
5
6
>>> soup.contents[0].contents
[<head><title>Page title</title></head>, <body><p id="firstpara" align="center">This is paragraph<b>one</b>.</p><p id="secondpara" align="blah">This is paragraph<b>two</b>.</p></body>]
>>> len(soup.contents[0].contents)
2
>>> type(soup.contents[0].contents[1])
<class 'BeautifulSoup.Tag'>

使用contents向后遍历树，使用parent向前遍历树

next 方法

获取树的子代元素，包括 Tag 对象 和 NavigableString 对象。

1
2
3
4
>>> head.next
<title>Page title</title>
>>> head.next.next
u'Page title'
1
2
3
4
5
>>> p1 = soup.p
>>> p1
<p id="firstpara" align="center">This is paragraph<b>one</b>.</p>
>>> p1.next
u'This is paragraph'

nextSibling 下一个兄弟对象 包括 Tag 对象 和 NavigableString 对象

 

1
2
3
4
>>> head.nextSibling
<body><p id="firstpara" align="center">This is paragraph<b>one</b>.</p><p id="secondpara" align="blah">This is paragraph<b>two</b>.</p></body>
>>> p1.next.nextSibling
<b>one</b>

与 nextSibling 相似的是 previousSibling，即上一个兄弟节点。

replacewith方法

将对象替换为，接受字符串参数

1
2
3
4
5
6
7
8
9
10
11
12
>>> head = soup.head
>>> head
<head><title>Page title</title></head>
>>> head.parent
<html><head><title>Page title</title></head><body><p id="firstpara" align="center">This is paragraph<b>one</b>.</p><p id="secondpara" align="blah">This is paragraph<b>two</b>.</p></body></html>
>>> head.replaceWith('head was replace')
>>> head
<head><title>Page title</title></head>
>>> head.parent
>>> soup
<html>head was replace<body><p id="firstpara" align="center">This is paragraph<b>one</b>.</p><p id="secondpara" align="blah">This is paragraph<b>two</b>.</p></body></html>
>>>

搜索方法

搜索提供了两个方法，一个是 find，一个是findAll。这里的两个方法(findAll和 find)仅对Tag对象以及，顶层剖析对象有效，但 NavigableString不可用。

findAll(name, attrs, recursive, text, limit, **kwargs)

接受一个参数，标记名

寻找文档所有 P标记，返回一个列表

1
2
3
4
>>> soup.findAll('p')
[<p id="firstpara" align="center">This is paragraph<b>one</b>.</p>, <p id="secondpara" align="blah">This is paragraph<b>two</b>.</p>]
>>> type(soup.findAll('p'))
<type 'list'>

寻找 id="secondpara"的 p 标记，返回一个结果集

1
2
3
>>> pid = type(soup.findAll('p',id='firstpara'))
>>> pid
<class 'BeautifulSoup.ResultSet'>

传一个属性或多个属性对

1
2
3
4
5
>>> p2 = soup.findAll('p',{'align':'blah'})
>>> p2
[<p id="secondpara" align="blah">This is paragraph<b>two</b>.</p>]
>>> type(p2)
<class 'BeautifulSoup.ResultSet'>

利用正则表达式

1
2
>>> soup.findAll(id=re.compile("para$"))
[<p id="firstpara" align="center">This is paragraph<b>one</b>.</p>, <p id="secondpara" align="blah">This is paragraph<b>two</b>.</p>]

读取和修改属性

1
2
3
4
5
6
7
8
9
10
11
12
>>> p1 = soup.p
>>> p1
<p id="firstpara" align="center">This is paragraph<b>one</b>.</p>
>>> p1['id']
u'firstpara'
>>> p1['id'] = 'changeid'
>>> p1
<p id="changeid" align="center">This is paragraph<b>one</b>.</p>
>>> p1['class'] = 'new class'
>>> p1
<p id="changeid" align="center" class="new class">This is paragraph<b>one</b>.</p>
>>>

剖析树基本方法就这些，还有其他一些，以及如何配合正则表达式。具体请看官方文档

 

3.2 BeautifulSoup.NavigableString对象方法

NavigableString  对象方法比较简单，获取其内容

1
2
3
4
5
6
7
8
9
>>> soup.title
<title>Page title</title>
>>> title = soup.title.next
>>> title
u'Page title'
>>> type(title)
<class 'BeautifulSoup.NavigableString'>
>>> title.string
u'Page title'

posted on 2014-04-13 12:22  rednuo  阅读(152)  评论(0)  编辑  收藏  举报

导航