Python爬虫之PyQuery使用（六）

Python爬虫之PyQuery使用

 

PyQuery简介

pyquery能够通过选择器精确定位 DOM 树中的目标并进行操作。pyquery相当于jQuery的python实现，可以用于解析HTML网页等。它的语法与jQuery几乎完全相同，对于使用过jQuery的人来说很熟悉，也很好上手。

 

初始化

有 4 种方法可以进行初始化：
可以通过传入 字符串、lxml、文件 或者 url 来使用PyQuery

?
1
2
3
4
5
6
7
from pyquery import PyQuery as pq
from lxml import etree
 
d = pq("<html></html>")#传入字符串
d = pq(etree.fromstring("<html></html>"))#传入lxml
d = pq(url='http://baidu.com/') #传入url
d = pq(filename=path_to_html_file) #传入文件

　　

 

基本CSS选择器

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
html='''
<html>
 <body>
  <ul class="mh-col">
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-1"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     歌手高空拍MV坠亡
    </a>
    <span class="mh-ico-up">
    </span>
   </li>
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-2"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">
     佛祖朱龙广金婚
    </a>
    <span class="mh-ico-down">
    </span>
   </li>
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-3"}' href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">
     常见租房陷阱
    </a>
    <span class="mh-ico-up">
    </span>
   </li>
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-4"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     靳东回应发错诗词
    </a>
   </li>
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-5"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     煤老板们的影视江湖
    </a>
   </li>
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-6"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     1024程序员节
    </a>
   </li>
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-7"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     美的合并小天鹅
    </a>
   </li>
   <li class="g-ellipsis">
    <a class="g-a-noline" data-md='{"b":"list","p":"1-8"}' href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     京昆高速4车相撞
    </a>
   </li>
 
  </ul>
 </body>
</html>
'''<br><br><br>

from pyquery import PyQuery as pq
doc = pq(html)

# 获取所有a标签
print(doc('body .mh-col li a'))


注意：
类名用.
id用#
标签用标签名

另外选择的是具有层级关系，从左到右，不是直接的父子的关系。

 

运行结果如下：

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<a class="g-a-noline" data-md="{"b":"list","p":"1-1"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E6%AD%8C%E6%89%8B%E9%AB%98%E7%A9%BA%E6%8B%8DMV%E5%9D%A0%E4%BA%A1&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     歌手高空拍MV坠亡
    </a>
    <a class="g-a-noline" data-md="{"b":"list","p":"1-2"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BD%9B%E7%A5%96%E6%9C%B1%E9%BE%99%E5%B9%BF%E9%87%91%E5%A9%9A&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D&fr=hao_360so_history_b" target="_blank">
     佛祖朱龙广金婚
    </a>
    <a class="g-a-noline" data-md="{"b":"list","p":"1-3"}" href="http://www.so.com/link?url=http%3A%2F%2Fbaike.so.com%2Fzt%2Fzufangfangpian.html%3Fsrc%3Dreci&q=%E5%91%A8%E6%9D%B0%E4%BC%A6%E7%9A%84%E6%AD%8C&ts=1540365398&t=c6890eee0e669832ca96d5223582d6e" target="_blank">
     常见租房陷阱
    </a>
    <a class="g-a-noline" data-md="{"b":"list","p":"1-4"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E9%9D%B3%E4%B8%9C%E5%9B%9E%E5%BA%94%E5%8F%91%E9%94%99%E8%AF%97%E8%AF%8D&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     靳东回应发错诗词
    </a>
   <a class="g-a-noline" data-md="{"b":"list","p":"1-5"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%85%A4%E8%80%81%E6%9D%BF%E4%BB%AC%E7%9A%84%E5%BD%B1%E8%A7%86%E6%B1%9F%E6%B9%96&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     煤老板们的影视江湖
    </a>
   <a class="g-a-noline" data-md="{"b":"list","p":"1-6"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=1024%E7%A8%8B%E5%BA%8F%E5%91%98%E8%8A%82&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     1024程序员节
    </a>
   <a class="g-a-noline" data-md="{"b":"list","p":"1-7"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E7%BE%8E%E7%9A%84%E5%90%88%E5%B9%B6%E5%B0%8F%E5%A4%A9%E9%B9%85&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     美的合并小天鹅
    </a>
   <a class="g-a-noline" data-md="{"b":"list","p":"1-8"}" href="https://www.so.com/s?ie=utf-8&src=know_side_nlp_sohot&q=%E4%BA%AC%E6%98%86%E9%AB%98%E9%80%9F4%E8%BD%A6%E7%9B%B8%E6%92%9E&ob_ext=%7B%22rsv_cq%22%3A%22%5Cu5468%5Cu6770%5Cu4f26%5Cu7684%5Cu6b4c%22%2C%22rt%22%3A%22%5Cu5b9e%5Cu65f6%5Cu70ed%5Cu641c%22%2C%22rclken%22%3Anull%7D" target="_blank">
     京昆高速4车相撞
    </a>
   

　　

操作

?
1
.find()<br>#查找父元素 <br>#注意：一个元素只有一个父元素 <br>body = items.parent() <br>print(body) <br><br><br>#查找祖先元素 <br>content = items.parents() <br>print(content) <br><br>#查找兄弟元素 <br>li = doc('.mh-col .g-ellipsis1') <br>print(li.siblings()) <br><br>#遍历 单个元素 <br>#遍历所有的a标签<br>alist =doc('.mh-col li a').items() <br>　　for a in alist: print(a)

　　

 

获取信息

 

?
1
2
3
4
5
6
7
8
9
10
11
12
13
获取属性
a =doc('.mh-col li .g-a-noline8')
print(a.attr['href'])
print(a.attr.href)
 
 
获取文本
a =doc('.mh-col li .g-a-noline8')
print(a.text())
 
获取HTML
a =doc('.mh-col li .g-a-noline8')
print(a.html())

　

简单的DOM操作

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#addClass、removerClass<br>#修改类名
a =doc('.mh-col li .g-a-noline8')
print(a)
a.removeClass('g-a-noline8')
print(a)
a.addClass('g-a-noline8')
print(a)
 
 
#attr、css<br>#修改属性和样式
a =doc('.mh-col li .g-a-noline8')
print(a)
a.attr('name','link')
print(a)
a.css('font-size','14px')
print(a)
 
#remove<br>#删除标签<br>li = doc('.mh-col .g-ellipsis8')<br>print(li)<br>li.find('a').remove()<br>print(li)

　　

更多的DOM操作：https://pyquery.readthedocs.io/en/latest/api.html

 

posted @   -零  阅读(1063)  评论(0)  编辑  收藏  举报
编辑推荐：
· 从 HTTP 原因短语缺失研究 HTTP/2 和 HTTP/3 的设计差异
· AI与.NET技术实操系列：向量存储与相似性搜索在 .NET 中的实现
· 基于Microsoft.Extensions.AI核心库实现RAG应用
· Linux系列：如何用heaptrack跟踪.NET程序的非托管内存泄露
· 开发者必知的日志记录最佳实践
阅读排行：
· TypeScript + Deepseek 打造卜卦网站：技术与玄学的结合
· Manus的开源复刻OpenManus初探
· AI 智能体引爆开源社区「GitHub 热点速览」
· 三行代码完成国际化适配，妙~啊~
· .NET Core 中如何实现缓存的预热？
点击右上角即可分享
微信分享提示
SQL AI 助手