爬了个爬(一)爬虫入门
一、爬虫定义
网络爬虫:(又被称为网页蜘蛛,网络机器人,在FOFA社区中间,更经常的称为网页追逐者),是一种按照一定的规则,自动的抓取万维网信息的程序或者脚本。
URI(Uniform Resource Identifier 统一资源标识符)一般由三部分组成:1.访问资源的命名机制(方案名、协议)2.存放资源的主机名(域名)3.资源自身的名称(路径)。
例:http://www.cisco.com/en/US/partners/index.html 方案名 (http) 域名 (www.cisco.com) 路径 (/en/US/partners/index.html)
URL是Uniform Resource Locator的缩写,译为“统一资源定位符”:URL的格式由下列三部分组成:
第一部分是协议(或称为服务方式);
第二部分是存有该资源的主机IP地址(有时也包括端口号);
第三部分是主机资源的具体地址。,如目录和文件名等。
第一部分和第二部分之间用“://”符号隔开,第二部分和第三部分用“/”符号隔开。第一部分和第二部分是不可缺少的,第三部分有时可以省略。
URL示例:
文件的URL: 用URL表示文件时,服务器方式用file表示,后面要有主机IP地址、文件的存取路径(即目录)和文件名等信息。有时可以省略目录和文件名,但“/”符号不能省略。 例:file://a:1234/b/c/d.txt 代表获取资源使用ftp协议,资源目标是a主机的1234端口的b目录下的c目录下的d.txt。
爬虫基本操作(简单概述)
URL指定内容获取到
- 发送Http请求:http://www.autohome.com.cn/news/
- 基于正则表达式获取内容
1)非定向爬虫,在互联网上广泛爬取,例如:各种搜索门户的网站
2)定向爬虫,针对性爬一个网站,例如:爬取煎蛋网的图片
二、Python实现爬虫
利用两个模块 requests,beatifulsoup;requests用于发送请求,beatifulsoup用于对获取的信息转换成标签。
requests
Python标准库中提供了:urllib、urllib2、httplib等模块以供Http请求,但是,它的 API 太渣了。它是为另一个时代、另一个互联网所创建的。它需要巨量的工作,甚至包括各种方法覆盖,来完成最简单的任务。
Requests 是使用 Apache2 Licensed 许可证的 基于Python开发的HTTP 库,其在Python内置模块的基础上进行了高度的封装,从而使得Pythoner进行网络请求时,变得美好了许多,使用Requests可以轻而易举的完成浏览器可有的任何操作。
#requests 基本使用 obj = requests.get("url") #获取html字符串信息 obj.encoding = "gbk" #自定编码类型 默认为utf8 ----> obj.encoding obj.content #转换为字节类型的数据 obj.text #转换为字符串类型的数据 obj.cookies #获取cookies obj.cookies.get_dict() #获取字典类型的cookies requests.get("url",cookies={'k1':"v1"}) #更对参数详见下边讲解;不同提交方式的参数类似!
1、GET请求
# 1、无参数实例 import requests ret = requests.get('https://github.com/timeline.json') print(ret.url) print(ret.text) # 2、有参数实例 实在url地址上传参。问号写入?key=value&key=value import requests payload = {'key1': 'value1', 'key2': 'value2'} ret = requests.get("http://httpbin.org/get", params=payload) print(ret.url) print(ret.text)
2、POST请求
# 1、基本POST实例 import requests payload = {'key1': 'value1', 'key2': 'value2'} ret = requests.post("http://httpbin.org/post", data=payload) print(ret.text) # 2、发送请求头和数据实例 import requests import json url = 'https://api.github.com/some/endpoint' payload = {'some': 'data'} headers = {'content-type': 'application/json'} ret = requests.post(url, data=json.dumps(payload), headers=headers) print(ret.text) print(ret.cookies)
3、其他请求
requests.get(url, params=None, **kwargs) requests.post(url, data=None, json=None, **kwargs) requests.put(url, data=None, **kwargs) requests.head(url, **kwargs) requests.delete(url, **kwargs) requests.patch(url, data=None, **kwargs) requests.options(url, **kwargs) # 以上方法均是在此方法的基础上构建 requests.request(method, url, **kwargs)
4、更对参数
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
def request(method, url, **kwargs): """Constructs and sends a :class:`Request <Request>`. :param method: method for the new :class:`Request` object. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`. :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`. :param json: (optional) json data to send in the body of the :class:`Request`. :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`. :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`. :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload. ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')`` or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers to add for the file. :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth. :param timeout: (optional) How long to wait for the server to send data before giving up, as a float, or a :ref:`(connect timeout, read timeout) <timeouts>` tuple. :type timeout: float or tuple :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed. :type allow_redirects: bool :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy. :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``. :param stream: (optional) if ``False``, the response content will be immediately downloaded. :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair. :return: :class:`Response <Response>` object :rtype: requests.Response Usage:: >>> import requests >>> req = requests.request('GET', 'http://httpbin.org/get') <Response [200]> """ 参数列表
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
def param_method_url(): # requests.request(method='get', url='http://127.0.0.1:8000/test/') # requests.request(method='post', url='http://127.0.0.1:8000/test/') pass def param_param(): # - 可以是字典 # - 可以是字符串 # - 可以是字节(ascii编码以内) # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params={'k1': 'v1', 'k2': '水电费'}) # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params="k1=v1&k2=水电费&k3=v3&k3=vv3") # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8')) # 错误 # requests.request(method='get', # url='http://127.0.0.1:8000/test/', # params=bytes("k1=v1&k2=水电费&k3=v3&k3=vv3", encoding='utf8')) pass def param_data(): # 可以是字典 # 可以是字符串 # 可以是字节 # 可以是文件对象 # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data={'k1': 'v1', 'k2': '水电费'}) # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data="k1=v1; k2=v2; k3=v3; k3=v4" # ) # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data="k1=v1;k2=v2;k3=v3;k3=v4", # headers={'Content-Type': 'application/x-www-form-urlencoded'} # ) # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # data=open('data_file.py', mode='r', encoding='utf-8'), # 文件内容是:k1=v1;k2=v2;k3=v3;k3=v4 # headers={'Content-Type': 'application/x-www-form-urlencoded'} # ) pass def param_json(): # 将json中对应的数据进行序列化成一个字符串,json.dumps(...) # 然后发送到服务器端的body中,并且Content-Type是 {'Content-Type': 'application/json'} requests.request(method='POST', url='http://127.0.0.1:8000/test/', json={'k1': 'v1', 'k2': '水电费'}) def param_headers(): # 发送请求头到服务器端 requests.request(method='POST', url='http://127.0.0.1:8000/test/', json={'k1': 'v1', 'k2': '水电费'}, headers={'Content-Type': 'application/x-www-form-urlencoded'} ) def param_cookies(): # 发送Cookie到服务器端 requests.request(method='POST', url='http://127.0.0.1:8000/test/', data={'k1': 'v1', 'k2': 'v2'}, cookies={'cook1': 'value1'}, ) # 也可以使用CookieJar(字典形式就是在此基础上封装) from http.cookiejar import CookieJar from http.cookiejar import Cookie obj = CookieJar() obj.set_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False, port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False) ) requests.request(method='POST', url='http://127.0.0.1:8000/test/', data={'k1': 'v1', 'k2': 'v2'}, cookies=obj) def param_files(): # 发送文件 # file_dict = { # 'f1': open('readme', 'rb') # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) # 发送文件,定制文件名 # file_dict = { # 'f1': ('test.txt', open('readme', 'rb')) # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) # 发送文件,定制文件名 # file_dict = { # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf") # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) # 发送文件,定制文件名 # file_dict = { # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', {'k1': '0'}) # } # requests.request(method='POST', # url='http://127.0.0.1:8000/test/', # files=file_dict) pass def param_auth(): from requests.auth import HTTPBasicAuth, HTTPDigestAuth ret = requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf')) print(ret.text) # ret = requests.get('http://192.168.1.1', # auth=HTTPBasicAuth('admin', 'admin')) # ret.encoding = 'gbk' # print(ret.text) # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass')) # print(ret) # def param_timeout(): # ret = requests.get('http://google.com/', timeout=1) # print(ret) # ret = requests.get('http://google.com/', timeout=(5, 1)) # print(ret) pass def param_allow_redirects(): ret = requests.get('http://127.0.0.1:8000/test/', allow_redirects=False) print(ret.text) def param_proxies(): # proxies = { # "http": "61.172.249.96:80", # "https": "http://61.185.219.126:3128", # } # proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'} # ret = requests.get("http://www.proxy360.cn/Proxy", proxies=proxies) # print(ret.headers) # from requests.auth import HTTPProxyAuth # # proxyDict = { # 'http': '77.75.105.165', # 'https': '77.75.105.165' # } # auth = HTTPProxyAuth('username', 'mypassword') # # r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth) # print(r.text) pass def param_stream(): ret = requests.get('http://127.0.0.1:8000/test/', stream=True) print(ret.content) ret.close() # from contextlib import closing # with closing(requests.get('http://httpbin.org/get', stream=True)) as r: # # 在此处理响应。 # for i in r.iter_content(): # print(i) def requests_session(): import requests session = requests.Session() ### 1、首先登陆任何页面,获取cookie i1 = session.get(url="http://dig.chouti.com/help/service") ### 2、用户登陆,携带上一次的cookie,后台对cookie中的 gpsd 进行授权 i2 = session.post( url="http://dig.chouti.com/login", data={ 'phone': "8615131255089", 'password': "xxxxxx", 'oneMonth': "" } ) i3 = session.post( url="http://dig.chouti.com/link/vote?linksId=8589623", ) print(i3.text) 参数示例
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
method 请求方法 url 请求路径 params get请求方式传参 data 传值,需要在外部转成字符串或bytes json 传值,自动json转换 headers 请求头 cookies:cookies files 文件 用于上传文件 对应元组(文件名,文件对象) auth 基本的验证规则。API认证定制请求头,公用的认证规则,统一规范! 例:配置家用路由,输入网址登录时候的登录弹窗,后台是把输入的内容以get传参的方法传递。 timeout: 请求or连接需要时间,有耗时。连接等待时间 参数: float:数据返回的时间 tuple:(连接超时时间,数据返回时间) allow_redirects 是否允许重定向;布尔值判定。 proxies 代理 委派别人发送请求一般常用代理IP地址,目的:防止IP被封 stream:数据流方式获取,以防突然大文件撑爆内存。实现边发边取;对应写入文件方法:iter_content()一点一点的写入文件 #https://..... 证书类型的HTML路径,访问前需要先进行安全认证! verify 是否携带证书 cert 证书文件 爬取数据:是否要带证书?True,带;False,不带 知乎,百度,可带可不带 老男孩;必须带
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
#import requests # 1. 调用关系 # requests.get() # requests.post() # requests.put() # requests.request('post') # 2.常用参数 # url = 'xxx', # params = {'k1': 'v1', 'nid': 888}, GET # cookies = {}, # headers = {}, # data = {}, # json = {} # requests.get( # url='xxx', # params={'k1':'v1','nid':888}, # cookies={}, # headers={} # ) # http://www.baidu.com?k1=v2&nid=888 # requests.post( # url='xxx', # params={'k1':'v1','nid':888}, # cookies={}, # headers={}, # data={}, # json={} # ) # 注意: 请求头,POST方式发送时会用上 # application/x-www-form-urlencoded, request.POST # requests.post(url='',data={},headers={'content-type': 'application/json'}) # requests.post(url='',json={}) # {'content-type': 'application/json'} # def param_auth(): #from requests.auth import HTTPBasicAuth, HTTPDigestAuth #两种认证规则类,区别在加密上 # ret = requests.get("http://192.168.1.1", auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf'))#类封装成对象!可以看源码,传递有加密解密操作! # ret = requests.get("http://192.168.1.1", auth=HTTPDigestAuth('wupeiqi', 'sdfasdfasdf')) # ret = requests.get("http://192.168.1.1", headers={'Authorization': "asdfasdfasdfasf"}) # print(ret.text) # ret = requests.get('http://192.168.1.1', # auth=HTTPBasicAuth('admin', 'admin')) # ret.encoding = 'gbk' # print(ret.text) # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass')) # print(ret) # # response = requests.get('http://www.adc.com',allow_redirects=True) # print(response.text) # http://www.adc.com # print(response.text) # http://www.baidu.com # # response = requests.get('url',stream=True) # with open('') as f: # f.write('xxx') # for line in response.iter_content(): # pass # # response.close() # # from contextlib import closing # with closing(requests.get('http://httpbin.org/get', stream=True)) as r: # # 在此处理响应。 # for i in r.iter_content(): # print(i) # 知乎,可带可不带,【知乎,xxxx】 # True,带 # False,不太 # 老男孩,必须 # requests.get('http://httpbin.org/get', stream=True,cert="xxxx.pem") from bs4.element import Tag # session,容器
BeautifulSoup
BeautifulSoup是一个模块,该模块用于接收一个HTML或XML字符串,然后将其进行格式化,之后遍可以使用他提供的方法进行快速查找指定元素,从而使得在HTML或XML中查找指定元素变得简单。
安装:
pip3 install beautifulsoup4
#beatifulsoup基本使用 soup = beatifulsoup(obj.text,'html.parser') 标签 = soup.find(name='xx') #获取单个标签 默认找到第一个 [标签,] = soup.find_all(...) #获取所有标签对象,以列表方式存储 标签.text #获取标签的文本 标签.attrs #获取标签属性 标签.get(...) #指定获取标签某个属性的属性值
使用示例:
#导入模块 from bs4 import BeautifulSoup #找到一段HTML字符串,可以通过request方式获取 html_doc = """ <html><head><title>The Dormouse's story</title></head> <body> asdf <div class="title"> <b>The Dormouse's story总共</b> <h1>f</h1> </div> <div class="story">Once upon a time there were three little sisters; and their names were <a class="sister0" id="link1">Els<span>f</span>ie</a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</div> ad<br/>sf <p class="story">...</p> </body> </html> """
#解析 html
# html.parser 内置
# lxml 需要先安装模块 性能高
soup = BeautifulSoup(html_doc, features="html.parser") #得到的全是标签对象 # 找到第一个a标签 tag1 = soup.find(name='a') # 找到所有的a标签 tag2 = soup.find_all(name='a') # 找到id=link2的标签 tag3 = soup.find(id="link2") or tag3 = soup.select('#link2')
BeautifulSoup模块中提供的方法
1. name,获取标签名称 or 设置
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('a') name = tag.name # 获取 print(name) tag.name = 'span' # 设置 print(tag
2. attrs,标签属性【获取所有 or 赋值】;get【指定获取某个属性】;del tag.attrs["kkkk"] 删除某一条属性
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('a') attrs = tag.attrs # 获取 print(attrs) tag.attrs = {'ik':123} # 设置 tag.attrs['id'] = 'iiiii' # 设置 print(tag)
3. children,所有子标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
body = soup.find('body') v = body.children print(v)
4.descendants,所有子子孙孙标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
body = soup.find('body') v = body.descendants
5. clear,将标签的所有子标签全部清空(保留标签名)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('body') tag.clear() print(soup)
6. decompose,递归的删除所有的标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
body = soup.find('body') body.decompose() print(soup)
7. extract,删除指定标签,并获取删除的标签(删除标签有返回值)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
taga = tag.find(name="a") taga.extract() print(tag)
8. decode,转换为字符串(含当前标签);decode_contents(不含当前标签)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
body = soup.find('body') v = body.decode() #v = body.decode_contents() print(v)
9. encode,转换为字节(含当前标签);encode_contents(不含当前标签)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
body = soup.find('body') v = body.encode() v = body.encode_contents() print(v)
10. find,获取匹配的第一个标签。属性(recursive 递归 text文本)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('a') print(tag) tag = soup.find(name='a', attrs={'class': 'sister'}, recursive=True, text='Lacie') tag = soup.find(name='a', class_='sister', recursive=True, text='Lacie') print(tag)
11. find_all,获取匹配的所有标签。属性(limit 限制,只找几个)
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tags = soup.find_all('a') print(tags) tags = soup.find_all('a',limit=1) print(tags) tags = soup.find_all(name='a', attrs={'class': 'sister'}, recursive=True, text='Lacie') #tags = soup.find(name='a', class_='sister', recursive=True, text='Lacie') print(tags) # ####### 列表类型的数据 ####### v = soup.find_all(name=['a','div']) print(v) v = soup.find_all(class_=['sister0', 'sister']) print(v) v = soup.find_all(text=['Tillie']) print(v, type(v[0])) v = soup.find_all(id=['link1','link2']) print(v) v = soup.find_all(href=['link1','link2']) print(v) # ####### 正则 ####### import re rep = re.compile('p') rep = re.compile('^p') v = soup.find_all(name=rep) print(v) rep = re.compile('sister.*') v = soup.find_all(class_=rep) print(v) rep = re.compile('http://www.oldboy.com/static/.*') v = soup.find_all(href=rep) print(v) # ####### 方法筛选 ####### def func(tag): return tag.has_attr('class') and tag.has_attr('id') v = soup.find_all(name=func) print(v) # ## get,获取标签属性 tag = soup.find('a') v = tag.get('id') print(v)
12. has_attr,检查标签是否具有该属性
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('a') v = tag.has_attr('id') print(v)
13. get_text,获取标签内部文本内容
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('a') v = tag.get_text('id') print(v)
14. index,检查标签在某标签中的索引位置
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('body') v = tag.index(tag.find('div')) print(v) tag = soup.find('body') for i,v in enumerate(tag): print(i,v)
15. is_empty_element,是否是空标签(标签内容是否为空)或者自闭合标签,判断是否是如下标签:'br' , 'hr', 'input', 'img', 'meta','spacer', 'link', 'frame', 'base'
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('br') v = tag.is_empty_element print(v)
16. 当前的关联标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
soup.next
soup.next_element
soup.next_elements
soup.next_sibling
soup.next_siblings
tag.previous
tag.previous_element
tag.previous_elements
tag.previous_sibling
tag.previous_siblings
tag.parent
tag.parents
17. 查找某标签的关联标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag.find_next(...) tag.find_all_next(...) tag.find_next_sibling(...) tag.find_next_siblings(...) tag.find_previous(...) tag.find_all_previous(...) tag.find_previous_sibling(...) tag.find_previous_siblings(...) tag.find_parent(...) tag.find_parents(...) # 参数同find_all
18. select,select_one, CSS选择器
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
soup.select("title") soup.select("p nth-of-type(3)") soup.select("body a") soup.select("html head title") tag = soup.select("span,a") soup.select("head > title") soup.select("p > a") soup.select("p > a:nth-of-type(2)") soup.select("p > #link1") soup.select("body > a") soup.select("#link1 ~ .sister") soup.select("#link1 + .sister") soup.select(".sister") soup.select("[class~=sister]") soup.select("#link1") soup.select("a#link2") soup.select('a[href]') soup.select('a[href="http://example.com/elsie"]') soup.select('a[href^="http://example.com/"]') soup.select('a[href$="tillie"]') soup.select('a[href*=".com/el"]') from bs4.element import Tag def default_candidate_generator(tag): for child in tag.descendants: if not isinstance(child, Tag): continue if not child.has_attr('href'): continue yield child tags = soup.find('body').select("a", _candidate_generator=default_candidate_generator) print(type(tags), tags) from bs4.element import Tag def default_candidate_generator(tag): for child in tag.descendants: if not isinstance(child, Tag): continue if not child.has_attr('href'): continue yield child tags = soup.find('body').select("a", _candidate_generator=default_candidate_generator, limit=1) print(type(tags), tags)
19. 标签的内容,string可以设置文本信息
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('span') print(tag.string) # 获取 tag.string = 'new content' # 设置 print(soup) tag = soup.find('body') print(tag.string) tag.string = 'xxx' print(soup) tag = soup.find('body') v = tag.stripped_strings # 递归内部获取所有标签的文本 print(v)
20.append在当前标签内部追加一个标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('body') tag.append(soup.find('a')) print(soup) from bs4.element import Tag obj = Tag(name='i',attrs={'id': 'it'}) obj.string = '我是一个新来的' tag = soup.find('body') tag.append(obj) print(soup)
21.insert在当前标签内部指定位置插入一个标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
from bs4.element import Tag obj = Tag(name='i', attrs={'id': 'it'}) obj.string = '我是一个新来的' tag = soup.find('body') tag.insert(2, obj) print(soup)
22. insert_after,insert_before 在当前标签后面或前面插入
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
from bs4.element import Tag obj = Tag(name='i', attrs={'id': 'it'}) obj.string = '我是一个新来的' tag = soup.find('body') tag.insert_before(obj) tag.insert_after(obj) print(soup)
23. replace_with 在当前标签替换为指定标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
from bs4.element import Tag obj = Tag(name='i', attrs={'id': 'it'}) obj.string = '我是一个新来的' tag = soup.find('div') tag.replace_with(obj) print(soup)
24. 创建标签之间的关系
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('div') a = soup.find('a') tag.setup(previous_sibling=a) print(tag.previous_sibling)
25. wrap,将指定标签把当前标签包裹起来
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
from bs4.element import Tag obj1 = Tag(name='div', attrs={'id': 'it'}) obj1.string = '我是一个新来的' tag = soup.find('a') v = tag.wrap(obj1) print(soup) tag = soup.find('a') v = tag.wrap(soup.find('p')) print(soup)
26. unwrap,去掉当前标签,将保留其包裹的标签
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
tag = soup.find('a') v = tag.unwrap() print(soup)
三、爬了个爬,代码示例
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
#!/usr/bin/env python # _*_ coding:utf-8 _*_ import os import requests from bs4 import BeautifulSoup response = requests.get("http://www.autohome.com.cn/news/") response.encoding = "gbk" #指定编码 soup = BeautifulSoup(response.text,"html.parser") #转换成标签对象 # 获取一个标签 # tag = soup.find(id="auto-channel-lazyload-article") #寻找id为XXX的标签 # h3 = tag.find(name="h3") #找父标签下的H3标签,注意获取的是一个对象 # print(h3,type(h3)) #获取指定信息,找到所有新闻的 标题,简介,url,图片 li_list = soup.find(id="auto-channel-lazyload-article").find_all(name="li") #找到这个父标签下所有的li标签 for li in li_list: title = li.find(name="h3")#获取标题 if not title: continue summary = li.find("p").text #获取简介 先找到p标签,由于是一个对象通过.text获取标签内的文本 url = li.find("a").get("href") #获取标题对应的超链接 img = li.find("img").get("src")#获取图片的url 现在找到img标签再获取标签的属性 #.attrs() 获取所有属性,字典类型;.get() 获取指定属性 # print(title.text,summary,url,img) #打印所有的文本信息 #将图片保存到本地 res = requests.get(img) #获取到的网络上的所有数据,.content是字节文本;.text是获取字符串文本 file_name = "imgs/%s"%img.rsplit("__",1)[-1] # print(file_name) with open(file_name,"wb") as f: f.write(res.content) #以字节的形式写入文件
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
#!/usr/bin/env python # _*_ coding:utf-8 _*_ import requests from bs4 import BeautifulSoup #第一步:GET方式,获取登录页面 及token 和 cookie r1 = requests.get("https://github.com/login") s1 = BeautifulSoup(r1.text,"html.parser") #获取csrf_token token = s1.find(name="input",attrs={"name":"authenticity_token"}).get("value") #获取网站返回的字典类型的cookies, r1_cookie_dict = r1.cookies.get_dict() #第二步:POST方式提交,将用户名,密码,token,cookies发送到服务端,注意登录认证的地址 #在浏览器,network中查找对应的状态码信息,仿照操作伪造POST请求 """ commit:Sign in utf8:✓ authenticity_token:CxKW+j7qVshR+w0fCCGl/QlYc9HbHwP/u/+oDAM/lbRyMu/mgNDRz5cIp1x/62k+sR8FmXgUUtxlLywNEa13IA== login: password: """ post_data = { "utf8":"✓", "authenticity_token":token, "login":"账户名", "password":"密码", "commit":"Sign in", #账户名密码请各位老大写上你有的测试哦!!! } r2 = requests.post("https://github.com/session",data=post_data,cookies=r1_cookie_dict) #获取登录成功之后的cookies r2_cookie_dict = r2.cookies.get_dict() #整合两次获取的cookies cookie_dict = {} cookie_dict.update(r1_cookie_dict) cookie_dict.update(r2_cookie_dict) #第三步:利用登录成功之后的cookies获取当前用户的信息 r3 = requests.get(url="https://github.com/settings/emails",cookies=cookie_dict) print(r3.text) #打印获取到的数据
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
#!/usr/bin/env python # _*_ coding:utf-8 _*_ #获取登录cookies #找到对应的标签,点赞 import requests from bs4 import BeautifulSoup #1. 获取get请求的cookies rsp1 = requests.get("http://dig.chouti.com/") r1_cookies_dict = rsp1.cookies.get_dict() print("第一次访问网页获取的cookies:",r1_cookies_dict) #打印访问请求写入的cookies #2. 发送用户名,验证码,cookies 伪造登录获取登录后的cookies url = "http://dig.chouti.com/login" login_data = { "phone":"86手机号", "password":"密码", "oneMonth":1, #注意你登录的时候,会有个86 代表中国 } rsp2 = requests.post(url=url,data=login_data,cookies=r1_cookies_dict) r2_cookies_dict = rsp2.cookies.get_dict() #获取登录之后的cookies print("登录之后的cookies:",r2_cookies_dict) #3. 找到某篇文章的id,点赞 #把两次获取的cookies合并,不知道哪个有用 cookies_dict = {} cookies_dict.update(r1_cookies_dict) cookies_dict.update(r2_cookies_dict) up_url = "http://dig.chouti.com/link/vote?linksId=13922486" #要点赞的url # 打开调试模式下的network,自己手动点赞会生成一个请求,复制响应的url即可,最后的唯一数字id代表这篇文章! rsp3 = requests.post(url=up_url,cookies=cookies_dict) #提交点赞的请求 print("推荐之后的返回结果:",rsp3.text) #打印返回的信息 # --------------------执行结果------------------------- # # 第一次访问网页获取的cookies: # {'gpsd': '4649402651b323ddaeb0cff7624d4790', 'JSESSIONID': 'aaaftsutyx3ntegAZtZ3v', 'route': '0c5178ac241ad1c9437c2aafd89a0e50'} # 登录之后的cookies: # {'puid': 'c87eb40bab0e61018bb376f845e837d6', 'gpid': '60b40a6ddba34a00b5d3b745e1c7eb4e'} # 推荐之后的返回结果: # {"result":{"code":"9999", "message":"推荐成功", "data":{"jid":"cdu_50391060132","likedTime":"1503912153915000", # "lvCount":"24","nick":"细雨蓝枫88888","uvCount":"1","voteTime":"小于1分钟前"}}}