《python网络数据采集》笔记2

1.网页表单与登陆窗口

Requests 库擅长处理那些复杂的 HTTP 请求、cookie、header（响应头和请求头）等内容。

1)表单提交

       import requests

       #字段

       params = {'firstname': 'Ryan', 'lastname': 'Mitchell'}

       #请求方式:post    表单提交页面

       r = requests.post("http://pythonscraping.com/pages/files/processing.php", data=params)

       print(r.text)

2)提交文件和图像

       #表单

       <form action="processing2.php" method="post" enctype="multipart/form-data">

              Submit a jpg, png, or gif: <input type="file" name="image"><br>

              <input type="submit" value="Upload File">

       </form>

       #提交文件

       import requests

       files = {'uploadFile': open('../files/Python-logo.png', 'rb')}

       r = requests.post("http://pythonscraping.com/pages/processing2.php",

       files=files)

       print(r.text)

3)处理登陆和cookie

       #从请求结果中获取 cookie，打印登录状态的验证结果,然后再通过 cookies 参数把 cookie 发送到简介页面。

       import requests

       params = {'username': 'Ryan', 'password': 'password'}

       r = requests.post("http://pythonscraping.com/pages/cookies/welcome.php", params)

       print(r.cookies.get_dict())

       r = requests.get("http://pythonscraping.com/pages/cookies/profile.php",cookies=r.cookies)

       print(r.text)

      

       #会话（session）对象（调用 requests.Session() 获取）会持续跟踪会话信息，像 cookie、header、运行 HTTP 协议的信息

       import requests

       session = requests.Session()

       params = {'username': 'username', 'password': 'password'}

       s = session.post("http://pythonscraping.com/pages/cookies/welcome.php", params)

       print(s.cookies.get_dict())

       s = session.get("http://pythonscraping.com/pages/cookies/profile.php")

       print(s.text)

      

       #Requests 库有一个 auth 模块专门用来处理 HTTP 认证：

       import requests

       from requests.auth import HTTPBasicAuth

       auth = HTTPBasicAuth('ryan', 'password')

       r = requests.post(url="http://pythonscraping.com/pages/auth/login.php", auth=auth)

       print(r.text)