Python爬虫2----------运用代理访问
为request添加一个代理,及将浏览器头部信息加入,随机从ip列表中拿出一个ip进行访问
注意函数参数的形式,如request.proxyhandler(协议,地址)
1 import urllib.request
2 import random
3 url='http://www.baidu.com'
4 iplist=['121.226.174.246:8080','210.38.1.142:8080','210.38.1.143:8080']
5 proxyhandler=urllib.request.ProxyHandler({'http':random.choice(iplist)})
6 openner=urllib.request.build_opener(proxyhandler)
7 openner.addheaders=[('User-Agent','Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0')]
8 urllib.request.install_opener(openner)
9
10 response=urllib.request.urlopen(url)
11 html=response.read().decode('utf-8')
12
13 print(html)