Python 爬虫常见的坑和解决方法

1.请求时出现HTTP Error 403: Forbidden

?

1 2 3 4 5

headers = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0'}    req = urllib.request.Request(url=url, headers=headers)    urllib.request.urlopen(req).read()

详细：https://www.2cto.com/kf/201309/242273.html

 

2.保存html内容时出现Python UnicodeEncodeError: 'gbk' codec can't encode character

将

?

1	f = open("out.html","w")

换成

?

1	f = open("out.html","w",encoding='utf-8')

详细：http://www.jb51.net/article/64816.htm