Scrapy 设置代理IP并查看
1.设置代理可参考文章
https://blog.csdn.net/qq_42712552/article/details/88906955
2.在middlewares.py文件中设置
找到xxx_DownloaderMiddleware下载器中间件类,我创建的项目为scrapy_sample,所以名称为ScrapySampleDownloaderMiddleware,在process_request方面中设置,代码如下所示:
def process_request(self, request, spider): # Called for each request that goes through the downloader # middleware. abuyun_proxy="http://xxxxx.com:9020" proxy_user=b"Hxxxxxxxxx" proxy_pass=b"48xxxxxxxx" #设置代理认证 proxyAuth = "Basic " + base64.b64encode(proxy_user + b":" + proxy_pass).decode()
#meta是一个字典类型 request.meta['proxy']=abuyun_proxy request.headers['Proxy-Authorization']=proxyAuth request.headers["Connection"] = "close" # Must either: # - return None: continue processing this request # - or return a Response object # - or return a Request object # - or raise IgnoreRequest: process_exception() methods of # installed downloader middleware will be called return None
3.激活下载器中间件
在settings.py中找到DOWNLOADER_MIDDLEWARES,如下所示:
DOWNLOADER_MIDDLEWARES = { 'scrapy_sample.middlewares.ScrapySampleDownloaderMiddleware': 543, }
4.验证
使用shell命令来请求
F:\python_work\scrapy_sample> scrapy shell https://www.cnblogs.com/MrHSR/p/16386803.html #查看请求头信息 In [1]: request.headers Out[1]: {b'Proxy-Authorization': b'Basic xxxxx', b'Connection': b'close', b'Accept': b'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', b'Accept-Language': b'en', b'User-Agent': b'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36', b'Accept-Encoding': b'gzip, deflate'} #查看meta信息 In [2]: request.meta['proxy'] Out[2]: 'http://xxxxx.com:9020'