step3: 创建jobbole爬虫

scrapy startproject Redbacktest
cd Redbacktest

创建jobbole爬虫

scrapy genspider jobbole2 blog.jobbole.com

从pycharm中导入后创建main文件

from scrapy.cmdline import execute

import sys
sys.path.append("D:\PycharmProjects\Redbacktest")
execute(['scrapy','crawl','jobbole2'])

调试前修改“君子协议”

ROBOTSTXT_OBEY = False

 

断点调试response是否获取到值