01-pycharm调试scrapy执行流程

1.pypi install -i https://pypi.douban.com/simple pypiwin32　 scrapy crawl jobbole

2. ArticleSpider/ArticleSpider/spiders/jobbole.py中,  def pass(self, response)断点,

3.根目录ArticleSpider中新建main.py文件,代码如下

import sys, os
from scrapy.cmdline import execute


sys.path.append(os.path.dirname(os.path.abspath(__file__)))
execute(['scrapy', 'crawl', 'jobbole'])

4. debug main.py文件, 查看jobbole.py 中 def pass函数返回的response

 

5.settings.py中

ROBOTSTXT_OBEY =  False