windows 安装scrapy:
1 . pip3 install wheel
2. pip3 install Twisted-18.9.0-cp36-cp36m-win_amd64.whl
3.pip3 install pywin32
4.scrapy测试
使用scrapy注意事项
1.scrapy startproject firstblood
2.scrapy genspider SpiderName www.x.com
3.进入settings目录设置
ROBOTSTXT_OBEY = False 不遵从rebots协议
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36'
LOG_LEVEL = 'ERROR'
ITEM_PIPELINES 管道持久化所执行的类以及级别
4.allowed_domains注释掉
