python3.7 Scrapy在Windows平台的安装
1.安装python并配置好环境变量
2.更新pip ,这步操作时遇到了问题,更新超时了,所以加上--default-timeout来解决超时问题:python -m pip --default-timeout=100 install --upgrade pip
3.安装lxml :pip intsall lxml
4.安装wheel: pip install wheel
5.在python lib网站上以下安装包,然后使用pip install xxx 安装:
Twisted-18.9.0-cp37-cp37m-win_amd64.whl
pywin32-224-cp37-cp37m-win_amd64.whl
Scrapy-1.6.0-py2.py3-none-any.whl
Python Lib网站:https://www.lfd.uci.edu/~gohlke/pythonlibs/
6.Scrapy调试,验证是否安装成功:
C:\Users\admin>scrapy
Scrapy 1.6.0 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
[ more ] More commands available when run from project directory
Use "scrapy <command> -h" to see more info about a command
部分安装日志如下:
C:\Users\admin>pip install pywin32-224-cp37-cp37m-win_amd64.whl
Processing c:\users\admin\pywin32-224-cp37-cp37m-win_amd64.whl
Installing collected packages: pywin32
Successfully installed pywin32-224
C:\Users\admin>pip install Scrapy-1.6.0-py2.py3-none-any.whl
Processing c:\users\admin\scrapy-1.6.0-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
Collecting pyOpenSSL (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)
100% |████████████████████████████████| 61kB 64kB/s
Collecting cssselect>=0.9 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/7b/44/25b7283e50585f0b4156960691d951b05d061abf4a714078393e51929b30/cssselect-1.0.3-py2.py3-none-any.whl
Collecting w3lib>=1.17.0 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/81/43/9dcf92a77f5f0afe4f4df2407d7289eea01368a08b64bda00dd318ca62a6/w3lib-1.20.0-py2.py3-none-any.whl
Requirement already satisfied: Twisted>=13.1.0 in d:\python37\lib\site-packages (from Scrapy==1.6.0) (18.9.0)
Collecting service-identity (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl
Collecting queuelib (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in d:\python37\lib\site-packages (from Scrapy==1.6.0) (1.12.0)
Requirement already satisfied: lxml in d:\python37\lib\site-packages (from Scrapy==1.6.0) (4.3.1)
Collecting parsel>=1.5 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/96/69/d1d5dba5e4fecd41ffd71345863ed36a45975812c06ba77798fc15db6a64/parsel-1.5.1-py2.py3-none-any.whl
Collecting cryptography>=2.3 (from pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/cd/ab/06deda5e26c4290d9973bbf2d53d09d09898f6f972576807c8e76ff13e76/cryptography-2.5-cp37-cp37m-win_amd64.whl (1.5MB)
100% |████████████████████████████████| 1.5MB 17kB/s
Requirement already satisfied: attrs>=17.4.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (18.2.0)
Requirement already satisfied: hyperlink>=17.1.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (18.0.0)
Requirement already satisfied: zope.interface>=4.4.2 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (4.6.0)
Requirement already satisfied: constantly>=15.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (15.1.0)
Requirement already satisfied: incremental>=16.10.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (17.5.0)
Requirement already satisfied: Automat>=0.3.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (0.7.0)
Requirement already satisfied: PyHamcrest>=1.9.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (1.9.0)
Collecting pyasn1-modules (from service-identity->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl (66kB)
100% |████████████████████████████████| 71kB 18kB/s
Collecting pyasn1 (from service-identity->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
100% |████████████████████████████████| 81kB 13kB/s
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/6b/a4/4d9ba87108405ef0612907c1da4595a2cb4fba91a18105e68c1b7751bfe3/cffi-1.12.0-cp37-cp37m-win_amd64.whl (170kB)
100% |████████████████████████████████| 174kB 11kB/s
Collecting asn1crypto>=0.21.0 (from cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
100% |████████████████████████████████| 102kB 18kB/s
Requirement already satisfied: idna>=2.5 in d:\python37\lib\site-packages (from hyperlink>=17.1.1->Twisted>=13.1.0->Scrapy==1.6.0) (2.8)
Requirement already satisfied: setuptools in d:\python37\lib\site-packages (from zope.interface>=4.4.2->Twisted>=13.1.0->Scrapy==1.6.0) (39.0.1)
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
100% |████████████████████████████████| 163kB 16kB/s
Building wheels for collected packages: PyDispatcher, pycparser
Building wheel for PyDispatcher (setup.py) ... done
Stored in directory: C:\Users\admin\AppData\Local\pip\Cache\wheels\88\99\96\cfef6665f9cb1522ee6757ae5955feedf2fe25f1737f91fa7f
Building wheel for pycparser (setup.py) ... done
Stored in directory: C:\Users\admin\AppData\Local\pip\Cache\wheels\f2\9a\90\de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511
Successfully built PyDispatcher pycparser
Installing collected packages: PyDispatcher, pycparser, cffi, asn1crypto, cryptography, pyOpenSSL, cssselect, w3lib, pyasn1, pyasn1-modules, service-identity, queuelib, parsel, Scrapy
Successfully installed PyDispatcher-2.0.5 Scrapy-1.6.0 asn1crypto-0.24.0 cffi-1.12.0 cryptography-2.5 cssselect-1.0.3 parsel-1.5.1 pyOpenSSL-19.0.0 pyasn1-0.4.5 pyasn1-modules-0.2.4 pycparser-2.19 queuelib-1.5.0 service-identity-18.1.0 w3lib-1.20.0