Scrapy爬虫-必备插件

必备插件：

lxml, an efficient XML and HTML parser

parsel, an HTML/XML data extraction library written on top of lxml

w3lib, a multi-purpose helper for dealing with URLs and web page encodings

twisted, an asynchronous networking framework https://twistedmatrix.com/Releases/Twisted/18.7/

cryptography and pyOpenSSL, to deal with various network-level security needs