FIRST SCRAPY PRJ
zpc@Lenovo-PC:/prj/pyscrapy/a$ scrapy startproject helloword
New Scrapy project 'helloword' created in:
/cygdrive/e/01.prj/pyscrapy/a/helloword
You can start your first spider with:
cd helloword
scrapy genspider example example.com
zpc@Lenovo-PC:/prj/pyscrapy/a/helloword$ scrapy genspider baidu www.baidu.com
Created spider 'baidu' using template 'basic' in module:
helloword.spiders.baidu
问题:
zpc@Lenovo-PC:/prj/pyscrapy/a/tutorial$ scrapy crawl dmoz
/cygdrive/e/01.prj/pyscrapy/a/tutorial/tutorial/spiders/dmoz_spider.py:3: ScrapyDeprecationWarning: tutorial.spiders.dmoz_spider.DmozSpider inherits from deprecated class scrapy.spider.BaseSpider, please inherit from scrapy.spider.Spider. (warning only on first subclass, there may be others)
class DmozSpider(BaseSpider):
2014-12-17 11:32:38+0000 [scrapy] INFO: Scrapy 0.24.4 started (bot: tutorial)
2014-12-17 11:32:38+0000 [scrapy] INFO: Optional features available: ssl, http11
2014-12-17 11:32:38+0000 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial'}
2014-12-17 11:32:40+0000 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-12-17 11:32:41+0000 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-12-17 11:32:41+0000 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-12-17 11:32:41+0000 [scrapy] INFO: Enabled item pipelines:
2014-12-17 11:32:41+0000 [dmoz] INFO: Spider opened
2014-12-17 11:32:41+0000 [dmoz] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-12-17 11:32:41+0000 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2014-12-17 11:32:41+0000 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
iconv [选项...] [文件...]
有如下选项可用:
输入/输出格式规范：
-f, --from-code=名称 原始文本编码
-t, --to-code=名称 输出编码
信息：
-l, --list 列举所有已知的字符集
输出控制：
-c 从输出中忽略无效的字符
-o, --output=FILE 输出文件
-s, --silent 关闭警告
--verbose 打印进度信息
iconv -f utf-8 -t gb2312 /server_test/reports/software_.txt > /server_test/reports/software_asserts.txt
URL encode decode
http://kb.cnblogs.com/page/133765/
(1) url编码：
import urllib
url = 'http://test.com/s?wd=哈哈'
url = url.decode('gbk', 'replace')
print urllib.quote(url.encode('utf-8', 'replace'))
结果: http%3a%2f%2ftest.com%2fs%3fwd%3d%e5%93%88%e5%93%88
(2) url解码:
import urllib
encoded_url = 'http%3a%2f%2ftest.com%2fs%3fwd%3d%e5%93%88%e5%93%88'
print urllib.unquote(encoded_url).decode('utf-8', 'replace').encode('gbk', 'replace')
http://blog.csdn.net/yelbosh/article/details/7551584
很好的资料(必看(自))
http://blog.javachen.com/2014/05/24/using-scrapy-to-cralw-data/
求救: 使用git clone时, 出现以下错误如何解决?
aaa$ git clone https://github.com/python-excel/xlwt.git
Cloning into 'xlwt'...
/usr/libexec/git-core/git-remote-https.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory
zpc@Lenovo-PC:/usr/libexec/git-core$ gdb /usr/libexec/git-core/git-remote-https.exe
(gdb) r
Starting program: /usr/libexec/git-core/git-remote-https.exe
[New Thread 3192.0x19d0]
gdb: unknown target exception 0xc0000135 at 0x77898f05
During startup program exited with code 0xc0000135.
http://blog.csdn.net/hjy_six/article/details/6862648
Windows和cygwin路径的转换
cygwin下的cd命令可以直接使用Windows的路径表示。
注：不要忘了加上单引号，因为\是bash元字符，用于转义。不用上单引号cd命令收到的参数值就不是C:\Windows\System32\drivers\etc，运行报错。
$cygpath -aw '/cygdrive/c/Windows/System32/drivers/etc'
C:\Windows\System32\drivers\etc
$cygpath -au 'C:\Windows\System32\drivers\etc'
/cygdrive/c/Windows/System32/drivers/etc
有cygpath命令来完成转换，相关的选项是：
-a, --absolute output absolute path
-w, --windows print Windows form of NAMEs (C:\WINNT)
-u, --unix (default) print Unix form of NAMEs (/cygdrive/c/winnt)
查看浏览器的user-agent
javascript:alert(navigator.userAgent)
firefox
# -*- encoding: gb2312 -*-
import codecs, sys
# 用codecs提供的open方法来指定打开的文件的语言编码，它会在读取的时候自动转换为内部unicode
bfile = codecs.open( " dddd.txt " , ' r ' , " big5 " )
# bfile = open("dddd.txt", 'r')
ss = bfile.read()
bfile.close()
VIM正则表达式
http://www.cnblogs.com/RigorosLee/archive/2011/05/13/2045806.html
使用lxml前注意事项：先确保html经过了utf-8解码，即code = html.decode('utf-8', 'ignore')，否则会出现解析出错情况。因为中文被编码成utf-8之后变成 '/u2541' 之类的形式，lxml一遇到 “/”就会认为其标签结束。
scrapy命令相关
不显示无用的log
scrapy crawl --nolog test
