通用网页抓取设计
导言
网页抓取是一个比较简单的工作,主要分为以下三步:一:分析要抓取的网页,获取待抓取的页面地址;二、用下载方式缓存网页,如果网站设置有访问时间限制,就要用这步来缓存;三、分析缓存中的网页,整理数据
在所有步骤中,对于所有的网站,只有获取换取地址、缓存网页、处理网页这几个方式不一定一样,其它的处理方式,都可以认为是一样的
总体接口设计
包括地址抓取、处理、文件缓存、消息传递委托类型,消息类型;
ICatch(抓取接口)
接口说明:
![](/Images/OutliningIndicators/ContractedBlock.gif)
1
![](/Images/OutliningIndicators/ExpandedBlockStart.gif)
2
![](/Images/OutliningIndicators/InBlock.gif)
3
![](/Images/OutliningIndicators/ExpandedBlockEnd.gif)
4
![](/Images/OutliningIndicators/None.gif)
5
![](/Images/OutliningIndicators/ExpandedBlockStart.gif)
6
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
7
![](/Images/OutliningIndicators/InBlock.gif)
8
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
9
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
10
![](/Images/OutliningIndicators/InBlock.gif)
11
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
12
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
13
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
14
![](/Images/OutliningIndicators/InBlock.gif)
15
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
16
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
17
![](/Images/OutliningIndicators/InBlock.gif)
18
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
19
![](/Images/OutliningIndicators/InBlock.gif)
20
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
21
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
22
![](/Images/OutliningIndicators/InBlock.gif)
23
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
24
![](/Images/OutliningIndicators/InBlock.gif)
25
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
26
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
27
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
28
![](/Images/OutliningIndicators/InBlock.gif)
29
![](/Images/OutliningIndicators/InBlock.gif)
30
![](/Images/OutliningIndicators/InBlock.gif)
31
![](/Images/OutliningIndicators/InBlock.gif)
32
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
33
![](/Images/OutliningIndicators/InBlock.gif)
34
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
35
![](/Images/OutliningIndicators/InBlock.gif)
36
![](/Images/OutliningIndicators/InBlock.gif)
37
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
38
![](/Images/OutliningIndicators/InBlock.gif)
39
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
40
![](/Images/OutliningIndicators/InBlock.gif)
41
![](/Images/OutliningIndicators/InBlock.gif)
42
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
43
![](/Images/OutliningIndicators/InBlock.gif)
44
![](/Images/OutliningIndicators/InBlock.gif)
45
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
46
![](/Images/OutliningIndicators/InBlock.gif)
47
![](/Images/OutliningIndicators/InBlock.gif)
48
![](/Images/OutliningIndicators/InBlock.gif)
49
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
50
![](/Images/OutliningIndicators/InBlock.gif)
51
![](/Images/OutliningIndicators/InBlock.gif)
52
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
53
![](/Images/OutliningIndicators/InBlock.gif)
54
![](/Images/OutliningIndicators/InBlock.gif)
55
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
56
![](/Images/OutliningIndicators/InBlock.gif)
57
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
58
![](/Images/OutliningIndicators/InBlock.gif)
59
![](/Images/OutliningIndicators/InBlock.gif)
60
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
61
![](/Images/OutliningIndicators/InBlock.gif)
62
![](/Images/OutliningIndicators/ExpandedBlockEnd.gif)
ICatchData(缓存网页接口)
接口:
![](/Images/OutliningIndicators/ContractedBlock.gif)
1
![](/Images/OutliningIndicators/ExpandedBlockStart.gif)
2
![](/Images/OutliningIndicators/InBlock.gif)
3
![](/Images/OutliningIndicators/ExpandedBlockEnd.gif)
4
![](/Images/OutliningIndicators/None.gif)
5
![](/Images/OutliningIndicators/ExpandedBlockStart.gif)
6
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
7
![](/Images/OutliningIndicators/InBlock.gif)
8
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
9
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
10
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
11
![](/Images/OutliningIndicators/InBlock.gif)
12
![](/Images/OutliningIndicators/InBlock.gif)
13
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
14
![](/Images/OutliningIndicators/InBlock.gif)
15
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
16
![](/Images/OutliningIndicators/InBlock.gif)
17
![](/Images/OutliningIndicators/InBlock.gif)
18
![](/Images/OutliningIndicators/InBlock.gif)
19
![](/Images/OutliningIndicators/InBlock.gif)
20
![](/Images/OutliningIndicators/InBlock.gif)
21
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
22
![](/Images/OutliningIndicators/InBlock.gif)
23
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
24
![](/Images/OutliningIndicators/InBlock.gif)
25
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
26
![](/Images/OutliningIndicators/InBlock.gif)
27
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
28
![](/Images/OutliningIndicators/InBlock.gif)
29
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
30
![](/Images/OutliningIndicators/InBlock.gif)
31
![](/Images/OutliningIndicators/ExpandedSubBlockStart.gif)
32
![](/Images/OutliningIndicators/InBlock.gif)
33
![](/Images/OutliningIndicators/ExpandedSubBlockEnd.gif)
34
![](/Images/OutliningIndicators/InBlock.gif)
35
![](/Images/OutliningIndicators/ExpandedBlockEnd.gif)
小结
只要根据不同的网站实现上面的两个接口,就可以实现网页的抓取,处理,再也不用管理浏览器的怎样加载文件