黑板客爬虫闯关之关卡一

                                                              黑板客爬虫闯关之关卡一

分析：从起始界面获得下一个界面的地址信息然后开始跳转，然后又在另外界面获得下一个界面的地址信息，直到通关

闯关地址：http://www.heibanke.com/lesson/crawler_ex00/

 注意二者的区别

import re
import datetime
import requests
def Go1(url,i):
    headers = {'authorization':'Client-ID c94869b36aa272dd62dfaeefed769d4115fb3189a9d1ec88ed457207747be626'}
    html =requests.get(url=url,headers=headers)
    text = html.text
    number = re.findall(r'数字([0-9]{5})',text)#匹配
    url = url +number[0]
    print(url+'     '+str(i))
    return url

def Go2(url,i):
    headers = {'authorization':'Client-ID c94869b36aa272dd62dfaeefed769d4115fb3189a9d1ec88ed457207747be626'}
    html =requests.get(url=url,headers=headers)
    text = html.text
    number = re.findall(r'数字是([0-9]{5})',text)#注意这是调整界面跟起始界面的区别，网页源码中多了一个'是'字
    url = 'http://www.heibanke.com/lesson/crawler_ex00/' + number[0]
    print(url+'     '+str(i))
    return url

def main():
    i=1
    url = 'http://www.heibanke.com/lesson/crawler_ex00/'
    begin_time=datetime.datetime.now()
    url = Go1(url,i)
    while True:
        i=i+1
        try:
            url = Go2(url,i)
        except:
            print('最后的界面地址是：'+url)
            print('耗时为：'+str(datetime.datetime.now()-begin_time))
            break;
main()

"""
结果：
http://www.heibanke.com/lesson/crawler_ex00/65392     1
http://www.heibanke.com/lesson/crawler_ex00/36133     2
http://www.heibanke.com/lesson/crawler_ex00/72324     3
http://www.heibanke.com/lesson/crawler_ex00/57633     4
http://www.heibanke.com/lesson/crawler_ex00/91251     5
http://www.heibanke.com/lesson/crawler_ex00/87016     6
http://www.heibanke.com/lesson/crawler_ex00/77055     7
http://www.heibanke.com/lesson/crawler_ex00/30366     8
http://www.heibanke.com/lesson/crawler_ex00/83679     9
http://www.heibanke.com/lesson/crawler_ex00/31388     10
http://www.heibanke.com/lesson/crawler_ex00/99446     11
http://www.heibanke.com/lesson/crawler_ex00/69428     12
http://www.heibanke.com/lesson/crawler_ex00/34798     13
http://www.heibanke.com/lesson/crawler_ex00/16780     14
http://www.heibanke.com/lesson/crawler_ex00/36499     15
http://www.heibanke.com/lesson/crawler_ex00/21070     16
http://www.heibanke.com/lesson/crawler_ex00/96749     17
http://www.heibanke.com/lesson/crawler_ex00/71822     18
http://www.heibanke.com/lesson/crawler_ex00/48739     19
http://www.heibanke.com/lesson/crawler_ex00/62816     20
http://www.heibanke.com/lesson/crawler_ex00/80182     21
http://www.heibanke.com/lesson/crawler_ex00/68171     22
http://www.heibanke.com/lesson/crawler_ex00/45458     23
http://www.heibanke.com/lesson/crawler_ex00/56056     24
http://www.heibanke.com/lesson/crawler_ex00/87450     25
http://www.heibanke.com/lesson/crawler_ex00/52695     26
http://www.heibanke.com/lesson/crawler_ex00/36675     27
http://www.heibanke.com/lesson/crawler_ex00/25997     28
http://www.heibanke.com/lesson/crawler_ex00/73222     29
http://www.heibanke.com/lesson/crawler_ex00/93891     30
http://www.heibanke.com/lesson/crawler_ex00/29052     31
http://www.heibanke.com/lesson/crawler_ex00/72996     32
http://www.heibanke.com/lesson/crawler_ex00/73999     33
http://www.heibanke.com/lesson/crawler_ex00/23814     34
http://www.heibanke.com/lesson/crawler_ex00/98084     35
http://www.heibanke.com/lesson/crawler_ex00/51103     36
http://www.heibanke.com/lesson/crawler_ex00/39603     37
http://www.heibanke.com/lesson/crawler_ex00/34316     38
http://www.heibanke.com/lesson/crawler_ex00/55719     39
http://www.heibanke.com/lesson/crawler_ex00/53685     40
http://www.heibanke.com/lesson/crawler_ex00/77771     41
http://www.heibanke.com/lesson/crawler_ex00/69187     42
http://www.heibanke.com/lesson/crawler_ex00/89677     43
http://www.heibanke.com/lesson/crawler_ex00/71935     44
http://www.heibanke.com/lesson/crawler_ex00/98538     45
http://www.heibanke.com/lesson/crawler_ex00/79152     46
http://www.heibanke.com/lesson/crawler_ex00/70999     47
http://www.heibanke.com/lesson/crawler_ex00/35102     48
http://www.heibanke.com/lesson/crawler_ex00/75956     49
http://www.heibanke.com/lesson/crawler_ex00/19122     50
最后的界面地址是：http://www.heibanke.com/lesson/crawler_ex00/19122
耗时为：0:01:40.219459
"""