高清图片爬取

目标网站：https://unsplash.com/

目的：爬取一些图片作为Java小程序的背景图

代码如下：

import requests,json,time,sys
from contextlib import closing
next_page =[]#存放下一页面网址
next_page_0='http://unsplash.com/napi/feeds/home'#起始爬取页面
next_page.append(next_page_0)#放入数组
headers = {'authorization':'Client-ID c94869b36aa272dd62dfaeefed769d4115fb3189a9d1ec88ed457207747be626'}
headers_1 = {'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36'}
filename=1
download_server='https://unsplash.com/photos/xxx/download?force=trues'
k=1
for i in range(40):
    photos_id = []#存放图片id
    req = requests.get(url=next_page[i],headers=headers,verify=False)
    html = json.loads(req.text)#html是经过一系列措施之后得到的网站信息，跟平常的静态网站信息类似，因为是动态态网站，所以不能直接得到
    next_page_t = html['next_page']#找到下一个页面的链接
    print(next_page_t +'找到了下一页的地址')
    next_page.append(next_page_t)
    for each in html['photos']:#进入页面的photos标签下寻找该页面下照片id
        photos_id.append(each['id'])#照片id存入数组
        print(each['id'])
        time.sleep(1)
    for j in range(len(photos_id)):#照片下载
        print('  正在爬取第%d张图片' % (k))
        k=k+1
        target = download_server.replace('xxx',photos_id[j])
        with closing(requests.get(url=target,stream=True,verify=False,headers=headers)) as r:
            with open(('E:/照片爬取/'+ '%d.jpg' % filename), 'ab+') as f:
                filename=filename+1
                for chunk in r.iter_content(chunk_size = 1024):
                    if chunk:
                       f.write(chunk)
    time.sleep(1)

爬取结果如下：

程序中预计爬取40张，但17张够用了，所以停止了爬取，但是存在一个问题，就是爬取的速度很慢，还没有想好怎么解决这个问题。。。。。