根据国家统计局的行政区划爬取阿里云地图边界
根据国家统计局的行政区划爬取阿里云地图边界
最近做一个大屏数据展示项目,需要用到全国地图及下钻功能,之前也写过一篇关于地图下钻的文章 https://www.cnblogs.com/weijiutao/p/13977011.html ,所用到的是Echarts 自带地图插件再加上很老的一份地图边界线,行政区划和边界线无法对应上,所以想到爬取一份最新的行政区划和边界线.
具体的操作是先从国家统计局 http://www.mca.gov.cn/article/sj/xzqh/2020/20201201.html 拿到一份县级以上行政区划的名称和代码,然后再去阿里云的dataV数据可视化平台 http://datav.aliyun.com/portal/school/atlas/area_selector 进行边界线数据的抓取.
先上爬取统计局县级以上行政区划的代码
1 # 统计局县级及以上行政区划 2 import requests 3 from lxml import html 4 5 etree = html.etree 6 import json 7 8 area_url = 'http://www.mca.gov.cn/article/sj/xzqh/2020/20201201.html' 9 response = requests.get(area_url) 10 result = etree.HTML(response.text) 11 areacode_list = result.xpath('//tr/td[2]/text()') 12 areaname_list = result.xpath('//tr/td[3]/text()') 13 arr = [] 14 for index, areacode in enumerate(areacode_list): 15 obj = { 16 'areacode': areacode, 17 'areaname': areaname_list[index] 18 # 'areaname': areaname_list[index+2] 19 } 20 arr.append(obj) 21 # jsonfile = open('area.json', 'w', encoding='utf-8') 22 # json.dump(arr, jsonfile, indent=2, ensure_ascii=False) 23 print(arr)
上面的代码具体解释我就不再累赘了,有兴趣的可以查看我之前的爬虫教程.
上面的代码具体落实到国家统计局的页面如下
在执行完上面的代码后我们查看打印的最后几个地区会发现问题,如下图
我们会发现820000应该是澳门特别行政区的行政区划代码,但是却成了台湾的,经过逐步排查发现海南省西沙区和南沙区没有行政区划,如下图
正好错出了两位,所以可以将上面的代码中第17行注释掉,第18行放开,这样就会向上错出两行来,然后将导出的两份json数据在三沙市那里进行数据的合并,这样就可以获取到一份完整的行政区划代码了.
接下来我们就根据拿到的行政区划区阿里云datav上来获取地图边界线.
根据上图右侧的链接我们可以获取到当前行政区划下的地图边界线,如下图
那我们就可以根据上面爬取到的县级以上行政区划再加上阿里云datav的链接地址就可以得到一份最新的地图边界线,代码如下:
1 # 根据行政区划查找阿里地图的边界 2 import requests 3 import json 4 import time 5 from lxml import html 6 7 etree = html.etree 8 9 # 打开一个json文件 10 data = open("./area.json", encoding='utf-8') 11 # 转换为python对象 12 json_result = json.load(data) 13 # print(json_result) 14 15 for area in json_result: 16 areacode = area['areacode'] 17 areaname = area['areaname'] 18 areacode_2 = areacode[: 2] 19 if int(areacode_2) > 0: 20 if areacode[-4:] == '0000': 21 print('\033[31m province---------->', areacode) 22 time.sleep(1) 23 try: 24 if areacode == '710000': 25 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '.json' 26 else: 27 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '_full.json' 28 response = requests.get(full_url) 29 result = json.loads(response.text) 30 31 jsonfile = open('data/province/' + areacode + '.json', 'w', encoding='utf-8') 32 json.dump(result, jsonfile, indent=2, ensure_ascii=False) 33 jsonfile.close() 34 35 jsonfile1 = open('data/province1/' + areaname + '.json', 'w', encoding='utf-8') 36 json.dump(result, jsonfile1, indent=2, ensure_ascii=False) 37 jsonfile1.close() 38 39 jsonfile2 = open('data/province2/' + areacode + '_' + areaname + '.json', 'w', encoding='utf-8') 40 json.dump(result, jsonfile2, indent=2, ensure_ascii=False) 41 jsonfile2.close() 42 43 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '.json' 44 response = requests.get(full_url) 45 result = json.loads(response.text) 46 47 jsonfile_json = open('data/province_json/province/' + areacode + '.json', 'w', encoding='utf-8') 48 json.dump(result, jsonfile_json, indent=2, ensure_ascii=False) 49 jsonfile_json.close() 50 51 jsonfile_json1 = open('data/province_json/province1/' + areaname + '.json', 'w', encoding='utf-8') 52 json.dump(result, jsonfile_json1, indent=2, ensure_ascii=False) 53 jsonfile_json1.close() 54 55 jsonfile_json2 = open('data/province_json/province2/' + areacode + '_' + areaname + '.json', 'w', 56 encoding='utf-8') 57 json.dump(result, jsonfile_json2, indent=2, ensure_ascii=False) 58 jsonfile_json2.close() 59 except: 60 print('\033[41m province错误') 61 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '.json' 62 jsonfile_error = open('data/province_error/' + areacode + '.json', 'w', encoding='utf-8') 63 result = {'areacode': areacode, 'url': full_url} 64 json.dump(result, jsonfile_error, indent=2, ensure_ascii=False) 65 jsonfile_error.close() 66 elif areacode[-2:] == '00': 67 print('\033[34m city---------->', areacode) 68 time.sleep(1) 69 error_list = ['441900', '442000', '460400', '620200'] 70 if areacode in error_list: 71 pass 72 else: 73 try: 74 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '_full.json' 75 response = requests.get(full_url) 76 result = json.loads(response.text) 77 78 jsonfile = open('data/city/' + areacode + '.json', 'w', encoding='utf-8') 79 json.dump(result, jsonfile, indent=2, ensure_ascii=False) 80 jsonfile.close() 81 82 jsonfile1 = open('data/city1/' + areaname + '.json', 'w', encoding='utf-8') 83 json.dump(result, jsonfile1, indent=2, ensure_ascii=False) 84 jsonfile1.close() 85 86 jsonfile2 = open('data/city2/' + areacode + '_' + areaname + '.json', 'w', encoding='utf-8') 87 json.dump(result, jsonfile2, indent=2, ensure_ascii=False) 88 jsonfile2.close() 89 90 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '.json' 91 response = requests.get(full_url) 92 result = json.loads(response.text) 93 94 jsonfile_json = open('data/city_json/city/' + areacode + '.json', 'w', encoding='utf-8') 95 json.dump(result, jsonfile_json, indent=2, ensure_ascii=False) 96 jsonfile_json.close() 97 98 jsonfile_json1 = open('data/city_json/city1/' + areaname + '.json', 'w', encoding='utf-8') 99 json.dump(result, jsonfile_json1, indent=2, ensure_ascii=False) 100 jsonfile_json1.close() 101 102 jsonfile_json2 = open('data/city_json/city2/' + areacode + '_' + areaname + '.json', 'w', encoding='utf-8') 103 json.dump(result, jsonfile_json2, indent=2, ensure_ascii=False) 104 jsonfile_json2.close() 105 except: 106 print('\033[41m city错误') 107 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '.json' 108 jsonfile_error = open('data/city_error/' + areacode + '.json', 'w', encoding='utf-8') 109 result = {'areacode': areacode, 'url': full_url} 110 json.dump(result, jsonfile_error, indent=2, ensure_ascii=False) 111 jsonfile_error.close() 112 else: 113 print('\033[35m county---------->', areacode) 114 time.sleep(1) 115 error_list = ['320611', '330103', '330104', '350402', '410322'] 116 if areacode in error_list: 117 pass 118 else: 119 try: 120 full_url = 'https://geo.datav.aliyun.com/areas_v3/bound/' + areacode + '.json' 121 response = requests.get(full_url) 122 result = json.loads(response.text) 123 124 jsonfile = open('data/county/' + areacode + '.json', 'w', encoding='utf-8') 125 json.dump(result, jsonfile, indent=2, ensure_ascii=False) 126 jsonfile.close() 127 128 jsonfile1 = open('data/county1/' + areaname + '.json', 'w', encoding='utf-8') 129 json.dump(result, jsonfile1, indent=2, ensure_ascii=False) 130 jsonfile1.close() 131 132 jsonfile2 = open('data/county2/' + areacode + '_' + areaname + '.json', 'w', encoding='utf-8') 133 json.dump(result, jsonfile2, indent=2, ensure_ascii=False) 134 jsonfile2.close() 135 except: 136 print('\033[41m county错误') 137 jsonfile_error = open('data/county_error/' + areacode + '.json', 'w', encoding='utf-8') 138 result = {'areacode': areacode, 'url': full_url} 139 json.dump(result, jsonfile_error, indent=2, ensure_ascii=False) 140 jsonfile_error.close()
最后贴上代码地址 https://gitee.com/vijtor/areacode-datav-py
