多线程采集
完善采集功能
post的数据获取:
如果header: content-type: application/x-www-form-urlencoded
request.body才会将数据封装给request.POST方法
requests 模块 post 数据的方式:
a.requests.post(settings.API_URL, data=json.dumps(res))
b.requests.post(settings.API_URL, json=res)
并发提高采集的效率:
python2:
进程池 : 有
线程池 : 无
python3:
进程池: 有
线程池: 有
核心代码:
hostnameList = self.get_hostnames()
from concurrent.futures import ThreadPoolExecutor
p = ThreadPoolExecutor(10)
for hostname in hostnameList:
p.submit(self.task, hostname)
唯一标识的问题
第一天:
新的数据: '{"basic": {"code": 10000, "data": {"os_platform": "linux", "os_version": "CentOS release 6.6 (Final)\\nKernel \\r on an \\\\m", "hostname": "c2.com"}}, "cpu": {"code": 10000, "data": {"cpu_count": 24, "cpu_physical_count": 2, "cpu_model": " Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz"}}, "disk": {"code": 10000, "data": {"0": {"slot": "0", "pd_type": "SAS", "capacity": "279.396", "model": "SEAGATE ST300MM0006 LS08S0K2B5NV"}, "1": {"slot": "1", "pd_type": "SAS", "capacity": "279.396", "model": "SEAGATE ST300MM0006 LS08S0K2B5AH"}, "2": {"slot": "2", "pd_type": "SATA", "capacity": "476.939", "model": "S1SZNSAFA01085L Samsung SSD 850 PRO 512GB EXM01B6Q"}, "3": {"slot": "3", "pd_type": "SATA", "capacity": "476.939", "model": "S1AXNSAF912433K Samsung SSD 840 PRO Series DXM06B0Q"}, "4": {"slot": "4", "pd_type": "SATA", "capacity": "476.939", "model": "S1AXNSAF303909M Samsung SSD 840 PRO Series DXM05B0Q"}, "5": {"slot": "5", "pd_type": "SATA", "capacity": "476.939", "model": "S1AXNSAFB00549A Samsung SSD 840 PRO Series DXM06B0Q"}}}, "memory": {"code": 10000, "data": {"DIMM #0": {"capacity": 1024, "slot": "DIMM #0", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #1": {"capacity": 0, "slot": "DIMM #1", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #2": {"capacity": 0, "slot": "DIMM #2", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #3": {"capacity": 0, "slot": "DIMM #3", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #4": {"capacity": 0, "slot": "DIMM #4", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #5": {"capacity": 0, "slot": "DIMM #5", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #6": {"capacity": 0, "slot": "DIMM #6", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #7": {"capacity": 0, "slot": "DIMM #7", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}}}, "nic": {"code": 10000, "data": {"eth0": {"up": true, "hwaddr": "00:1c:42:a5:57:7a", "ipaddrs": "10.211.55.4", "netmask": "255.255.255.0"}}}}'
数据清洗
数据入库:
server:
id hostname sn os_platform os_version
1 c1.com dsbahdsa linux 7.0
第二天:
新的数据: '{"basic": {"code": 10000, "data": {"os_platform": "linux", "os_version": "CentOS release 7.2(Final)\\nKernel \\r on an \\\\m", "hostname": "c2.com"}}, "cpu": {"code": 10000, "data": {"cpu_count": 24, "cpu_physical_count": 2, "cpu_model": " Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz"}}, "disk": {"code": 10000, "data": {"0": {"slot": "0", "pd_type": "SAS", "capacity": "279.396", "model": "SEAGATE ST300MM0006 LS08S0K2B5NV"}, "1": {"slot": "1", "pd_type": "SAS", "capacity": "279.396", "model": "SEAGATE ST300MM0006 LS08S0K2B5AH"}, "2": {"slot": "2", "pd_type": "SATA", "capacity": "476.939", "model": "S1SZNSAFA01085L Samsung SSD 850 PRO 512GB EXM01B6Q"}, "3": {"slot": "3", "pd_type": "SATA", "capacity": "476.939", "model": "S1AXNSAF912433K Samsung SSD 840 PRO Series DXM06B0Q"}, "4": {"slot": "4", "pd_type": "SATA", "capacity": "476.939", "model": "S1AXNSAF303909M Samsung SSD 840 PRO Series DXM05B0Q"}, "5": {"slot": "5", "pd_type": "SATA", "capacity": "476.939", "model": "S1AXNSAFB00549A Samsung SSD 840 PRO Series DXM06B0Q"}}}, "memory": {"code": 10000, "data": {"DIMM #0": {"capacity": 1024, "slot": "DIMM #0", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #1": {"capacity": 0, "slot": "DIMM #1", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #2": {"capacity": 0, "slot": "DIMM #2", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #3": {"capacity": 0, "slot": "DIMM #3", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #4": {"capacity": 0, "slot": "DIMM #4", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #5": {"capacity": 0, "slot": "DIMM #5", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #6": {"capacity": 0, "slot": "DIMM #6", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}, "DIMM #7": {"capacity": 0, "slot": "DIMM #7", "model": "DRAM", "speed": "667 MHz", "manufacturer": "Not Specified", "sn": "Not Specified"}}}, "nic": {"code": 10000, "data": {"eth0": {"up": true, "hwaddr": "00:1c:42:a5:57:7a", "ipaddrs": "10.211.55.4", "netmask": "255.255.255.0"}}}}'
数据入库:
老的数据:
server:
id hostname sn os_platform os_version
1 c1.com dsbahdsa linux 7.0
2
3
现在做法:
取SN作为唯一标识, 来获取老的数据
改进的做法:
1. 业务逻辑解决 : 如果公司不采集虚拟机的信息, 直接用sn
2. 用hostname作为唯一标识
agent需要遵守的步骤:
a. 装机完成之后, 需要给每一台服务器分配一个唯一的主机名
b. 将分配好的主机名录入到web管理系统中
因此, 在没有收集资产的情况下, 数据库中 已经录好了主机的主机名
c. 将采集的客户端代码放到服务器的某一个目录下面, 然后开始定时执行代码脚本
d. 开始第一次采集
将主机名(c1.com)记录到一个文件中
e. 将服务器分配给开发用
f. 12点开始收集资产信息, 11:59,班长修改了主机名, c1.com---> c2000.com,
汇报数据的时候, 永远以第一次采集的文件中的主机名为标准
g: '{"basic": {"code": 10000, "data": {"os_platform": "linux", "os_version": "CentOS release 7.2(Final)\\nKernel \\r on an \\\\m", "hostname": "c1.com}}'
API验证
非对称加密:
生成一对公钥私钥
对称加密:
第一种方式:
服务端:
server_token = "bdsjabfjsldabjfdsnbajfndjakfnjdsanfjd"
client_token = request.META.get('HTTP_TOKEN')
if server_token != client_token:
return HttpResponse('非法用户')
客户端:
import requests
client_token = "bdsjabfjsldabjfdsnbajfndjakfnjdsanfjd"
res = requests.get('http://127.0.0.1:8000/api/', headers = {"Token":client_token})
第二种方式:
加密和加盐
if server_time - client_time > 5:
return HttpResponse('[第一关] 小伙子 超时了....')
tmp = "%s|%s" % (server_token, client_time)
m = hashlib.md5()
m.update(bytes(tmp, encoding='utf-8'))
server_md5 = m.hexdigest()
if client_md5 != server_md5:
return HttpResponse('[第二关 数据被修改了]')
第三种方式：
设置过期时间（将访问过的token存入到redis中, 设置过期时间, 保证列表的长度在一定的范围之内）
if client_md5_token in key_record3
return HttpResponse('[第三关] 已经被访问过了')
else:
key_record[client_md5_token] = client_time + 10
return HttpResponse('非常重要的数据')
