【Python之路】特别篇--服务商API认证、Restful、一致性哈希
API加密方式
1/ 加密方式:
Md5 (随机字符串 + 时间戳)
2/ 发送方式:
http://127.0.0.1:8888/index?pid= MD5加密值 | 时间戳 | 序号
服务端接收:
(1) 判断时间戳 是否在有效期内
(2) 判断url地址是否已经访问过
(3) 判断md5加密值 与 服务端生成的md5 是否匹配
import hashlib access_record = [ ] PID_LIST = [ 'abcd', 'ddd', 'ks', ] class MainHandler(tornado.web.RequestHandler): def get(self): import time # 获取url中全部数据 pid = self.get_argument('pid', None) # 获取变量 m5, client_time, i = pid.split('|') server_time = time.time() # 时间超过10s禁止 if server_time > float(client_time) + 10: self.write(' Error !! ') return # 处理10s内容重复的请求 if pid in access_record: self.write(' Error !! ') return access_record.append(pid) pid = PID_LIST[int(i)] ramdom_str = "%s|%s" %(pid, client_time) h = hashlib.md5() h.update(bytes(ramdom_str, encoding='utf-8')) server_m5 = h.hexdigest() # print(m5,server_m5) if m5 == server_m5: self.write("Hello, world") else: self.write(' Error !! ')
客户端: 生成带规则的url连接 -> SDK
import requests import time import hashlib url = 'http://127.0.0.1:8888/index?pid=%s' pid = 'aabc' client_time = time.time() h = hashlib.md5() ramdom_str = "%s|%s" %(pid, client_time) h.update(bytes(ramdom_str,encoding='utf-8')) m5 = h.hexdigest() url_payloads = '%s|%s|%d'%(m5,client_time,0) url = url%(url_payloads) print(url) ret = requests.get(url) print(ret.text)
Access_LIST 可以用Redis 设置过期时间,到期限自动清空内容。
Restful
一、名称
REST，即Representational State Transfer的缩写。我对这个词组的翻译是 "表现层状态转化" 。
如果一个架构符合REST原则，就称它为RESTful架构。
二、资源（Resources）
REST的名称"表现层状态转化"中，省略了主语。"表现层"其实指的是"资源"（Resources）的"表现层"。
所谓"资源"，就是网络上的一个实体，或者说是网络上的一个具体信息。它可以是一段文本、一张图片、一首歌曲、一种服务，总之就是一个具体的实在。你可以用一个URL（统一资源定位符）指向它，每种资源对应一个特定的URL。要获取这个资源，访问它的URL就可以，因此URL就成了每一个资源的地址或独一无二的识别符。
三、表现层（Representation）
"资源"是一种信息实体，它可以有多种外在表现形式。我们把"资源"具体呈现出来的形式，叫做它的"表现层"（Representation）。
比如，文本可以用txt格式表现，也可以用HTML格式、XML格式、JSON格式表现，甚至可以采用二进制格式；图片可以用JPG格式表现，也可以用PNG格式表现。
URL只代表资源的实体，不代表它的形式。严格地说，有些网址最后的".html"后缀名是不必要的，因为这个后缀名表示格式，属于"表现层"范畴，而URL应该只代表"资源"的位置。它的具体表现形式，应该在HTTP请求的头信息中用Accept和Content-Type字段指定，这两个字段才是对"表现层"的描述。
四、状态转化（State Transfer）
访问一个网站，就代表了客户端和服务器的一个互动过程。在这个过程中，势必涉及到数据和状态的变化。
互联网通信协议HTTP协议，是一个无状态协议。这意味着，所有的状态都保存在服务器端。因此，如果客户端想要操作服务器，必须通过某种手段，让服务器端发生"状态转化"（State Transfer）。而这种转化是建立在表现层之上的，所以就是"表现层状态转化"。
客户端用到的手段，只能是HTTP协议。具体来说，就是HTTP协议里面，四个表示操作方式的动词：GET、POST、PUT、DELETE。它们分别对应四种基本操作：GET用来获取资源，POST用来新建资源（也可以用于更新资源），PUT用来更新资源，DELETE用来删除资源。
五、综述
综合上面的解释，我们总结一下什么是RESTful架构：
（1）每一个URL代表一种资源；
（2）客户端和服务器之间，传递这种资源的某种表现层；
（3）客户端通过四个HTTP动词，对服务器端资源进行操作，实现"表现层状态转化"。
六、误区
RESTful架构有一些典型的设计误区。
最常见的一种设计错误，就是URI包含动词。因为"资源"表示一种实体，所以应该是名词，URI不应该有动词，动词应该放在HTTP协议中。
举例来说，某个URI是/posts/show/1，其中show是动词，这个URI就设计错了，正确的写法应该是/posts/1，然后用GET方法表示show。
比如网上汇款，从账户1向账户2汇款500元，错误的URI是：
POST /accounts/1/transfer/500/to/2
正确的写法是把动词transfer改成名词transaction，资源不能是动词，但是可以是一种服务：
POST /transaction HTTP/1.1 Host: 127.0.0.1 from=1&to=2&amount=500.00
另一个设计误区，就是在URI中加入版本号：
http://www.example.com/app/1.0/foo http://www.example.com/app/1.1/foo http://www.example.com/app/2.0/foo
因为不同的版本，可以理解成同一种资源的不同表现形式，所以应该采用同一个URI。版本号可以在HTTP请求头信息的Accept字段中进行区分
Accept: vnd.example-com.foo+json; version=1.0 Accept: vnd.example-com.foo+json; version=1.1 Accept: vnd.example-com.foo+json; version=2.0
一致性哈希
py2.7版本哈希
# -*- coding: utf-8 -*- """ hash_ring ~~~~~~~~~~~~~~ Implements consistent hashing that can be used when the number of server nodes can increase or decrease (like in memcached). Consistent hashing is a scheme that provides a hash table functionality in a way that the adding or removing of one slot does not significantly change the mapping of keys to slots. More information about consistent hashing can be read in these articles: "Web Caching with Consistent Hashing": http://www8.org/w8-papers/2a-webserver/caching/paper2.html "Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web (1997)": http://citeseerx.ist.psu.edu/legacymapper?did=38148 Example of usage:: memcache_servers = ['192.168.0.246:11212', '192.168.0.247:11212', '192.168.0.249:11212'] ring = HashRing(memcache_servers) server = ring.get_node('my_key') :copyright: 2008 by Amir Salihefendic. :license: BSD """ import math import sys from bisect import bisect if sys.version_info >= (2, 5): import hashlib md5_constructor = hashlib.md5 else: import md5 md5_constructor = md5.new class HashRing(object): def __init__(self, nodes=None, weights=None): """`nodes` is a list of objects that have a proper __str__ representation. `weights` is dictionary that sets weights to the nodes. The default weight is that all nodes are equal. """ self.ring = dict() self._sorted_keys = [] self.nodes = nodes if not weights: weights = {} self.weights = weights self._generate_circle() def _generate_circle(self): """Generates the circle. """ total_weight = 0 for node in self.nodes: total_weight += self.weights.get(node, 1) for node in self.nodes: weight = 1 if node in self.weights: weight = self.weights.get(node) factor = math.floor((40*len(self.nodes)*weight) / total_weight); for j in xrange(0, int(factor)): b_key = self._hash_digest( '%s-%s' % (node, j) ) for i in xrange(0, 3): key = self._hash_val(b_key, lambda x: x+i*4) self.ring[key] = node self._sorted_keys.append(key) self._sorted_keys.sort() def get_node(self, string_key): """Given a string key a corresponding node in the hash ring is returned. If the hash ring is empty, `None` is returned. """ pos = self.get_node_pos(string_key) if pos is None: return None return self.ring[ self._sorted_keys[pos] ] def get_node_pos(self, string_key): """Given a string key a corresponding node in the hash ring is returned along with it's position in the ring. If the hash ring is empty, (`None`, `None`) is returned. """ if not self.ring: return None key = self.gen_key(string_key) nodes = self._sorted_keys pos = bisect(nodes, key) if pos == len(nodes): return 0 else: return pos def iterate_nodes(self, string_key, distinct=True): """Given a string key it returns the nodes as a generator that can hold the key. The generator iterates one time through the ring starting at the correct position. if `distinct` is set, then the nodes returned will be unique, i.e. no virtual copies will be returned. """ if not self.ring: yield None, None returned_values = set() def distinct_filter(value): if str(value) not in returned_values: returned_values.add(str(value)) return value pos = self.get_node_pos(string_key) for key in self._sorted_keys[pos:]: val = distinct_filter(self.ring[key]) if val: yield val for i, key in enumerate(self._sorted_keys): if i < pos: val = distinct_filter(self.ring[key]) if val: yield val def gen_key(self, key): """Given a string key it returns a long value, this long value represents a place on the hash ring. md5 is currently used because it mixes well. """ b_key = self._hash_digest(key) return self._hash_val(b_key, lambda x: x) def _hash_val(self, b_key, entry_fn): return (( b_key[entry_fn(3)] << 24) |(b_key[entry_fn(2)] << 16) |(b_key[entry_fn(1)] << 8) | b_key[entry_fn(0)] ) def _hash_digest(self, key): m = md5_constructor() m.update(key) return map(ord, m.digest())
py3.5版本哈希
# -*- coding: utf-8 -*- """ hash_ring ~~~~~~~~~~~~~~ Implements consistent hashing that can be used when the number of server nodes can increase or decrease (like in memcached). Consistent hashing is a scheme that provides a hash table functionality in a way that the adding or removing of one slot does not significantly change the mapping of keys to slots. More information about consistent hashing can be read in these articles: "Web Caching with Consistent Hashing": http://www8.org/w8-papers/2a-webserver/caching/paper2.html "Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web (1997)": http://citeseerx.ist.psu.edu/legacymapper?did=38148 Example of usage:: memcache_servers = ['192.168.0.246:11212', '192.168.0.247:11212', '192.168.0.249:11212'] ring = HashRing(memcache_servers) server = ring.get_node('my_key') :copyright: 2008 by Amir Salihefendic. :license: BSD """ import math import sys from bisect import bisect if sys.version_info >= (2, 5): import hashlib md5_constructor = hashlib.md5 else: import md5 md5_constructor = md5.new class HashRing(object): def __init__(self, nodes=None, weights=None): """`nodes` is a list of objects that have a proper __str__ representation. `weights` is dictionary that sets weights to the nodes. The default weight is that all nodes are equal. """ self.ring = dict() self._sorted_keys = [] self.nodes = nodes if not weights: weights = {} self.weights = weights self._generate_circle() def _generate_circle(self): """Generates the circle. """ total_weight = 0 for node in self.nodes: total_weight += self.weights.get(node, 1) for node in self.nodes: weight = 1 if node in self.weights: weight = self.weights.get(node) factor = math.floor((40*len(self.nodes)*weight) / total_weight); for j in range(0, int(factor)): b_key = self._hash_digest( '%s-%s' % (node, j) ) for i in range(0, 3): key = self._hash_val(b_key, lambda x: x+i*4) self.ring[key] = node self._sorted_keys.append(key) self._sorted_keys.sort() def get_node(self, string_key): """Given a string key a corresponding node in the hash ring is returned. If the hash ring is empty, `None` is returned. """ pos = self.get_node_pos(string_key) if pos is None: return None return self.ring[ self._sorted_keys[pos] ] def get_node_pos(self, string_key): """Given a string key a corresponding node in the hash ring is returned along with it's position in the ring. If the hash ring is empty, (`None`, `None`) is returned. """ if not self.ring: return None key = self.gen_key(string_key) nodes = self._sorted_keys pos = bisect(nodes, key) if pos == len(nodes): return 0 else: return pos def iterate_nodes(self, string_key, distinct=True): """Given a string key it returns the nodes as a generator that can hold the key. The generator iterates one time through the ring starting at the correct position. if `distinct` is set, then the nodes returned will be unique, i.e. no virtual copies will be returned. """ if not self.ring: yield None, None returned_values = set() def distinct_filter(value): if str(value) not in returned_values: returned_values.add(str(value)) return value pos = self.get_node_pos(string_key) for key in self._sorted_keys[pos:]: val = distinct_filter(self.ring[key]) if val: yield val for i, key in enumerate(self._sorted_keys): if i < pos: val = distinct_filter(self.ring[key]) if val: yield val def gen_key(self, key): """Given a string key it returns a long value, this long value represents a place on the hash ring. md5 is currently used because it mixes well. """ b_key = self._hash_digest(key) return self._hash_val(b_key, lambda x: x) def _hash_val(self, b_key, entry_fn): return (( b_key[entry_fn(3)] << 24) |(b_key[entry_fn(2)] << 16) |(b_key[entry_fn(1)] << 8) | b_key[entry_fn(0)] ) def _hash_digest(self, key): m = md5_constructor() m.update(bytes(key, encoding='utf-8')) return list(m.digest())
区别在于py2.7下
m = md5_constructor() m.update(key) return map(ord, m.digest())
digest() 返回的是字符串, 最后map函数调用ord,返回数字列表
py3.5中: digest() 返回的是字节类型，for循环后就能得到数字，省去了ord这一步，即使用 list进行for循环即可生成数字列表！
def _hash_digest(self, key):
m = md5_constructor()
m.update(bytes(key, encoding='utf-8'))
return list(m.digest())
例子:
假设有5台memcache服务器 ( 利用ip进行哈希 )
memcache_servers = ['192.168.0.245:11212',
'192.168.0.246:11212',
'192.168.0.247:11212',
'192.168.0.248:11212',
'192.168.0.249:11212']
主机ip 哈希完成后形成的哈希值如下
cacheA key1
cacheB key2
cacheC key3
cacheD key4
cacheE key5
然后5台节点围绕称一个环形，如图
主机形成哈希值后，如果需要对某个值进行hash,(这里和普通hash不一样的就是这里只hash不求余)，当hash求出一个数值后，
查看这个值介于哪两个值之间。比如介于key4和key5之间，然后从key4这个落点开始顺时针查找，最先遇到的第一个cache服务器后，就把这个服务器当作此次缓存的落点，
从而把这个值放到这个落点上。如图:
在使用普通的哈希时，当其中任意一台机器down掉后，我们需要重计算落点，因为除数已经发生了改变，所以都要重新设置缓存。
而使用一致性哈希，当其中一台机器down掉后，只有它上面到它这一段的环路缓存失效了，如：当cacheC down掉后，掉落在cacheB到cacheC之间的值都将失效，那么失效的落点将继续顺时针查找cache服务器，所以新的落点就会降落到cacheD上。
memcache服务器中，为某些ip添加权重，就是对环上设置多个相同的key增加概率
memcache_servers = ['192.168.0.246:11212', '192.168.0.247:11212', '192.168.0.249:11212'] weights = { '192.168.0.246:11212': 1, '192.168.0.247:11212': 2, '192.168.0.249:11212': 10000 } ring = HashRing(memcache_servers, weights) server = ring.get_node('my_key1') print(server) # 连接mem(server) # mem.set()
