Django rest framework源码分析（3）----节流
自定义节流
有些时候为了对用户的访问频率进行限制和防止爬虫，需要在规定的时间中对用户访问的次数进行限制
下面自定义一个用户每分钟只能访问3次，代码如下：
from rest_framework.throttling import BaseThrottle import time VISIT_RECORD = {} #保存访问记录 class VisitThrottle(BaseThrottle): '''60s内只能访问3次''' def __init__(self): self.history = None #初始化访问记录 def allow_request(self,request,view): #获取用户ip (get_ident) remote_addr = self.get_ident(request) ctime = time.time() #如果当前IP不在访问记录里面，就添加到记录 if remote_addr not in VISIT_RECORD: VISIT_RECORD[remote_addr] = [ctime,] #键值对的形式保存 return True #True表示可以访问 #获取当前ip的历史访问记录 history = VISIT_RECORD.get(remote_addr) #初始化访问记录 self.history = history #如果有历史访问记录，并且最早一次的访问记录离当前时间超过60s，就删除最早的那个访问记录， #只要为True，就一直循环删除最早的一次访问记录 while history and history[-1] < ctime - 60: history.pop() #如果访问记录不超过三次，就把当前的访问记录插到第一个位置（pop删除最后一个） if len(history) < 3: history.insert(0,ctime) return True def wait(self): '''还需要等多久才能访问''' ctime = time.time() return 60 - (ctime - self.history[-1])
上面的大码就是当用户第一次访问的时候把它的IP地址和当前访问的时间添加到字典 VISIT_RECORD 中， 循环取出最先添加的时间判断其时间有没有过了60s，如果过了则将其删掉，然后在看列表的长度是否小于3次，如果小于3次证明，是可以访问的的。wait表示的是还需要等待多久可以继续访问
我们在使用的时候，只需在类视图中简单在类属性 throttle_classes = [ 节流控制的类] 即可
下面我们在用户登陆的时候来简单的测试下，代码代码如下：
class AuthView(APIView): """ 用于用户登录认证 """ authentication_classes = [] permission_classes = [] throttle_classes = [VisitThrottle,] # 进行访问次数的限制 def post(self,request,*args,**kwargs): ret = {'code':1000,'msg':None} try: user = request._request.POST.get('username') pwd = request._request.POST.get('password') obj = models.UserInfo.objects.filter(username=user,password=pwd).first() if not obj: ret['code'] = 1001 ret['msg'] = "用户名或密码错误" # 为登录用户创建token token = md5(user) # 存在就更新，不存在就创建 models.UserToken.objects.update_or_create(user=obj,defaults={'token':token}) ret['token'] = token except Exception as e: print(e) ret['code'] = 1002 ret['msg'] = '请求异常' return JsonResponse(ret)
我们连续访问3次登陆测试的结果如下
节流源码分析
源码入口 dispatch 代码如下
def dispatch(self, request, *args, **kwargs): """ `.dispatch()` is pretty much the same as Django's regular dispatch, but with extra hooks for startup, finalize, and exception handling. """ self.args = args self.kwargs = kwargs # 对原生的request进行封装 request = self.initialize_request(request, *args, **kwargs) self.request = request self.headers = self.default_response_headers # deprecate? try: self.initial(request, *args, **kwargs) # Get the appropriate handler method if request.method.lower() in self.http_method_names: handler = getattr(self, request.method.lower(), self.http_method_not_allowed) else: handler = self.http_method_not_allowed response = handler(request, *args, **kwargs) except Exception as exc: response = self.handle_exception(exc) self.response = self.finalize_response(request, response, *args, **kwargs) return self.response
执行认证 self.initial 代码如下
def initial(self, request, *args, **kwargs): """ Runs anything that needs to occur prior to calling the method handler. """ self.format_kwarg = self.get_format_suffix(**kwargs) # Perform content negotiation and store the accepted info on the request neg = self.perform_content_negotiation(request) request.accepted_renderer, request.accepted_media_type = neg # Determine the API version, if versioning is in use. version, scheme = self.determine_version(request, *args, **kwargs) request.version, request.versioning_scheme = version, scheme # Ensure that the incoming request is permitted # 实现认证 self.perform_authentication(request) # 权限判断 self.check_permissions(request) # 访问频率控制 self.check_throttles(request)
访问频率控制self.check_throttles(request)代码如下：
def check_throttles(self, request): """ Check if request should be throttled. Raises an appropriate exception if the request is throttled. """ for throttle in self.get_throttles(): if not throttle.allow_request(request, self): self.throttled(request, throttle.wait())
在上面的源码中我们可以知道，如果没有通过会返回 false 执行 self.throttled(request, throttle.wait()),抛出异常，代码如下
def throttled(self, request, wait): """ If request is throttled, determine what kind of exception to raise. """ raise exceptions.Throttled(wait)
如果 返回True表示可以继续访问，让我们来继续追踪 self.get_throttles() 代码如下：
def get_throttles(self): """ Instantiates and returns the list of throttles that this view uses. """ # 返回访问频率控制的列表 return [throttle() for throttle in self.throttle_classes]
通过上面的源码我们知道为什么我们在需要进行节流限制的接口中设置类属性 throttle_class = [ 节流限制的类 ] ，还有就是在我们节流限制的类中为什么要重写
allow_request(request, self) 和 wait 方法
进行全局节流的配置
上面是我们自己定义的节流控制，走我们自己定义的类属性 throttle_class = [ 节流限制的类 ] 如果我们想使用父类的从而实现全局的配置又该如何去实现呢
父类中的 thtottle_classes 设置如下
所以我们只需在配置文件中的 REST_FRAMEWORK 添加 DEFAULT_THROTTLE_CLASSES 的路径即可，配置如下
REST_FRAMEWORK = { # 全局使用的认证类 "DEFAULT_AUTHENTICATION_CLASSES":['api.utils.auth.FirstAuthtication','api.utils.auth.Authtication', ], "UNAUTHENTICATED_USER":lambda :"匿名用户", "UNAUTHENTICATED_TOKEN":None, "DEFAULT_PERMISSION_CLASSES":['api.utils.permission.SVIPPermission'],# 默认的权限认证 "DEFAULT_THROTTLE_CLASSES":["api.utils.throttle.VisitThrottle"], # 进行节流的限制 }
根据上面节流限制类路径的定义，我们在应用 api 下的utils目录下创建 throttle.py 把代码如下
from rest_framework.throttling import BaseThrottle,SimpleRateThrottle import time VISIT_RECORD = {} class VisitThrottle(BaseThrottle): def __init__(self): self.history = None def allow_request(self,request,view): # 1. 获取用户IP remote_addr = self.get_ident(request) ctime = time.time() if remote_addr not in VISIT_RECORD: VISIT_RECORD[remote_addr] = [ctime,] return True history = VISIT_RECORD.get(remote_addr) self.history = history while history and history[-1] < ctime - 60: history.pop() if len(history) < 3: history.insert(0,ctime) return True # return True # 表示可以继续访问 # return False # 表示访问频率太高，被限制 def wait(self): # 还需要等多少秒才能访问 ctime = time.time() return 60 - (ctime - self.history[-1])
我们的视图类中就不需要再添加节流的类属性的配置了，就可以实现节流的控制。简单的代码如下
class AuthView(APIView): """ 用于用户登录认证 """ authentication_classes = [] permission_classes = [] def post(self,request,*args,**kwargs): pass
内置的控制频率类
上面是写的自定义节流，drf内置了很多节流的类，用起来比较方便。
（1）BaseThrottle
- 自己要写allow_request和wait方法
- get_ident就是获取ip
class BaseThrottle(object): """ Rate throttling of requests. """ def allow_request(self, request, view): """ Return `True` if the request should be allowed, `False` otherwise. """ raise NotImplementedError('.allow_request() must be overridden') def get_ident(self, request): """ Identify the machine making the request by parsing HTTP_X_FORWARDED_FOR if present and number of proxies is > 0. If not use all of HTTP_X_FORWARDED_FOR if it is available, if not use REMOTE_ADDR. """ xff = request.META.get('HTTP_X_FORWARDED_FOR') remote_addr = request.META.get('REMOTE_ADDR') num_proxies = api_settings.NUM_PROXIES if num_proxies is not None: if num_proxies == 0 or xff is None: return remote_addr addrs = xff.split(',') client_addr = addrs[-min(num_proxies, len(addrs))] return client_addr.strip() return ''.join(xff.split()) if xff else remote_addr def wait(self): """ Optionally, return a recommended number of seconds to wait before the next request. """ return None
我们上面的例子就是通过继承 BaseThrottle 来实现的，相对较麻烦一点
（2）SimpleRateThrottle
class SimpleRateThrottle(BaseThrottle): """ A simple cache implementation, that only requires `.get_cache_key()` to be overridden. The rate (requests / seconds) is set by a `rate` attribute on the View class. The attribute is a string of the form 'number_of_requests/period'. Period should be one of: ('s', 'sec', 'm', 'min', 'h', 'hour', 'd', 'day') Previous request information used for throttling is stored in the cache. """ cache = default_cache timer = time.time cache_format = 'throttle_%(scope)s_%(ident)s' scope = None #这个值自定义，写什么都可以 THROTTLE_RATES = api_settings.DEFAULT_THROTTLE_RATES def __init__(self): if not getattr(self, 'rate', None): self.rate = self.get_rate() self.num_requests, self.duration = self.parse_rate(self.rate) def get_cache_key(self, request, view): """ Should return a unique cache-key which can be used for throttling. Must be overridden. May return `None` if the request should not be throttled. """ raise NotImplementedError('.get_cache_key() must be overridden') def get_rate(self): """ Determine the string representation of the allowed request rate. """ if not getattr(self, 'scope', None): msg = ("You must set either `.scope` or `.rate` for '%s' throttle" % self.__class__.__name__) raise ImproperlyConfigured(msg) try: return self.THROTTLE_RATES[self.scope] except KeyError: msg = "No default throttle rate set for '%s' scope" % self.scope raise ImproperlyConfigured(msg) def parse_rate(self, rate): """ Given the request rate string, return a two tuple of: <allowed number of requests>, <period of time in seconds> """ if rate is None: return (None, None) num, period = rate.split('/') num_requests = int(num) duration = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400}[period[0]] return (num_requests, duration) def allow_request(self, request, view): """ Implement the check to see if the request should be throttled. On success calls `throttle_success`. On failure calls `throttle_failure`. """ if self.rate is None: return True self.key = self.get_cache_key(request, view) if self.key is None: return True self.history = self.cache.get(self.key, []) self.now = self.timer() # Drop any requests from the history which have now passed the # throttle duration while self.history and self.history[-1] <= self.now - self.duration: self.history.pop() if len(self.history) >= self.num_requests: return self.throttle_failure() return self.throttle_success() def throttle_success(self): """ Inserts the current request's timestamp along with the key into the cache. """ self.history.insert(0, self.now) self.cache.set(self.key, self.history, self.duration) return True def throttle_failure(self): """ Called when a request to the API has failed due to throttling. """ return False def wait(self): """ Returns the recommended next request time in seconds. """ if self.history: remaining_duration = self.duration - (self.now - self.history[-1]) else: remaining_duration = self.duration available_requests = self.num_requests - len(self.history) + 1 if available_requests <= 0: return None return remaining_duration / float(available_requests)
让我们来继续追踪以下其内部的源码，我们来看一下 函数
def allow_request(self, request, view): """ Implement the check to see if the request should be throttled. On success calls `throttle_success`. On failure calls `throttle_failure`. """ if self.rate is None: return True # 获取一个用于缓存的键来记录限流的 self.key = self.get_cache_key(request, view) if self.key is None: return True self.history = self.cache.get(self.key, []) self.now = self.timer() # Drop any requests from the history which have now passed the # throttle duration while self.history and self.history[-1] <= self.now - self.duration: self.history.pop() if len(self.history) >= self.num_requests: return self.throttle_failure() return self.throttle_success()
我们可以看到 它的内部首先判断 self.rate 是否为None ，我们不知道其是什么不放追踪以下看看
if not getattr(self, 'rate', None): self.rate = self.get_rate() self.num_requests, self.duration = self.parse_rate(self.rate)
get_rate 的代码如下
def get_rate(self): """ Determine the string representation of the allowed request rate. """ if not getattr(self, 'scope', None): msg = ("You must set either `.scope` or `.rate` for '%s' throttle" % self.__class__.__name__) raise ImproperlyConfigured(msg) try: return self.THROTTLE_RATES[self.scope] except KeyError: msg = "No default throttle rate set for '%s' scope" % self.scope raise ImproperlyConfigured(msg)
我们看到 如果 没有定义self.scope这个属性的话，直接会抛出异常，所以在这里如果我们要是继承 SimpleRateThrottle 类来实现限流的话，必须定义类属性 scope ，这个是需要我们自己写的
让我们继续追踪代码下面的 self.THROTTLE_RATES[self.scope]
THROTTLE_RATES = api_settings.DEFAULT_THROTTLE_RATES
我们可以看到它是从配置文件中读取 DEFAULT_THROTTLE_CLASSES 可以看到它是从配置文件中根据键 self.scrope去进行读取的
让我们来继续追踪 allow_request 中的 self.key = self.get_cache_key(request, view) 其代码如下：
def get_cache_key(self, request, view): """ Should return a unique cache-key which can be used for throttling. Must be overridden. May return `None` if the request should not be throttled. """ raise NotImplementedError('.get_cache_key() must be overridden')
在上面的源码中，我们发现其抛出了一个异常， 这个是用来 获取一个用于缓存的键来记录限流的，所以这个方法我们要重写，重写的代码如下
from rest_framework.throttling import BaseThrottle,SimpleRateThrottle class VisitThrottle(SimpleRateThrottle): scope = "NBA" def get_cache_key(self, request, view): return self.get_ident(request) # 默认根据IP作为键 class UserThrottle(SimpleRateThrottle): scope = "NBAUser" def get_cache_key(self, request, view): return request.user.username # 登陆后用户名作为键
下缅的流程就和我们自定义的限流的逻辑一样了，早这里就不做陈述了
继承 SimpleRateThrottle的写法如下
我们可以通过继承SimpleRateThrottle类，来实现节流，会更加的简单，因为SimpleRateThrottle里面都帮我们写好了，
from rest_framework.throttling import SimpleRateThrottle class VisitThrottle(SimpleRateThrottle): '''匿名用户60s只能访问三次（根据ip）''' scope = 'NBA' #这里面的值，自己随便定义，settings里面根据这个值配置Rate def get_cache_key(self, request, view): #通过ip限制节流 return self.get_ident(request) class UserThrottle(SimpleRateThrottle): '''登录用户60s可以访问10次''' scope = 'NBAUser' #这里面的值，自己随便定义，settings里面根据这个值配置Rate def get_cache_key(self, request, view): return request.user.username
（2）settings.py
#全局 REST_FRAMEWORK = { #节流 "DEFAULT_THROTTLE_CLASSES":['API.utils.throttle.UserThrottle'], #全局配置，登录用户节流限制（10/m） "DEFAULT_THROTTLE_RATES":{ 'NBA':'3/m', #没登录用户3/m，NBA就是scope定义的值 'NBAUser':'10/m', #登录用户10/m，NBAUser就是scope定义的值 } }
（3）views.py
局部配置方法
class AuthView(APIView): . . . # 默认的节流是登录用户（10/m）,AuthView不需要登录，这里用匿名用户的节流（3/m） throttle_classes = [VisitThrottle,] .
说明：
- API.utils.throttle.UserThrottle 这个是全局配置（根据ip限制，10/m）
- DEFAULT_THROTTLE_RATES --->>>设置访问频率的
- throttle_classes = [VisitThrottle,] --->>>局部配置（不适用settings里面默认的全局配置）
总结
基本使用
- 创建类，继承BaseThrottle, 实现：allow_request ，wait
- 创建类，继承SimpleRateThrottle, 实现： get_cache_key， scope='NBA' (配置文件中的key)
全局
#节流 "DEFAULT_THROTTLE_CLASSES":['API.utils.throttle.UserThrottle'], #全局配置，登录用户节流限制（10/m） "DEFAULT_THROTTLE_RATES":{ 'NBA':'3/m', #没登录用户3/m，NBA就是scope定义的值 'NBAUser':'10/m', #登录用户10/m，NBAUser就是scope定义的值 } }
局部
throttle_classes = [VisitThrottle,]
