【原】k8s ingress-nginx 针对指定 User-Agent 爬虫进行限速
针对指定 User-Agent 进行限速
修改模板
将 ingress-nginx 模板nginx.tmpl提取出来，设置成 configmap 挂载到 ingress-nginx 中去使用
添加 map
添加如下内容到模板中
533 map $http_user_agent $agent {
534 default "";
535 #~curl $http_user_agent;
536 ~*apachebench $http_user_agent;
537 ~*spider $http_user_agent;
538 ~*bot $http_user_agent;
539 ~*slurp $http_user_agent;
540 ~*hello $http_user_agent;
541 ~*chrome $http_user_agent;
542 }
543
544 limit_conn_zone $agent zone=conn_lyj_com:10m;
545 limit_req_zone $agent zone=req_lyj_com:10m rate=1r/s;
546
547 limit_req zone=req_lyj_com burst=1 nodelay;
第一种：返回错误页面
添加错误页面
针对指定 UA 添加错误页面
添加 598-603 行内容到模板中
594 ## start server {{ $server.Hostname }}
595 server {
596 server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }};
597
598 recursive_error_pages on;
599 proxy_intercept_errors on;
600 if ($http_user_agent !~ "(hello|chrome)"){
601 set $err_page "https://www.qq.com/404/";
602 }
603 error_page 503 $err_page;
604
605 {{ if gt (len $cfg.BlockUserAgents) 0 }}
606 if ($block_ua) {
607 return 403;
608 }
609 {{ end }}
610 {{ if gt (len $cfg.BlockReferers) 0 }}
611 if ($block_ref) {
612 return 403;
613 }
614 {{ end }}
615
616 {{ template "SERVER" serverConfig $all $server }}
617
618 {{ if not (empty $cfg.ServerSnippet) }}
619 # Custom code snippet configured in the configuration configmap
620 {{ $cfg.ServerSnippet }}
621 {{ end }}
实际文件
容器中的文件内容如下
481 ## start server lyj.5179.top
482 server {
483 server_name lyj.5179.top ;
484
485 recursive_error_pages on;
486 proxy_intercept_errors on;
487 if ($http_user_agent !~ "(hello|chrome)"){
488 set $err_page "https://www.qq.com/404/";
489 }
490 error_page 503 $err_page;
491
492 listen 80 ;
493 listen 443 ssl http2 ;
测试
指定 UA 为 hello
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -A"hello" -o /dev/null ;echo;done
200
200
302
302
302
302
302
302
302
302
302
302
302
报错日志
10.32.0.1 - - [11/Dec/2020:09:24:15 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36" 486 0.000 [default-nginx-80] [] - - - - 621293a69b12f4b2f552ddefcfe677d5
2020/12/11 09:24:15 [error] 123#123: *1597 limiting requests, excess: 1.332 by zone "req_lyj_com", client: 10.32.0.1, server: lyj.5179.top, request: "GET / HTTP/1.1", host: "lyj.5179.top:30080"
不指定 UA
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -o /dev/null ;echo;done
200
200
200
200
200
200
200
200
200
第二种：不返回错误页面，返回 444
添加错误码
添加 598 行
594 ## start server {{ $server.Hostname }}
595 server {
596 server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }};
597
598 limit_req_status 444;
599
600 {{ if gt (len $cfg.BlockUserAgents) 0 }}
601 if ($block_ua) {
602 return 403;
603 }
604 {{ end }}
605 {{ if gt (len $cfg.BlockReferers) 0 }}
606 if ($block_ref) {
607 return 403;
608 }
测试
指定 ua
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -A"hello" -o /dev/null ;echo;done
200
200
000
000
000
000
000
000
...
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -A"hello" ;echo;done
HTTP/1.1 200 OK
Date: Fri, 11 Dec 2020 11:59:17 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
ETag: "5fbd044b-264"
Accept-Ranges: bytes
HTTP/1.1 200 OK
Date: Fri, 11 Dec 2020 11:59:17 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
ETag: "5fbd044b-264"
Accept-Ranges: bytes
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
如果指定其他的状态码就不是 000 了
报连接被重置
日志：
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - 5e431fb99cf77d32bc39d3657f6fc3ca
2020/12/11 12:01:03 [error] 38#38: *5185 limiting requests, excess: 1.896 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - f58bc7feaa516f39cc9a4103fdb87ca5
2020/12/11 12:01:03 [error] 38#38: *5186 limiting requests, excess: 1.884 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - f4d43616589ab93e4ae28c9815dd1d33
2020/12/11 12:01:03 [error] 37#37: *5187 limiting requests, excess: 1.872 by zone "req_lyj_com", client: 10.44.0.0, server: lyj.5179.top, request: "HEAD / HTTP/1.1", host: "lyj.5179.top"
10.44.0.0 - - [11/Dec/2020:12:01:03 +0000] "HEAD / HTTP/1.1" 444 0 "-" "hello" 70 0.000 [default-nginx-80] [] - - - - b68eaaa44f8ebdcaa406c651b8b0f2ad
不指定 UA
➜ liyongjiandeMBP.lan [/Users/liyongjian] for i in {1..100};do curl -H "Host:lyj.5179.top" 192.168.101.201:30080 -I -s -w '%{http_code}' -o /dev/null ;echo;done
200
200
200
200
200
200
200
200
