ingress 日志切割
背景
ingress本质上也是个nginx,多个项目日志全部往access.log中输入就显得杂乱无章可读性不高,不利于日志阅读关键信息获取,排错就异常痛苦。
so 闲话少叙开干!
1、部署ingress
我这里ingress是用helm部署的
[root@master ~]# helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION myingress default 1 2024-05-28 15:40:52.042004305 +0800 CST deployed nginx-ingress-controller-9.3.18 1.4.0 [root@master ~]#
还不知道咋部署的或者也想简单快速搭建起ingress测试环境,可以参考我前面写的博客,
2、部署测试应用
apache
[root@master nginx-ingress-controller]# cat apache.yaml kind: Deployment apiVersion: apps/v1 metadata: name: web01 namespace: default spec: replicas: 1 selector: matchLabels: app: httpd01 template: metadata: labels: app: httpd01 spec: containers: - name: httpd image: httpd:latest --- apiVersion: v1 kind: Service metadata: name: httpd-svc namespace: default spec: selector: app: httpd01 ports: - protocol: TCP port: 80 targetPort: 80
tomcat
[root@master nginx-ingress-controller]# cat tomcat.yaml kind: Deployment apiVersion: apps/v1 metadata: name: web02 namespace: default spec: replicas: 1 selector: matchLabels: app: tomcat01 template: metadata: labels: app: tomcat01 spec: containers: - name: tomcat image: tomcat:8.5.45 --- apiVersion: v1 kind: Service metadata: name: tomcat-svc namespace: default spec: selector: app: tomcat01 ports: - protocol: TCP port: 8080 targetPort: 8080
检查一下应用情况
[root@master nginx-ingress-controller]# kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/myingress-nginx-ingress-controller-796ddc4cc7-d7pqq 1/1 Running 0 52m pod/web01-c94b5464f-xj848 1/1 Running 0 29h pod/web02-6d96d48778-ntl5w 1/1 Running 0 29h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/httpd-svc ClusterIP 10.68.62.49 <none> 80/TCP 29h service/kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 91d service/myingress-nginx-ingress-controller NodePort 10.68.133.2 <none> 80:32080/TCP,443:32013/TCP 52m service/tomcat-svc ClusterIP 10.68.78.237 <none> 8080/TCP 29h [root@master nginx-ingress-controller]#
3、测试应用访问情况
分别创建apache和tomcat的ingress
[root@master nginx-ingress-controller]# cat ingress-httpd.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: httpd-ingress namespace: default annotations: nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.httpd.com http: paths: - backend: service: name: httpd-svc port: number: 80 path: / pathType: Prefix
[root@master nginx-ingress-controller]# cat ingress-tomcat.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tomcat-ingress namespace: default annotations: nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.tomcat.com http: paths: - backend: service: name: tomcat-svc port: number: 8080 path: /tomcat pathType: Prefix
添加本地host访问验证
[root@master nginx-ingress-controller]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.0.223 www.httpd.com 192.168.0.223 www.tomcat.com # 访问apache [root@master nginx-ingress-controller]# curl www.httpd.com:32080 <html><body><h1>It works!</h1></body></html> # 访问tomcat [root@master nginx-ingress-controller]# curl -I www.tomcat.com:32080/tomcat HTTP/1.1 200 Date: Tue, 28 May 2024 08:38:00 GMT Content-Type: text/html;charset=UTF-8 Connection: keep-alive
查看access日志
I have no name!@myingress-nginx-ingress-controller-796ddc4cc7-d7pqq:/var/log/nginx$ more access.log 172.20.196.128 - - [28/May/2024:07:42:37 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.29.0" 84 0.001 [default-httpd-svc-80] [] 172.20.196.134:80 0 0.001 200 d6c33f2841728141dab47129e3b3f0f0 172.20.196.128 - - [28/May/2024:07:42:43 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.29.0" 83 0.001 [default-httpd-svc-80] [] 172.20.196.134:80 45 0.001 200 aa51f172de9a0cdb699347d624c7608f 172.20.196.128 - - [28/May/2024:07:43:08 +0000] "GET /tomcat HTTP/1.1" 200 11211 "-" "curl/7.29.0" 90 0.004 [default-tomcat-svc-8080] [] 172.20.196.135:8080 11204 0.005 200 689d4315f3767be1ed9b2d1bae2b7361 172.20.196.128 - - [28/May/2024:08:38:00 +0000] "HEAD /tomcat HTTP/1.1" 200 0 "-" "curl/7.29.0" 91 0.003 [default-tomcat-svc-8080] [] 172.20.196.135:8080 0 0.003 200 3eb1c665887e3a2351324b1c4fcfecc7
可以看到两个应用日志都在一个文件中
4、设置日志格式
修改valus.yaml文件,配置日志格式(这一步不是必要的,但是既然日志做切割了,为什么不让格式更好看一些呢)
[root@master nginx-ingress-controller]# vim values.yaml
...
## ref: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ ## config: log-format: | log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
更新helm应用
[root@master nginx-ingress-controller]# helm upgrade myingress -f values.yaml --namespace default .
修改apache应用日志的输出文件位置
[root@master ~]# kubectl edit ingress httpd-ingress apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/server-snippet: | # 添加这两行 access_log /var/log/nginx/httpd_access.log; nginx.ingress.kubernetes.io/use-regex: "true" name: httpd-ingress namespace: default spec: ingressClassName: nginx rules: - host: www.httpd.com http: paths: - backend: service: name: httpd-svc port: number: 80 path: / pathType: Prefix
修改好之后在去看nginx controller pod 日志文件路径
I have no name!@myingress-nginx-ingress-controller-796ddc4cc7-d7pqq:/var/log/nginx$ ls access.log error.log httpd_access.log
再访问几次看看内容
I have no name!@myingress-nginx-ingress-controller-796ddc4cc7-d7pqq:/var/log/nginx$ more httpd_access.log 172.20.196.128 - - [28/May/2024:07:52:43 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.29.0" 172.20.196.128 - - [28/May/2024:07:52:45 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.29.0"
到这里日志按项目切割就完成了
5、按日期切割
其实这里我主要纠结点在于ingress是在k8s中的,而不是二进制的,是不是不太好操作呢。 其实是自己瞎自己O(∩_∩)O
步骤跟二进制的nginx日志按日期切割一样
第一步:重命名旧的access.log 第二部:执行 kill -USR1 `cat /tmp/nginx/nginx.pid` 让ingress生成新的access.log
具体日志实现
#!/bin/bash set -eu log_dir=/var/log/nginx #源日志目录 back_logs_path="$log_dir/backup/$(date -d 'yesterday' +'%F')" #备份日志目录 pod_name=$(kubectl get pod -n test| grep controller | awk '{print $1}') # 检查是否找到 pod if [ -z "$pod_name" ]; then echo "No pod found matching 'controller'" exit 1 fi # 创建备份目录 kubectl exec "$pod_name" -n test -- sh -c "mkdir -p ${back_logs_path}" # 获取ingress进程号 ng_pid=$(kubectl exec "$pod_name" -n test -- cat /tmp/nginx/nginx.pid) # 检查ng_pid是否为空 if [ -z "$ng_pid" ]; then echo "Failed to get nginx PID." exit 1 fi kubectl exec "$pod_name" -n test -- sh -c "find $log_dir -maxdepth 1 -name '*access*' | xargs -I {} mv {} {}.$(date -d 'yesterday' +'%F')" kubectl exec "$pod_name" -n test -- sh -c "find $log_dir -maxdepth 1 -name '*access*' | xargs -I {} mv {} $back_logs_path" kubectl exec "$pod_name" -n test -- sh -c "kill -USR1 $ng_pid"
辛苦你看到最后,奖励一张美女图片