k8s中ingress-nginx-controller什么时候会重新加载以及如何避免频繁加载配置的策略
1、概述
我们都知道，当我们在k8s中创建了一个ingress对象之后，ingress-nginx-controller都会根据我们在ingress中配置的规则呢，重新生成一个nginx.conf的配置文件，然后通过重新加载配置的方法，来使得的nginx.conf的配置生效。
那么问题是，当我们在k8s中进行哪些操作的时候，会触发ingress-nginx-controller中的nginx.conf进行重新加载？
本文档，我们就通过一些测试，来看下，哪些操作，哪些动作，会导致nginx.conf这个配置文件被重新的加载。
2、配置重新加载的一些情况
在ingress的官方文档中，我们看到，当出现如下的情况的时候，需要重新加载配置：
- 创建新的ingress资源
- 在ingrss中增加TLS的部分
- ingress中注释的变化，并且应西乡不只一个upstream的配置
- ingress中增加或者减少一个path
- 删除ingress,service,secret
- ingress中丢失的一些引用的对象，变得可用了（或者增加上了），比如说，service或者secret
- secret的更新
看到这些，其实，不是很有直观的感受，那么我们针对上面提到的情况做一些实验，看一下，到底nginx的配置是否有变化，是否被重新的加载，然后配置有什么样的变化。
OK，我们开始··· ···
3、针对每种情况的实验
3.1、创建新的ingress
创建ingress对象之前，我们先看下ingress-nginx-controller容器中nginx.conf配置文件的时间和nginx进程的启动时间
bash-5.1$ ls -l nginx.conf ;ps -ef | grep nginx
-rw-r--r-- 1 www-data www-data 25615 Apr 18 06:45 nginx.conf
1 www-data 0:00 /usr/bin/dumb-init -- /nginx-ingress-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --configmap=ingress-nginx/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key
8 www-data 5h46 /nginx-ingress-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --configmap=ingress-nginx/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key
33 www-data 0:02 nginx: master process /usr/local/nginx/sbin/nginx -c /etc/nginx/nginx.conf
1422 www-data 0:00 vi nginx.conf
5460 www-data 0:27 nginx: worker process
5461 www-data 0:27 nginx: worker process
5462 www-data 0:27 nginx: worker process
5463 www-data 0:27 nginx: worker process
5464 www-data 0:40 nginx: worker process
5465 www-data 0:27 nginx: worker process
5466 www-data 0:26 nginx: worker process
5467 www-data 0:26 nginx: worker process
5468 www-data 0:01 nginx: cache manager process
5746 www-data 0:00 grep nginx
bash-5.1$ 
我们通过下面的命令，创建一个ingress对象
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-new-ingress
namespace: default
spec:
ingressClassName: nginx
rules:
- host: nginx-new.k8s.com
http:
paths:
- backend:
service:
name: nginx-new
port:
number: 80
path: /
pathType: Prefix
EOF查看创建好的ingress
[root@nccztsjb-node-23 ~]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx-new-ingress nginx nginx-new.k8s.com 80 5s
[root@nccztsjb-node-23 ~]#进入ingress-nginx-controller容器中查看nginx的配置文件的时间及容器重新加载的时间
bash-5.1$ ls -l nginx.conf ;ps -ef | grep nginx
-rw-r--r-- 1 www-data www-data 29592 Apr 20 03:23 nginx.conf
1 www-data 0:00 /usr/bin/dumb-init -- /nginx-ingress-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --configmap=ingress-nginx/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key
8 www-data 5h46 /nginx-ingress-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --configmap=ingress-nginx/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key
33 www-data 0:02 nginx: master process /usr/local/nginx/sbin/nginx -c /etc/nginx/nginx.conf
1422 www-data 0:00 vi nginx.conf
5795 www-data 0:00 nginx: worker process
5796 www-data 0:00 nginx: worker process
5797 www-data 0:00 nginx: worker process
5798 www-data 0:00 nginx: worker process
5799 www-data 0:00 nginx: worker process
5800 www-data 0:00 nginx: worker process
5801 www-data 0:00 nginx: worker process
5802 www-data 0:00 nginx: worker process
5803 www-data 0:00 nginx: cache manager process
6074 www-data 0:00 grep nginx
我们发现，nginx.conf配置文件的时间已经发生了变化，并且，各个worker进程的运行时间也发生了变化。说明配置进行了重新的加载。
3.2、ingress中增加或减少path
对上面的那个yaml配置文件中，增加一个path路径，项下面这样
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-new-ingress
namespace: default
spec:
ingressClassName: nginx
rules:
- host: nginx-new.k8s.com
http:
paths:
- backend:
service:
name: nginx-new
port:
number: 80
path: /
pathType: Prefix
- backend:
service:
name: nginx-new01
port:
number: 81
path: /new01
pathType: Prefix
EOF更新了之后，查看ingress-nginx-controller配置的变化
配置文件的时间发生了变化，同样的，通过ingress-nginx-controller pod的日志，也可以看到是否进行了加载
kubectl logs -f -n ingress-nginx ingress-nginx-controller-z6bhs
...省略...
W0420 06:00:27.398445 8 controller.go:988] Error obtaining Endpoints for Service "default/nginx-new01": no object matching key "default/nginx-new01" in local store
I0420 06:00:27.398665 8 controller.go:155] "Configuration changes detected, backend reload required"
I0420 06:00:27.536060 8 controller.go:172] "Backend successfully reloaded"
I0420 06:00:27.537326 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-z6bhs", UID:"402f44fb-cd6b-41e3-a233-675da8fe394c", APIVersion:"v1", ResourceVersion:"258445", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration从这里的日志我们可以看到，配置发生了变化，需要进行后端的reload.(Configuration changes detected, backend reload required)，然后是pod由于配置的变化而触发的nginx的reload的操作。。
type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
OK，通过这个实验，我们也发现了，当ingress规则中path增加的时候，会触发重新的加载。
那我们再次，变更，通过下面的减少path，查看效果（这个是在上面的基础上的配置变更）
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-new-ingress
namespace: default
spec:
ingressClassName: nginx
rules:
- host: nginx-new.k8s.com
http:
paths:
- backend:
service:
name: nginx-new
port:
number: 80
path: /
pathType: Prefix
EOFingress-nginx-controller的pod的日志
W0420 06:09:47.154332 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test": no object matching key "test01/nginx-test" in local store
W0420 06:09:47.154444 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test-02": no object matching key "test01/nginx-test-02" in local store
W0420 06:09:47.154466 8 controller.go:988] Error obtaining Endpoints for Service "default/nginx-new": no object matching key "default/nginx-new" in local store
I0420 06:09:47.264935 8 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.111s renderingIngressLength:2 renderingIngressTime:0s admissionTime:29.6kBs testedConfigurationSize:0.111}
I0420 06:09:47.265017 8 main.go:101] "successfully validated configuration, accepting" ingress="default/nginx-new-ingress"
I0420 06:09:47.270238 8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx-new-ingress", UID:"40040233-dd6e-4309-a62e-70ed34d0403e", APIVersion:"networking.k8s.io/v1", ResourceVersion:"11464712", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0420 06:09:47.270311 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test": no object matching key "test01/nginx-test" in local store
W0420 06:09:47.270362 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test-02": no object matching key "test01/nginx-test-02" in local store
W0420 06:09:47.270502 8 controller.go:988] Error obtaining Endpoints for Service "default/nginx-new": no object matching key "default/nginx-new" in local store
I0420 06:09:47.270757 8 controller.go:155] "Configuration changes detected, backend reload required"
I0420 06:09:47.416746 8 controller.go:172] "Backend successfully reloaded"
I0420 06:09:47.417550 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-z6bhs", UID:"402f44fb-cd6b-41e3-a233-675da8fe394c", APIVersion:"v1", ResourceVersion:"258445", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration从日志发现，也是触发了nginx的reload的操作。
3.3、删除ingress对象
在这个步骤里面呢，我们删除一下，刚刚实验用的ingress对象，查看下是否会发生reload的操作
kubectl delete ingress nginx-new-ingress执行完上面的命令之后，我们查看ingress-nginx-controller的pod日志
W0420 06:12:16.618607 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test": no object matching key "test01/nginx-test" in local store
W0420 06:12:16.618696 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test-02": no object matching key "test01/nginx-test-02" in local store
I0420 06:12:16.618899 8 controller.go:155] "Configuration changes detected, backend reload required"
I0420 06:12:16.820738 8 controller.go:172] "Backend successfully reloaded"
I0420 06:12:16.821728 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-z6bhs", UID:"402f44fb-cd6b-41e3-a233-675da8fe394c", APIVersion:"v1", ResourceVersion:"258445", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration同样的，监测到配置文件发生了变化，然后进行了reload的操作。
这点就和实际是相符合的。
3.4、service变得可用
刚我们上面的实验，就是创建了一个ingress的对象，但是呢，后端并没有一个对应service和pod，这次我们创建一个ingress，同时看后端的service有效了之后，会如何，在增加个有效的pod又如何
好，首先我们还是先创建个ingress对象
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-new-ingress
namespace: default
spec:
ingressClassName: nginx
rules:
- host: nginx-new.k8s.com
http:
paths:
- backend:
service:
name: nginx-new
port:
number: 80
path: /
pathType: Prefix
EOFpod的日志
W0420 06:22:48.569666 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test": no object matching key "test01/nginx-test" in local store
W0420 06:22:48.569806 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test-02": no object matching key "test01/nginx-test-02" in local store
W0420 06:22:48.569846 8 controller.go:988] Error obtaining Endpoints for Service "default/nginx-new": no object matching key "default/nginx-new" in local store
I0420 06:22:48.687580 8 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.118s renderingIngressLength:2 renderingIngressTime:0.001s admissionTime:29.6kBs testedConfigurationSize:0.119}
I0420 06:22:48.687654 8 main.go:101] "successfully validated configuration, accepting" ingress="default/nginx-new-ingress"
I0420 06:22:48.694129 8 store.go:424] "Found valid IngressClass" ingress="default/nginx-new-ingress" ingressclass="nginx"
I0420 06:22:48.694394 8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx-new-ingress", UID:"297dc73c-82d8-4085-a008-b46a1a9bc775", APIVersion:"networking.k8s.io/v1", ResourceVersion:"11465915", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0420 06:22:48.694695 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test": no object matching key "test01/nginx-test" in local store
W0420 06:22:48.694725 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test-02": no object matching key "test01/nginx-test-02" in local store
W0420 06:22:48.694741 8 controller.go:988] Error obtaining Endpoints for Service "default/nginx-new": no object matching key "default/nginx-new" in local store
I0420 06:22:48.694853 8 controller.go:155] "Configuration changes detected, backend reload required"
I0420 06:22:48.820448 8 controller.go:172] "Backend successfully reloaded"
I0420 06:22:48.820978 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-z6bhs", UID:"402f44fb-cd6b-41e3-a233-675da8fe394c", APIVersion:"v1", ResourceVersion:"258445", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration可以看到是无法获得对应的endpoints的日志
Error obtaining Endpoints for Service "default/nginx-newOK，那我们再创建一个对应的service看看
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx-new
name: nginx-new
spec:
ports:
- name: 80-80
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx-new
type: ClusterIP
EOF查看ingress-nginx-controller的pod日志
W0420 06:26:10.403081 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test": no object matching key "test01/nginx-test" in local store
W0420 06:26:10.403214 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test-02": no object matching key "test01/nginx-test-02" in local store
W0420 06:26:10.403338 8 controller.go:1083] Service "default/nginx-new" does not have any active Endpoint.
I0420 06:26:10.404248 8 controller.go:155] "Configuration changes detected, backend reload required"
I0420 06:26:10.587655 8 controller.go:172] "Backend successfully reloaded"
I0420 06:26:10.588417 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-z6bhs", UID:"402f44fb-cd6b-41e3-a233-675da8fe394c", APIVersion:"v1", ResourceVersion:"258445", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration发现，还是进行了reload的操作。也就是说，当service和ingress关联上了之后，变得有效了之后，会重新进行nginx的reload的操作。
OK，接下去呢，我们创建一个对应的pod，看看效果
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-new
spec:
selector:
matchLabels:
app: nginx-new
replicas: 2
template:
metadata:
labels:
app: nginx-new
spec:
containers:
- image: 172.20.58.152/middleware/nginx:1.21.4
imagePullPolicy: IfNotPresent
name: nginx-new
EOF[root@nccztsjb-node-23 ~]# kubectl get endpoints nginx-new
NAME ENDPOINTS AGE
nginx-new 172.39.157.203:80,172.39.21.75:80 4m2sservice是和pod进行关联，但是ingress-nginx-pod没有看到reload的操作。
3.5、删除service
我们看下，删除了service之后，是否会发生nginx的reload
kubectl delete svc nginx-newW0420 06:31:50.095619 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test": no object matching key "test01/nginx-test" in local store
W0420 06:31:50.095691 8 controller.go:988] Error obtaining Endpoints for Service "test01/nginx-test-02": no object matching key "test01/nginx-test-02" in local store
W0420 06:31:50.095736 8 controller.go:988] Error obtaining Endpoints for Service "default/nginx-new": no object matching key "default/nginx-new" in local store
I0420 06:31:50.096049 8 controller.go:155] "Configuration changes detected, backend reload required"
I0420 06:31:50.270016 8 controller.go:172] "Backend successfully reloaded"
I0420 06:31:50.270428 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-z6bhs", UID:"402f44fb-cd6b-41e3-a233-675da8fe394c", APIVersion:"v1", ResourceVersion:"258445", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration通过日志可以看到，当将ingress中使用的service对象删除了之后，会发生nginx的reload的操作。
OK，今天我们的实验，就先到这里，后面关于证书的，sercret在ingress中的使用等研究好了，再进行发布。
