k8s 1.23.17 Ingress-nginx 的部署
相关链接：
- GitHub：https://github.com/kubernetes/ingress-nginx
- kubernete：https://kubernetes.io/zh-cn/docs/concepts/services-networking/ingress
1.1 部署
# 查看当前版api版本
kubectl explain Ingress
KIND: Ingress
VERSION: networking.k8s.io/v1
....注：查看ingress和自己本地的k8s版本是否对应上，在GitHub上有表格参考。
mkdir -p /root/ingress && cd /root/ingress
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.3/deploy/static/provider/baremetal/deploy.yaml
cat deploy.yaml | grep image:
image: k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
# 替换镜像
sed -i 's#k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2#registry.cn-hangzhou.aliyuncs.com/imges/controller:v1.1.3#' deploy.yaml
sed -i 's#k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660#registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1#' deploy.yaml
# 后端svc访问改成NodePort
apiVersion: v1
kind: Service
metadata:
....
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
....
type: NodePort
# 起pod
kubectl apply -f .
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o name)
kubectl exec $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
kubectl get pod -n ingress-nginx注：镜像已经从外网上传到自己在阿里云的镜像中心。
1.2 启用后端，写入配置文件执行
mkdir -p /root/ingress ; cd /root/ingress
cat > backend.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: default-http-backend
labels:
app.kubernetes.io/name: default-http-backend
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: default-http-backend
template:
metadata:
labels:
app.kubernetes.io/name: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
image: registry.cn-hangzhou.aliyuncs.com/imges/defaultbackend-amd64:1.5
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
labels:
app.kubernetes.io/name: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
app.kubernetes.io/name: default-http-backend
EOF部署：
kubectl apply -f backend.yaml
kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.98.68.205 <none> 80:31460/TCP,443:32201/TCP 72m
ingress-nginx-controller-admission ClusterIP 10.99.214.213 <none> 443/TCP 72mmkdir -p /root/ingress ; cd /root/ingress
cat > /root/ingress/deploy-demo.yaml <<EOF
#创建service为myapp
apiVersion: v1
kind: Service
metadata:
name: myapp
namespace: default
spec:
selector:
app: myapp
release: canary
ports:
- name: http
targetPort: 80
port: 80
---
#创建后端服务的pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-backend-pod
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: myapp
release: canary
template:
metadata:
labels:
app: myapp
release: canary
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v2
ports:
- name: http
containerPort: 80
EOF
kubectl apply -f deploy-demo.yaml
# 查看pod启动
kubectl get pod -l app=myapp
NAME READY STATUS RESTARTS AGE
myapp-backend-pod-9f9b5bd95-5d487 1/1 Running 0 23m
myapp-backend-pod-9f9b5bd95-k87tc 1/1 Running 0 23m
myapp-backend-pod-9f9b5bd95-vssh7 1/1 Running 0 23mcat > /root/ingress/ingress-myapp.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-myapp
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: "myapp.magedu.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: myapp
port:
number: 80
EOF
kubectl apply -f ingress-myapp.yaml
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-myapp <none> myapp.magedu.com 192.168.80.49 80 22m如果有报错：。
kubectl get validatingwebhookconfigurations
ingress-nginx-admission
kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admissionC:\Windows\System32\drivers\etc\hosts
k8s ip地址 myapp.magedu.com/etc/hosts
k8s ip地址 myapp.magedu.com
curl myapp.magedu.com:31460
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>http://myapp.magedu.com:31460
# 1.准备证书，在k8s的master节点操作
mkdir -p /root/ingress/tls ; cd /root/ingress/tls
openssl genrsa -out tls.key 2048
openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=myapp.magedu.com
# 2.生成secret，在k8s的master节点操作
kubectl create secret tls myapp-ingress-secret --cert=tls.crt --key=tls.key
# 3.查看secret
kubectl get secret | grep myapp
myapp-ingress-secret kubernetes.io/tls 2 100s
# 4.查看tomcat-ingress-secret详细信息
kubectl describe secrets myapp-ingress-secret
Name: myapp-ingress-secret
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1310 bytes
tls.key: 1704 bytes
# 5.创建ingress
cat > ingress-tomcat-tls.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-myapp-https
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- myapp.magedu.com
secretName: myapp-ingress-secret
rules:
- host: "myapp.magedu.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: myapp
port:
number: 80
EOF
# 6.创建ingress
kubectl apply -f ingress-tomcat-tls.yaml
ingress.networking.k8s.io/ingress-myapp-https created
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-myapp <none> myapp.magedu.com 192.168.80.49 80 22m
ingress-myapp-https <none> myapp.magedu.com 192.168.80.49 80, 443 9m10s
vim deploy.yaml
kind: Deployment //改为DaemonSet控制器
# replicas: 1 //删除replicas
spec:
template:
spec:
hostNetwork: true //使用HostNetwork
nodeSelector: //修改节点选择，亲和度
custom/ingress-controller-ready: true主机打上lable（master主机不部署，直接打上污点）：
kubectl get node
kubectl label nodes k8s-node01 custom/ingress-controller-ready=true
kubectl label nodes k8s-node02 custom/ingress-controller-ready=true
kuectl taint nodes k8s-master node-role.kubernetes.io/master=true:NoSchedule
kubectl apply -f deploy.yaml
# 查看ingress pod的情况
kubectl get pod -n ingress-nginx -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-9lrgn 0/1 Completed 0 65m 172.16.58.253 k8s-node02 <none> <none>
ingress-nginx-admission-patch-n75rg 0/1 Completed 0 65m 172.16.58.254 k8s-node02 <none> <none>
ingress-nginx-controller-5q227 1/1 Running 4 (63m ago) 64m 192.168.80.48 k8s-node01 <none> <none>
ingress-nginx-controller-n64bk 1/1 Running 0 61m 192.168.80.49 k8s-node02 <none> <none>注：这里80端口不能被其他服务占用，否则controller容器会启动失败。
mkdir -p /root/ingress ; cd /root/ingress
cat > /root/ingress/deploy-demo.yaml <<EOF
#创建service为myapp
apiVersion: v1
kind: Service
metadata:
name: myapp
namespace: default
spec:
selector:
app: myapp
release: canary
ports:
- name: http
targetPort: 80
port: 80
---
#创建后端服务的pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-backend-pod
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: myapp
release: canary
template:
metadata:
labels:
app: myapp
release: canary
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v2
ports:
- name: http
containerPort: 80
EOF
kubectl apply -f deploy-demo.yaml
kubectl get pod -l app=myapp
NAME READY STATUS RESTARTS AGE
myapp-backend-pod-9f9b5bd95-9zfh4 1/1 Running 0 47m
myapp-backend-pod-9f9b5bd95-cfdzs 1/1 Running 0 47m
myapp-backend-pod-9f9b5bd95-gvfx5 1/1 Running 0 47m创建ingress：
cat > /root/ingress/ingress-myapp.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-myapp
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: "myapp.magedu.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: myapp
port:
number: 80
EOF
kubectl apply -f ingress-myapp.yaml
kubectl get ingress
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-myapp <none> myapp.magedu.com 192.168.80.48,192.168.80.49 80 45m本地解析：
| 主机 | 地址 | 端口 |
|---|---|---|
| k8s-node01 | 192.168.80.48 | nginx启动端口：3080，负载均衡端口：根据ingress svc自己生成的NodePort的端口 |
| k8s-node02 | 192.168.80.49 | nginx启动端口：3080，负载均衡端口：根据ingress svc自己生成的NodePort的端口 |
| vip | 192.168.80.66 |
keepalived+nginx 实现 nginx-ingress-controller
vim deploy.yaml
kind: Deployment //改为DaemonSet控制器
# replicas: 1 //删除replicas
spec:
template:
spec:
hostNetwork: true //使用HostNetwork
nodeSelector: //修改节点选择，亲和度
custom/ingress-controller-ready: true
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
....
spec:
....
type: NodePort #后端svc访问改成NodePort
# node主机打标签
kubectl label nodes k8s-node01 custom/ingress-controller-ready=true
kubectl label nodes k8s-node02 custom/ingress-controller-ready=true
kubectl taint nodes k8s-master node-role.kubernetes.io/master=true:NoScheduleapt install nginx keepalived -y
sudo useradd nginx -G www-data80端口给nginx-ingress-controller，所以这里nginx的端口分配到3080上，如果后面不想访问pod的ingress带端口访问也可以直接将80端口给nginx，然后nginx自动负载到ingress的svc
# 修改默认端口
cd /etc/nginx/sites-enabled
cat default
listen 3080 default_server;
listen [::]:3080 default_server;
# 重启nginx
systemctl restart nginx.service
netstat -lntup | grep 3080
tcp 0 0 0.0.0.0:3080 0.0.0.0:* LISTEN 263469/nginx: maste
tcp6 0 0 :::3080 :::* LISTEN 263469/nginx: maste
# 查看ingress本地端口
kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.107.200.122 <none> 80:30152/TCP,443:31001/TCP 27m #后端访问端口
ingress-nginx-controller-admission ClusterIP 10.109.213.65 <none> 443/TCP 27m添加负载：
cd /etc/nginx ; cp nginx.conf nginx.conf_bak
cat > nginx.conf <<EOF
load_module /usr/lib/nginx/modules/ngx_stream_module.so;
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream ingress {
server 192.168.80.48:30152; # #这里配置成要访问的地址
server 192.168.80.49:30152;
}
server {
listen 80; #需要监听的端口
proxy_pass ingress;
}
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
EOF
# 检查格式
nginx -tcat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
# 检查脚本
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_NGINX {
state MASTER
interface ens33 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.80.66/24
}
track_script {
check_nginx
}
}
EOFcat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}
# 检查脚本
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_NGINX {
state BACKUP
interface ens33 # 修改为实际网卡名
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.80.66/24
}
track_script {
check_nginx
}
}
EOFkeepalived 检查脚本（注意脚本内的端口是需要监听的端口）：
cat > /etc/keepalived/check_nginx.sh <<"EOF"
#!/bin/bash
count=$(ss -antp |grep 80 |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh重启服务：
systemctl daemon-reload
systemctl start nginx keepalived
systemctl enable nginx keepalived查看效果：
本地解析：192.168.80.66 myapp.magedu.com
问题处理：
# 查看pod运行情况
kubectl describe pod -n ingress-nginx ingress-nginx-controller-74c6bcdc65-dttbv
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 115s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-74c6bcdc65-dttbv to k8s-node02
Warning FailedMount 114s (x2 over 115s) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
Normal Pulled 112s kubelet Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.1" already present on machine
Normal Created 112s kubelet Created container controller
Normal Started 112s kubelet Started container controller
Normal RELOAD 110s nginx-ingress-controller NGINX reload triggered due to a change in configuration日志报错信息：MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found。
参考文档：https://juejin.cn/post/7099413476967514125#heading-5
查看：
kubectl get secret -A|grep ingress
ingress-nginx default-token-bk2hf kubernetes.io/service-account-token 3 10m
ingress-nginx ingress-nginx-admission Opaque 3 10m
ingress-nginx ingress-nginx-admission-token-fg9jn kubernetes.io/service-account-token 3 10m
ingress-nginx ingress-nginx-token-qd9n9 kubernetes.io/service-account-token 3 10m修改如下：
vim deploy.yaml
404 terminationGracePeriodSeconds: 300
405 volumes:
406 - name: webhook-cert
407 secret:
408 secretName: ingress-nginx-admission-token-fg9jn # 修改成我们上面找到的secret重新部署：
kubectl apply -f deploy.yaml
# 查看日志看有没有之前的报错
kubectl describe pod -n ingress-nginx ingress-nginx-controller-85bf97579b-zgvdh
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-85bf97579b-zgvdh to k8s-node02
Normal Pulled 15m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.1" already present on machine
Normal Created 15m kubelet Created container controller
Normal Started 15m kubelet Started container controller
Normal RELOAD 15m nginx-ingress-controller NGINX reload triggered due to a change in configuration🙈🙉🙊🐵
