k8s 1.23.17 Ingress-nginx 的部署

1. 安装部署ingress-nginx

前言：坑比较多，pod启动running成功也并不代表ingress就能使用，需要进去pod内查看logs还有pod的描述也要看看是否有报错😒😓。

文件地址：

相关链接：

1.1 部署

# 查看当前版api版本
kubectl explain Ingress
KIND:     Ingress
VERSION:  networking.k8s.io/v1
....

注：查看ingress和自己本地的k8s版本是否对应上，在GitHub上有表格参考。

mkdir -p /root/ingress && cd /root/ingress
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.3/deploy/static/provider/baremetal/deploy.yaml

cat deploy.yaml | grep image:
image: k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660

# 替换镜像
sed  -i 's#k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2#registry.cn-hangzhou.aliyuncs.com/imges/controller:v1.1.3#' deploy.yaml
sed  -i 's#k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660#registry.cn-hangzhou.aliyuncs.com/chenby/kube-webhook-certgen:v1.1.1#' deploy.yaml

# 后端svc访问改成NodePort
apiVersion: v1
kind: Service
metadata:
  ....
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  ....
  type: NodePort

# 起pod
kubectl apply -f .

POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o name)
kubectl exec $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version

kubectl get pod -n ingress-nginx

注：镜像已经从外网上传到自己在阿里云的镜像中心。

1.2 启用后端，写入配置文件执行

mkdir -p /root/ingress ; cd /root/ingress
cat > backend.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app.kubernetes.io/name: default-http-backend
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: default-http-backend
  template:
    metadata:
      labels:
        app.kubernetes.io/name: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: registry.cn-hangzhou.aliyuncs.com/imges/defaultbackend-amd64:1.5 
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    app.kubernetes.io/name: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app.kubernetes.io/name: default-http-backend
EOF

部署：

kubectl apply -f backend.yaml

kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.98.68.205    <none>        80:31460/TCP,443:32201/TCP   72m
ingress-nginx-controller-admission   ClusterIP   10.99.214.213   <none>        443/TCP                      72m

2. 测试ingress

2.1 pod和svc创建

mkdir -p /root/ingress ; cd /root/ingress
cat > /root/ingress/deploy-demo.yaml <<EOF
#创建service为myapp
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: default
spec:
  selector:
    app: myapp
    release: canary
  ports:
  - name: http
    targetPort: 80
    port: 80
---
#创建后端服务的pod
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-backend-pod
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
EOF
kubectl apply -f deploy-demo.yaml

# 查看pod启动
kubectl get pod -l app=myapp
NAME                                READY   STATUS    RESTARTS   AGE
myapp-backend-pod-9f9b5bd95-5d487   1/1     Running   0          23m
myapp-backend-pod-9f9b5bd95-k87tc   1/1     Running   0          23m
myapp-backend-pod-9f9b5bd95-vssh7   1/1     Running   0          23m

2.2 ingress创建

cat > /root/ingress/ingress-myapp.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: "myapp.magedu.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: myapp
            port:
              number: 80
EOF
kubectl apply -f ingress-myapp.yaml
 
kubectl get ingress
NAME                  CLASS    HOSTS              ADDRESS         PORTS     AGE
ingress-myapp         <none>   myapp.magedu.com   192.168.80.49   80        22m

如果有报错：Error from server (InternalError): error when creating "ingress-tomcat-tls.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": remote error: tls: internal error root@k8s-master01:~/ingress/tls# kubectl get validatingwebhookconfigurations。

kubectl get validatingwebhookconfigurations
ingress-nginx-admission
kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

windows修改本地host文件，进行访问 C:\Windows\System32\drivers\etc\hosts，k8s地址绑定域名。

k8s ip地址 myapp.magedu.com

Linux修改本地hosts文件，/etc/hosts，k8s地址绑定域名。

k8s ip地址 myapp.magedu.com
curl myapp.magedu.com:31460
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>

http://myapp.magedu.com:31460

2.3 使用https访问

# 1.准备证书，在k8s的master节点操作
mkdir -p /root/ingress/tls ; cd /root/ingress/tls
openssl genrsa -out tls.key 2048
openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=myapp.magedu.com

# 2.生成secret，在k8s的master节点操作
kubectl create secret tls myapp-ingress-secret --cert=tls.crt --key=tls.key

# 3.查看secret
kubectl get secret | grep myapp
myapp-ingress-secret                 kubernetes.io/tls                     2      100s

# 4.查看tomcat-ingress-secret详细信息
kubectl describe secrets myapp-ingress-secret
Name:         myapp-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1310 bytes
tls.key:  1704 bytes

# 5.创建ingress
cat > ingress-tomcat-tls.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp-https
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - myapp.magedu.com
    secretName: myapp-ingress-secret
  rules:
  - host: "myapp.magedu.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: myapp
            port:
              number: 80
EOF

# 6.创建ingress
kubectl apply -f ingress-tomcat-tls.yaml
ingress.networking.k8s.io/ingress-myapp-https created

kubectl get ingress
NAME                  CLASS    HOSTS              ADDRESS         PORTS     AGE
ingress-myapp         <none>   myapp.magedu.com   192.168.80.49   80        22m
ingress-myapp-https   <none>   myapp.magedu.com   192.168.80.49   80, 443   9m10s

2.4 验证https

3. hostNetwork方式实现nodeIP:80访问

3.1 修改文件和主机打标签

修改配置文件：

vim deploy.yaml
kind: Deployment	//改为DaemonSet控制器
# replicas: 1		//删除replicas
spec:
  template:
    spec:
      hostNetwork: true	//使用HostNetwork
      nodeSelector:		//修改节点选择，亲和度
        custom/ingress-controller-ready: true

主机打上lable（master主机不部署，直接打上污点）：

kubectl get node
kubectl label nodes k8s-node01 custom/ingress-controller-ready=true
kubectl label nodes k8s-node02 custom/ingress-controller-ready=true
kuectl taint nodes k8s-master node-role.kubernetes.io/master=true:NoSchedule
kubectl apply -f deploy.yaml

# 查看ingress pod的情况
kubectl get pod -n ingress-nginx -owide
NAME                                   READY   STATUS      RESTARTS      AGE   IP              NODE         NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-9lrgn   0/1     Completed   0             65m   172.16.58.253   k8s-node02   <none>           <none>
ingress-nginx-admission-patch-n75rg    0/1     Completed   0             65m   172.16.58.254   k8s-node02   <none>           <none>
ingress-nginx-controller-5q227         1/1     Running     4 (63m ago)   64m   192.168.80.48   k8s-node01   <none>           <none>
ingress-nginx-controller-n64bk         1/1     Running     0             61m   192.168.80.49   k8s-node02   <none>           <none>

注：这里80端口不能被其他服务占用，否则controller容器会启动失败。

3.2 验证结果

mkdir -p /root/ingress ; cd /root/ingress
cat > /root/ingress/deploy-demo.yaml <<EOF
#创建service为myapp
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: default
spec:
  selector:
    app: myapp
    release: canary
  ports:
  - name: http
    targetPort: 80
    port: 80
---
#创建后端服务的pod
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-backend-pod
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
EOF
kubectl apply -f deploy-demo.yaml
kubectl get pod -l app=myapp
NAME                                READY   STATUS    RESTARTS   AGE
myapp-backend-pod-9f9b5bd95-9zfh4   1/1     Running   0          47m
myapp-backend-pod-9f9b5bd95-cfdzs   1/1     Running   0          47m
myapp-backend-pod-9f9b5bd95-gvfx5   1/1     Running   0          47m

创建ingress：

cat > /root/ingress/ingress-myapp.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: "myapp.magedu.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: myapp
            port:
              number: 80
EOF
kubectl apply -f ingress-myapp.yaml
 
kubectl get ingress
kubectl get ingress
NAME            CLASS    HOSTS              ADDRESS                       PORTS   AGE
ingress-myapp   <none>   myapp.magedu.com   192.168.80.48,192.168.80.49   80      45m

本地解析：

  • 192.168.80.48 myapp.magedu.com

  • 192.168.80.49 myapp.magedu.com

4. ingress高可用

主机 地址 端口
k8s-node01 192.168.80.48 nginx启动端口：3080，负载均衡端口：根据ingress svc自己生成的NodePort的端口
k8s-node02 192.168.80.49 nginx启动端口：3080，负载均衡端口：根据ingress svc自己生成的NodePort的端口
vip 192.168.80.66 访问端口：80

通过 keepalived+nginx 实现 nginx-ingress-controller高可用

4.1 修改文件和主机打标签

vim deploy.yaml
kind: Deployment	//改为DaemonSet控制器
# replicas: 1		//删除replicas
spec:
  template:
    spec:
      hostNetwork: true	//使用HostNetwork
      nodeSelector:		//修改节点选择，亲和度
        custom/ingress-controller-ready: true

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  ....
spec:
  ....
  type: NodePort #后端svc访问改成NodePort

# node主机打标签
kubectl label nodes k8s-node01 custom/ingress-controller-ready=true
kubectl label nodes k8s-node02 custom/ingress-controller-ready=true
kubectl taint nodes k8s-master node-role.kubernetes.io/master=true:NoSchedule

4.1 nginx配置

apt install nginx keepalived -y
sudo useradd nginx -G www-data

注：这里因为80端口给nginx-ingress-controller，所以这里nginx的端口分配到3080上，如果后面不想访问pod的ingress带端口访问也可以直接将80端口给nginx，然后nginx自动负载到ingress的svc随机分配的端口上去访问pod的svc。

修改nginx配置：

  • k8s-node01、k8s-node02

# 修改默认端口
cd /etc/nginx/sites-enabled
cat default
listen 3080 default_server;
listen [::]:3080 default_server;

# 重启nginx
systemctl restart  nginx.service
netstat -lntup  | grep 3080
tcp        0      0 0.0.0.0:3080            0.0.0.0:*               LISTEN      263469/nginx: maste
tcp6       0      0 :::3080                 :::*                    LISTEN      263469/nginx: maste

# 查看ingress本地端口
kubectl get svc  -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.107.200.122   <none>        80:30152/TCP,443:31001/TCP   27m #后端访问端口
ingress-nginx-controller-admission   ClusterIP   10.109.213.65    <none>        443/TCP                      27m

添加负载：

  • k8s-node01、k8s-node02

cd /etc/nginx ; cp nginx.conf nginx.conf_bak
cat > nginx.conf <<EOF
load_module /usr/lib/nginx/modules/ngx_stream_module.so;
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream ingress {
       server 192.168.80.48:30152;   # #这里配置成要访问的地址
       server 192.168.80.49:30152;
    }

    server {
       listen 80; #需要监听的端口
       proxy_pass ingress;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

}
EOF

# 检查格式
nginx -t

4.2 keepalived配置

  • k8s-node01

cat > /etc/keepalived/keepalived.conf << EOF
global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
} 

# 检查脚本
vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_NGINX { 
    state MASTER 
    interface ens33 # 修改为实际网卡名
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 100    # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    # 虚拟IP
    virtual_ipaddress { 
        192.168.80.66/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

  • k8s-node02

cat > /etc/keepalived/keepalived.conf << EOF
global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_BACKUP
} 

# 检查脚本
vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_NGINX { 
    state BACKUP 
    interface ens33 # 修改为实际网卡名
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 90     # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    # 虚拟IP
    virtual_ipaddress { 
        192.168.80.66/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

keepalived 检查脚本（注意脚本内的端口是需要监听的端口）：

cat > /etc/keepalived/check_nginx.sh  <<"EOF"
#!/bin/bash
count=$(ss -antp |grep 80 |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    exit 1
else
    exit 0
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh

重启服务：

systemctl daemon-reload
systemctl start nginx keepalived
systemctl enable nginx keepalived

查看效果：

本地解析：192.168.80.66 myapp.magedu.com

问题处理：

# 查看pod运行情况
kubectl describe pod -n ingress-nginx ingress-nginx-controller-74c6bcdc65-dttbv
Events:
  Type     Reason       Age                  From                      Message
  ----     ------       ----                 ----                      -------
  Normal   Scheduled    115s                 default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-74c6bcdc65-dttbv to k8s-node02
  Warning  FailedMount  114s (x2 over 115s)  kubelet                   MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulled       112s                 kubelet                   Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.1" already present on machine
  Normal   Created      112s                 kubelet                   Created container controller
  Normal   Started      112s                 kubelet                   Started container controller
  Normal   RELOAD       110s                 nginx-ingress-controller  NGINX reload triggered due to a change in configuration

日志报错信息：MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found。

参考文档：https://juejin.cn/post/7099413476967514125#heading-5

查看：

kubectl get secret -A|grep ingress
ingress-nginx     default-token-bk2hf                              kubernetes.io/service-account-token   3      10m
ingress-nginx     ingress-nginx-admission                          Opaque                                3      10m
ingress-nginx     ingress-nginx-admission-token-fg9jn              kubernetes.io/service-account-token   3      10m
ingress-nginx     ingress-nginx-token-qd9n9                        kubernetes.io/service-account-token   3      10m

修改如下：

vim deploy.yaml
404       terminationGracePeriodSeconds: 300
405       volumes:
406         - name: webhook-cert
407           secret:
408             secretName: ingress-nginx-admission-token-fg9jn # 修改成我们上面找到的secret

重新部署：

kubectl apply -f deploy.yaml
# 查看日志看有没有之前的报错
kubectl describe pod -n ingress-nginx ingress-nginx-controller-85bf97579b-zgvdh
Events:
  Type    Reason     Age   From                      Message
  ----    ------     ----  ----                      -------
  Normal  Scheduled  15m   default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-85bf97579b-zgvdh to k8s-node02
  Normal  Pulled     15m   kubelet                   Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.1" already present on machine
  Normal  Created    15m   kubelet                   Created container controller
  Normal  Started    15m   kubelet                   Started container controller
  Normal  RELOAD     15m   nginx-ingress-controller  NGINX reload triggered due to a change in configuration

🙈🙉🙊🐵

posted @   kerwin-  阅读(679)  评论(0)  编辑  收藏  举报
相关博文：
·  k8s 1.19.11 Ingress-nginx 的部署
·  k8s 1.22.10 Ingress-nginx 的部署
·  k8s1.25.3 安装ingress
·  k8s对外服务之Ingress
·  K8s Ingress-nginx 部署和实例演示
阅读排行：
· TypeScript + Deepseek 打造卜卦网站：技术与玄学的结合
· 阿里巴巴 QwQ-32B真的超越了 DeepSeek R-1吗？
· 【译】Visual Studio 中新的强大生产力特性
· 10年+ .NET Coder 心语 ── 封装的思维：从隐藏、稳定开始理解其本质意义
· 【设计模式】告别冗长if-else语句：使用策略模式优化代码结构
点击右上角即可分享
微信分享提示
SQL AI 助手