第九章 Kubernetes进阶之将公司项目部署至k8s中

　　Kubernetes集群搭建完毕以后可以将公司项目部署至其中

　　1.准备工作与注意事项

　　部署项目情况

　　1，业务架构及服务（dubbo，spring cloud）

　　2，第三方服务，例如mysql,redis,zookeeper,eruka,mq

　　3，服务之间怎么通信？

　　4，资源消耗：硬件资源，带宽。

　　部署项目时用到的k8s资源

　　1，使用namespace进行不同项目隔离，或者不同环境(test,prod,dev)

　　2，无状态应用（deployment）

　　3，有状态应用（statefulset，pv，pvc）

　　4，暴露外部访问（Service，ingress）

　　5，secert，configmap

　　2.准备基础镜像并推送至镜像仓库

　　3.部署PHP/Java项目

　　1，项目构建（java） CI/CD环境这个阶段自动完成（代码拉取->代码编译构建->镜像打包->推送到镜像仓库）

　　2，编写yaml文件，使用镜像

　　kubectl->yaml->镜像仓库拉取镜像->service(集群内部访问)/Ingress暴露给外网用户

　　准备环境，在192.168.1.61上配置NFS服务器共享目录是/ifs/kubernetes

　　配置PV动态供给

　　下载nfs支持的yaml文件 下载地址https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client/deploy

　　下载以下三个yaml文件

# cat rbac.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

 

# cat class.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  #应用使用哪一个class创建PV
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

 　　deployment.yaml需要修改指定的NFS服务器地址

# cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.1.61
            - name: NFS_PATH
              value: /ifs/kubernetes
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.61
            path: /ifs/kubernetes

 　　应用

kubectl apply -f rbac.yaml 
kubectl apply -f class.yaml 
kubectl apply -f deployment.yaml 

 　　部署Harbor镜像仓库部署地址为192.168.1.61

　　克隆演示代码

git clone https://github.com/lizhenliang/tomcat-java-demo.git

 　　2023-11-06原演示代码地址已失效已经上传到以下地址

https://gitee.com/liuyueming/tomcat-java-demo.git

　　

　　准备mvn和java环境用于构建war包

yum -y install maven java-1.8.0-openjdk.x86_64

 　　修改配置文件修改数据库地址和密码以后构建修改配置文件

src/main/resources/application.yml

 　　

mvn clean install -Dmaven.test.skip=true

 　　部署tomcat是无状态，通过service即ingrss暴露

　　deployment

　　service

　　ingress

　　部署MySQL是有状态部署，需要使用PV进行数据存储以及实现自动供给

　　statefulset

　　headless service

 　　pv，pvc（storageclass）

　　构建好以后创建镜像

cd tomcat-java-demo
#构建镜像包
docker build -t 192.168.1.61/project/java-demo .
#推送至镜像仓库
docker push 192.168.1.61/project/java-demo

 　　推送镜像仓库前需要登录docker login 192.168.1.61默认用户名密码是admin Harbor12345

　　以及配置好Harbor信任

# cat /etc/docker/daemon.json 
{
          "registry-mirrors": ["https://7sl94zzz.mirror.aliyuncs.com"],
          "insecure-registries": ["192.168.1.61"]
}

 　　新建jave-demo文件夹用于本次部署

　　创建一个test命名空间

# cat namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: test

 　　

kubectl apply -f namespace.yaml

 

　　查看已经创建了test的命名空间

# kubectl get ns
NAME            STATUS   AGE
default         Active   41h
ingress-nginx   Active   16h
kube-public     Active   41h
kube-system     Active   41h
test            Active   43s

 　　创建create放置在命名空间test下 email随便指定一个即可registry-pull-secret为指定secret名称需要配置在拉取镜像的deployment中

kubectl create secret docker-registry registry-pull-secret --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@123.com -n test

 　　PS：使用以上方式创建的secret无法拉取镜像需要使用以下方式创建

　　自定义命名空间为test 需要先创建命名空间test否则无法应用会报错

# cat registry-pull-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: registry-pull-secret
  namespace: test
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuNjEiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuNiAobGludXgpIgoJfQp9
type: kubernetes.io/dockerconfigjson

# kubectl apply -f registry-pull-secret.yaml 

 　　配置文件的秘钥在已经登录Harbor的主机使用以下命令获取

cat /root/.docker/config.json | base64 -w0

 

　　查看

# kubectl get secret -n test
NAME                   TYPE                                  DATA   AGE
default-token-5mnvg    kubernetes.io/service-account-token   3      11m
registry-pull-secret   kubernetes.io/dockerconfigjson        1      50s

 　　创建deployment

# cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-java-demo
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      project: www
      run: java-demo
  template:
    metadata:
      labels:
        project: www
        run: java-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: tomcat
        image: 192.168.1.61/project/java-demo
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 0.5
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20

 　　创建

kubectl apply -f deployment.yaml

 　　查看 命名空间是test

# kubectl get pod -n test
NAME                                READY   STATUS    RESTARTS   AGE
tomcat-java-demo-66466d7bc7-27q4v   1/1     Running   0          23m
tomcat-java-demo-66466d7bc7-hjdgt   1/1     Running   0          23m
tomcat-java-demo-66466d7bc7-x8zm9   1/1     Running   0          23m

 　　使用service发布刚刚运行的应用

# cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  name: tomcat-java-demo
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    project: www
    run: java-demo
  type: NodePort
status:
  loadBalancer: {}

 　　PS:注意service的命名空间也需要设置为test否则创建的service无法访问后端的tomcat

 　　应用

kubectl apply -f service.yaml 

 　　查看

# kubectl get svc -n test
NAME               TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
tomcat-java-demo   NodePort   10.0.0.44    <none>        80:40995/TCP   4m52s

 　　没有部署ingress之前可以使用node ip加随机的nodeport端口进行访问

　　部署ingress 部署ingress之前需要确保已经部署好ingress控制器 查看是否已经部署控制器使用命令

kubectl get pod -n ingress-nginx

 

# cat ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: simple-fanout-example
  namespace: test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  #定义访问域名
  - host: java.ctnrs.com
    http:
      paths:
      - path: /
        backend:
          #转发到哪个server下
          serviceName: tomcat-java-demo
          servicePort: 80

 　　应用

kubectl apply -f ingress.yaml 

 　　查看

# kubectl get ingress -n test
NAME                    HOSTS            ADDRESS   PORTS   AGE
simple-fanout-example   java.ctnrs.com             80      7s

 　　设置hosts可以通过域名访问http://java.ctnrs.com/

　　但是没有配置MySQL只能看见页面无法实现功能

　　部署mysql

# cat mysql.yaml 
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    project: java-demo
    run: mysql
spec:
  ports:
  - port: 3306
    name: mysql
  #不需要clusterIP
  clusterIP: None
  selector:
    project: java-demo
    run: mysql

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  selector:
    matchLabels:
      project: java-demo
      run: mysql
  serviceName: "mysql"
  template:
    metadata:
      labels:
        project: java-demo
        run: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5.7
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 2Gi

 　　应用

kubectl apply -f mysql.yaml

 　　查看

 

 　　在对应的nfs服务器会创建一个文件夹用于持久保存MySQL数据

 

 

　　查看pv和pvc

kubectl get pv,pvc

 　　　MySQL的连接地址是db-0.mysql.default 可以创建一个busybox测试

 kubectl run -it --rm --image=busybox bash

 

# ping db-0.mysql.default
PING db-0.mysql.default (172.17.9.3): 56 data bytes
64 bytes from 172.17.9.3: seq=0 ttl=64 time=0.084 ms
64 bytes from 172.17.9.3: seq=1 ttl=64 time=0.076 ms

 　　修改项目的数据库配置

cd tomcat-java-demo
vim src/main/resources/application.yml

 

 

 　　重新打包

mvn clean install -Dmaven.test.skip=true

 　　重新打包docker镜像

docker build -t 192.168.1.61/project/java-demo:1.0 .

 　　推送至Harbor镜像仓库

docker push 192.168.1.61/project/java-demo:1.0

 　　登录私有镜像仓库查看已经有了1.0版本

 

 　　修改deployment.yaml修改一下版本号

 

 　　滚动更新

kubectl apply -f deployment.yaml

 

 

 　　滚动更新是先现在新的镜像包启动以后再删除原有包

　　拷贝数据库脚本至容器

kubectl cp tables_ly_tomcat.sql db-0:/

 　　登录容器导入数据库表

#登录容器
kubectl exec -it db-0 bash
#登录数据库
mysql -uroot -p123456
#导入表
mysql> source tables_ly_tomcat.sql;

 　　因为连接了数据库所以登录web页面java.ctnrs.com即可实现该程序的简单功能添加信息

 　　

　　部署PHP项目　　

　　本次部署WordPress　　

　　下载示例代码

git clone https://github.com/lizhenliang/php-demo

 　　修改数据库配置

#cd php-demo
vim wp-config.php

 

 

 　　查看Dockerfile文件

# cat Dockerfile 
FROM lizhenliang/nginx-php:latest
MAINTAINER www.ctnrs.com
ADD . /usr/local/nginx/html 

 　　构建提交到私有仓库

docker build -t 192.168.1.61/project/php-demo:1.0 .
docker push 192.168.1.61/project/php-demo:1.0

 　　删除刚刚部署java的namespace为test的命名空间

kubectl delete -f namespace.yaml

 　　删除刚刚部署的数据库MySQL

kubectl delete -f mysql.yaml

 　　查看是否已经删除干净

#命名空间test的pod svc ingress都没有了
# kubectl get pod,svc,ingress -n test
No resources found.
#数据库db-0的坡道也没有了，保留了nfs
# kubectl get pod
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-7665588bd7-66q7w   1/1     Running   0          69m

 　　创建namespace名字为test

# cat namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: test

#kubectl apply -f namespace.yaml

 　　查看

# kubectl get ns
NAME            STATUS   AGE
default         Active   2d17h
ingress-nginx   Active   41h
kube-public     Active   2d17h
kube-system     Active   2d17h
test            Active   17s

 　　创建Harbor加密认证

# cat registry-pull-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: registry-pull-secret
  namespace: test
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuNjEiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuNiAobGludXgpIgoJfQp9
type: kubernetes.io/dockerconfigjson
 
# kubectl apply -f registry-pull-secret.yaml 

 　　查看

# kubectl get secret -n test
NAME                   TYPE                                  DATA   AGE
default-token-kklxx    kubernetes.io/service-account-token   3      86s
registry-pull-secret   kubernetes.io/dockerconfigjson        1      9s

 　　创建php deployment

# cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-demo
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      project: www
      run: php-demo
  template:
    metadata:
      labels:
        project: www
        run: php-demo
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: php
        image: 192.168.1.61/project/php-demo:1.0
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 0.5
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 60
          timeoutSeconds: 20
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 60
          timeoutSeconds: 20

# kubectl apply -f deployment.yaml

 　　查看

# kubectl get pod -n test
NAME                        READY   STATUS    RESTARTS   AGE
php-demo-5c4b869ffd-2cwzr   0/1     Running   0          15s
php-demo-5c4b869ffd-7wt48   0/1     Running   0          15s
php-demo-5c4b869ffd-gsm4t   0/1     Running   0          15s

 　　创建service对应刚刚创建的deployment

# cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  name: php-demo
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    project: www
    run: php-demo
  type: NodePort
status:
  loadBalancer: {}

# kubectl apply -f service.yaml 
service/php-demo created

 　　查看

# kubectl get svc -n test
NAME       TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
php-demo   NodePort   10.0.0.14    <none>        80:49142/TCP   14s

 　　创建ingress 创建ingress之前确保已经部署了ingress-nginx

# kubectl get pod -n ingress-nginx
NAME                             READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-42gsg   1/1     Running   2          41h
nginx-ingress-controller-5r29z   1/1     Running   2          41h

 

# cat ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: simple-fanout-example
  namespace: test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  #定义访问域名
  - host: php.ctnrs.com
    http:
      paths:
      - path: /
        backend:
          #转发到哪个server下
          serviceName: php-demo
          servicePort: 80

# kubectl apply -f ingress.yaml 
ingress.extensions/simple-fanout-example created

 　　查看

# kubectl get ingress -n test
NAME                    HOSTS           ADDRESS   PORTS   AGE
simple-fanout-example   php.ctnrs.com             80      25s

 　　设置hosts以后即可使用域名php.ctnrs.com访问